Deadline Entry Points
YOUR CODE HERE
With each new version of Deadline, our customers have more options to adapt Deadline to their unique needs. Among other things, customers can extend, wrap, and/or replace Deadline functionality with their own code in several areas, which are referred to as "entry points". In this article we'll outline Deadline's entry points, which should be helpful for those considering using Deadline and may also contain a few pleasant surprises for those already using Deadline.
THE DEADLINE CONTEXT
Before getting to the list of entry points, let’s go over an important point with regard to programmatically interacting with Deadline.
Deadline has its own internal Python 2.7 interpreter which incorporates Python for .NET. Python for .Net gives the Python interpreter access to the .Net Framework in addition to standard cPython modules. Deadline's internal interpreter also exposes the Deadline API and is used for executing Deadline components authored in Python and also for executing user-defined Python scripts under the various conditions described in this article.
Whenever a Python script is being executed by Deadline's internal Python interpreter, we say it is executing within the Deadline Context. The majority of Deadline entry points outlined below are executed within the Deadline Context. And this means that Deadline Plugins are not compiled code, but rather are Python interpreted code. The notable exceptions to running in the Deadline Context are the Web Service and Power Management, which will be covered in their respective sections.
DeadlineCommand, and its sibling DeadlineCommandBG ("BG" is short for "background", not Brothers Gibb), are Deadline's command-line tools for interacting with Deadline, and they work on all of Deadline's supported platforms. DeadlineCommand supports several sub-commands that allow you to directly query and control nearly every aspect of Deadline. One subcommand in particular, ExecuteScript, allows you to run your own Python code within the Deadline Context.
./deadlinecommand ExecuteScript /path/to/myscript.py
Any program or script that can issue shell calls to the operating system can wrap DeadlineCommand or DeadlineCommandBG to access deep interaction with Deadline.
Application Plugins (referred to as Job Plugins in the past) tell Deadline how to interact with the software applications and scripts for which jobs can be submitted. Deadline 9 ships with over 80 application plugins, and you can modify the existing plugins or create your own from scratch. It's not uncommon for facilities to create in-house Deadline Application Plugins so that they can use Deadline to manage work done by in-house tools.
Application Plugins are placed in their own folder within ../<DeadlineRepository>/custom/plugins/
- Feature Blog Article: Plug it in, Plug it in - Creating New Plugins
- Online User Manual: Application Plugins
There are four additional entry points associated with application plugins where you can apply your own code. For these I will quote the well-stated descriptions from the docs:
JobPreLoad.py: The JobPreLoad.py file is an optional script that will be executed by the Worker prior to loading a job that uses this plugin. It is placed in the <your_repo>/plugins/MyPlugin/ directory of your Repository, where MyPlugin is the name of the specific app plugin you wish the script to execute against. Note that in this case, the file does not share its name with the plugin folder.
PluginPreLoad.py: The PluginPreLoad.py file is an optional script that will be executed by the Worker prior to executing any Python script for the plugin (MyPlugin.py or JobPreLoad.py), and any pre or post job or task script for the current job. It is placed in the <your_repo>/plugins/MyPlugin/ directory of your Repository, where MyPlugin is the name of the specific app plugin you wish the script to execute against. Note that in this case, the file does not share its name with the plugin folder.
GlobalJobPreload.py: The GlobalJobPreLoad.py file is a functionally identical script to a JobPreLoad.py script except that it is placed in the <your_repo>/plugins directory of your Repository and will execute for ALL Deadline app plugin types. Additionally, a GlobalJobPreLoad.py script file placed in <your_repo>/custom/plugins will override the same named script file if one is present in <your_repo>/plugins. Note, if present this GlobalJobPreLoad.py script will execute prior to any application plugin specific JobPreLoad.py script that might be present.
GlobalPluginPreLoad.py: The GlobalPluginPreLoad.py file is a functionally identical script to a PluginPreLoad.py script except that it is placed in the <your_repo>/plugins directory of your Repository and will execute for ALL Deadline app plugin types. Additionally, a GlobalPluginPreLoad.py script file placed in <your_repo>/custom/plugins will override the same named script file if one is present in <your_repo>/plugins. Note, if present this GlobalPluginPreLoad.py script will execute prior to any application plugin specific PluginPreLoad.py script that might be present.
When a Job is submitted to Deadline, whether from an integrated submitter, from Monitor, or programmatically via the command line or APIs, the generic options for the Job are described by a plain text file, called the JobInfo file, which is passed as one of the Job's submission parameters. Each generic option is described by a line within the file that has the form <key>=<value>, where key is the name of an option and value the setting for the option.
There are five keys, each of which is named for an execution condition and each of which is optional, that can be placed in the JobInfo file to instruct Deadline to execute a script. The associated value is simply the path to the script(s) that may be placed anywhere visible to all Deadline clients (or at least to the ones from which the scripts might be called).
Here are the five keys, along with a brief descriptions taken from the User Manual:
ScriptDependencies=<scriptPath,scriptPath,scriptPath> : Specifies what Python script files will be executed to determine if a job can resume (default = blank). These script paths must be identified using full paths, and multiple paths can be separated with commas. See the Scripting section of the documentation for more information on script dependencies.
PreJobScript=<path to script> : Specifies a full path to a Python script to execute when the job initially starts rendering (default = blank).
PostJobScript=<path to script> : Specifies a full path to a Python script to execute when the job completes (default = blank).
PreTaskScript=<path to script> : Specifies a full path to a Python script to execute before each task starts rendering (default = blank).
PostTaskScript=<path to script> : Specifies a full path to a Python script to execute after each task completes (default = blank).
- Online User Manual: Manual Job Submission
When various conditions transpire within the Deadline environment, Deadline will trigger "Events". You can create your own scripts, called Event Plugins, to be called when these Events transpire, which include such things as Job submissions, Job completion, a Worker starting a Job, Deadline's Housecleaning cycle, and many other conditions. Event Plugins are one of Deadline's most powerful features with regard to pipeline integration.
Event Plugins are placed in their own folder within ../<DeadlineRepository>/custom/events/
Deadline's Power Management feature allows Deadline to start and stop machines based on idle detection and thermal conditions. For Idle Shutdown, Machine Startup, and Machine Restart, a command can be run in place of the built-in functions. The command may be any system command or executable, which might include calling Python or DeadlineCommand. Tags can be used within the text of the command and they will be replaced with their respective values when Deadline issues the command. If a command is specified, then it should ultimately accomplish what was intended (shutdown, startup, restart), but it also affords an opportunity to execute any custom code that may be desired (e.g, for tracking purposes or status updates to external systems).
Note: Since the command to be run is any system command, it does not have to be authored in Python and does not run in the Deadline Context.
- Online User Manual: Power Management
Monitor can be extended in several places, which I have categorized into Submission Scripts, General Scripts, and Right-Click Scripts below.
- Feature Blog Article: Hello Monitor - An Introduction to Monitor Scripting
SUBMISSION SCRIPTS (MONITOR, LAUNCHER)
Submission Scripts allow users to submit Jobs from the Deadline UI, as opposed to from within an application via an integrated submitter or via the command line with DeadlineCommand. A Submission Script will appear in two places: One place is in the menu bar of Deadline Monitor under the Submit menu item, and the other place is from the Launcher Icon under the Submit menu item.
Submission scripts are placed in ../<DeadlineRepository>/custom/scripts/Submission/
- Online User Manual: Submission Scripts
General Scripts allow any sort of custom action to be triggered from within the Monitor or Launcher UIs. In Monitor they appear under the Scripts menu item, and from the Launcher tray icon they also appear under the Scripts menu item. As the online docs state: "Under the hood, there technically isn’t anything different between General and Submission scripts. The only real difference is that they show up under different menus, which is just to help keep scripts semantically separated."
Monitor Scripts are placed in ../<DeadlineRepository>/custom/scripts/General
- Online User Manual General Scripts
The bulk of the information displayed in Deadline Monitor is conveyed through topic-specific panels. For each of the following panels, you can add your own scripts to the context menu that appears when a user right-clicks on an item listed in the panel:
- Job Panel (../<DeadlineRepository>/custom/scripts/Jobs)
- Task Panel (../<DeadlineRepository>/custom/scripts/Tasks)
- Pulse Panel (../<DeadlineRepository>/custom/scripts/Pulse)
- Balancer Panel (../<DeadlineRepository>/custom/scripts/Balancer)
- Limit Panel (../<DeadlineRepository>/custom/scripts/Limits)
- Job Report Panel (../<DeadlineRepository>/custom/scripts/JobReports)
- Online User Manual Customizing Script Display
Deadline Balancer is the component of Deadline that is responsible for launching and terminating virtual machines (VMs) based on demand in the queue. There are two areas within Balancer that can be customized, Cloud Plugins and Balancer Plugins, that are discussed in the following sections.
The algorithm that Balancer uses to determine how many virtual machines should be running in each Cloud Region is also pluggable. Balancer gathers up state information and Cloud Region settings and then passes that information to the currently-selected Balancer Plugin. The plugin must then return the absolute counts (and type) of VMs that should be running in each region. Deadline ships with its own Default Algorithm plugin, which is suitable for most use cases, but if the Default Algorithm does not meet your needs, you can create your own.
Balancer Plugins are placed in their own folder within ../<DeadlineRepository>/custom/balancer/
- Online User Manual: Balancer Plugins
DEADLINE WEB SERVICE
The Deadline Web Service exposes a RESTful HTTP API for interacting with Deadline. This is often preferred to wrapping DeadlineCommand for long-lived, formal services that interact with Deadline.
RESTFUL HTTP API
An advantage of interacting with Deadline via a RESTful HTTP API is that the interaction does not necessarily have to be coded in Python. It can be coded in any language that can communicate via HTTP REST.
- Online User Manual: REST Overview
WEB SERVICE SCRIPTS
It is also possible to expose your own custom functions through the Web Service. This is done by authoring Web Service scripts. Like other Deadline Plugins, these are authored in Python 2.7 and are executed within the Deadline Context.
Web Service Scripts are placed in ../<DeadlineRepository>/custom/scripts/WebService
- Online User Manual: Web Service Scripts
STANDALONE PYTHON API
Deadline ships with a Python wrapper for the Web Service's RESTful HTTP API. This provides an alternative to running scripts via DeadlineCommand with the ExecuteScript subcommand, which is beneficial in cases where interaction with Deadline is needed from a machine that does not have the Deadline Client software installed. Only Python 2.7 or later and the wrapper API module files are needed.
MORE TO COME
This blog entry demonstrates the high degree of code customization that can be applied to Deadline. I expect the degree of access into Deadline to further expand and mature in future releases. And, of course, you are always welcome to make requests via our Support Forum and Support Ticket systems.
And, as we always like to mention in our customization-related articles, be sure to check out Deadline's public GitHub repository for a wealth of examples.