Be Environment Friendly with your Jobs!
Version: Deadline 9.0
Managing your render environment can sometimes be complicated. For example, perhaps you're using different versions of the same plugin with Maya for different render jobs, and you're relying on the MAYA_PLUG_IN_PATH environment variable to do so. Thankfully, Deadline offers a great deal of control over the environment of the render process, and allows you to modify the environment in a safe manner on a per job basis.
SET ENVIRONMENT VARIABLES FOR WORKER
The Deadline Workere, which controls the render process, will naturally inherit its environment from the process that started it (ie: the Deadline Launcher), or from the system if launched directly. The Worker environment will then be inherited by the render process when it is started.
If you would like the Worker to start with additional environment variables defined, you can set them in a shell or a script prior to starting the Worker. Here's an example of setting the Worker environment from a command prompt:
To show how the Worker environment is affected, we submitted a Python job that simply prints out the values of certain environment variables to the log. After the Worker finished the job, this was the output in the log:
... 0: INFO: Process is now running 0: STDOUT: DucksSay=Goose 0: STDOUT: MuttsSay=Woof 0: STDOUT: Done. 0: INFO: Process exit code: 0 ...
While this is one way to set the environment for the Worker and any render processes that it starts up, it's not very flexible. In the following sections, we'll take a look at how to set environment variables on a per job or per plugin basis.
SET JOB ENVIRONMENT VARIABLES DURING SUBMISSION
If you need some environment variables set for a single render job, you can specify them in the Job Info File during Manual Job Submission. For example, if you wanted to set the environment variable "Path" to "C:\", you could simply add this line to the Job Info File:
When this job gets picked up by a Worker, the Worker will set the "Path" environment variable for the render process, and override the current "Path" value. Note that this new "Path" value is only set for the render process, and not the Worker itself, which means that any jobs the Worker picks up in the future will not be affected by this change to the environment.
You can specify as many environment variables as you want in the Job Info File, as long as you follow the format of numbering them as "EnvironmentKeyValue#":
EnvironmentKeyValue0=Path=C:\ EnvironmentKeyValue1=SomeKey=SomeValue EnvironmentKeyValue2=DucksSay=Quack EnvironmentKeyValue3=TrucksSay=Vroom ...
By default, the render process will inherit the full Worker environment, and then any job environment variables will be applied. In addition, any environment variables specified by a job will take precedence over existing variables in the Worker's environment. For example, if both the Worker and the job have "DucksSay" defined in the environment, the job's value for "DucksSay" will be used. If you do not want the render process to inherit the Worker's environment, and only use the environment variables defined by the job, you can set the "UseJobEnvironmentOnly" flag in the Job Info File to "True":
Another option you have is to submit the job with the same environment as the process that submitted it. This can be useful if you want the job to render with the environment that is currently defined on your workstation. To enable this, simply set the "IncludeEnvironment" flag in the Job Info File to "True":
During submission, your current environment will automatically be applied to the job. If you use this in conjunction with the "UseJobEnvironmentOnly" flag above, the render process environment will be exactly the same as your current environment.
Continuing our example from earlier, we resubmitted the Python job with the four "EnvironmentKeyValue#" lines defined in the Job Info File. After the Worker finished the job, this was the output in the log:
... 0: STDOUT: DucksSay=Quack 0: STDOUT: MuttsSay=Woof 0: STDOUT: Path=C:\ 0: STDOUT: SomeKey=SomeValue 0: STDOUT: TrucksSay=Vroom 0: STDOUT: Done. 0: INFO: Process exit code: 0 ...
Note that the value of "DucksSay" is now the value set by the job.
SET JOB ENVIRONMENT VARIABLES FROM THE MONITOR
If you would like to set the environment variables for a job after it has been submitted, you can do so from the Deadline Monitor. Just select one or more jobs in the Job Panel, right click and select "Modify Job Properties...", and then select the "Environment" page from the list on the left.
You can use this page to add new variables, or edit and remove existing variables. You can also set the "UseJobEnvironmentOnly" setting here as well. After updating your environment variables, simply press the "OK" button to save them.
SET JOB ENVIRONMENT FROM PLUGIN
In addition to setting job environment variables at submission time, you can also set the environment variables at render time. This can be useful if you find you're always setting the same environment variables for the same type of job. Note that environment variables set at render time will take precedent over environment variables set at the job or Worker level.
To do this, you'll need to dust off your Python coding skills and customize the Application Plugin associated with the job. There are two functions that can be used to set an environment variable from a plugin:
- SetProcessEnvironmentVariable(key, value): Setting an environment variable this way will set it for all processes started by this plugin. Note this function is only available for Advanced plugins, and should be called outside of any ManagedProcess callbacks. More information on Advanced plugins and ManagedProcesses can be found here.
- SetEnvironmentVariable(key, value): Setting an environment variable this way only effects a single process being started. Note that this function should be used in Simple plugins or in ManagedProcess callbacks. When used in ManagedProcess callbacks, it will take precedence over any environment variables defined by the SetProcessEnvironmentVariable function.
Let's return to the example of our Python job. As the Python plugin is a Simple plugin, we can only use SetEnvironmentVariable. Let's add the following line to InitializeProcess:
Now if we run our job again, we get this output:
... 0: STDOUT: DucksSay=Quack 0: STDOUT: MuttsSay=Woof 0: STDOUT: Path=C:\ 0: STDOUT: SomeKey=SomeOtherValue 0: STDOUT: TrucksSay=Vroom 0: STDOUT: Done. 0: INFO: Process exit code: 0 ...
Note that the value of "SomeKey" has been changed, as the variables defined in the plugin take precedence over the ones defined in the job.
Preserving and modifying the environment of a render process is straightforward and easy in Deadline! You can define how the environment should be set at multiple different levels, and rest easy knowing that it will be set in an intuitive and predictable way.
As a reminder, here is the order in which environment variables take precedence:
Slave ---> Job ---> SetProcessEnvironmentVariable ---> SetEnvironmentVariable