Blastoff With The Launcher
Version: Deadline 8.0
While the Deadline Launcher is primarily used for launching (pun absolutely intended) the other Deadline applications, this often overlooked and under appreciated application is actually responsible for many things, including:
- handling remote commands
- automatic upgrading and downgrading
- keeping other Deadline applications running
- popup notifications
- Deadline Worker scheduling and idle detection
- checking for changes to Auto-Configuration
It acts as the central point of the Deadline Client installation, and is integral to keeping the other Deadline applications running smoothly. Therefore, it is recommended that you keep the Launcher running on every machine in your render farm.
LAUNCHER? WHAT LAUNCHER?
The Launcher is installed during the Client Installation, and is automatically configured to start on machine startup, so it's probably already running on your machine right now! Where is it though? If you didn't install the Launcer as a service or daemon, it will be hiding in your system tray or notification area on the desktop.
If you chose to install the Launcher as a service or daemon, you won't be able to actually see it, but it's there, watching you, like a ghost. Now that we are all terrified, let us cover some of the Launcher's responsibilities.
Remote Commands are messages sent from one Deadline Client machine to another, and they typically originate from the Deadline Monitor on the sending machine. They can be used to start or stop different Deadline applications, set what action a Worker should take after completing its current Task, or modify the machine state (ie: shutdown, restart, start up or suspend the machine). You can also send commands to execute arbitrary commands.
To use Remote Commands, you must first enable Remote Administration in the Repository Configuration. This can be done from the Monitor by selecting Tools > Configure Repository Options, and then by selecting the Client Setup page from the list on the left. Note that you must have Super User privileges to access the Repository Options.
See the Remote Control documentation for more information on using Remote Commands.
HOW IS THE LAUNCHER INVOLVED?
As previously mentioned, the Launcher handles all incoming Remote Commands. Therefore, if you send a Remote Command to a Deadline Client machine to start a Worker, you're actually sending that command to the Launcher on that machine. If the command being sent requires interacting with a running Deadline Application, such as telling Pulse to restart for example, then the Launcher will open communication with the target Deadline application and send it the command.
Note that all Remote Commands are sent through the Launcher Listening Port, which can be set in the system-wide deadline.ini file on each machine. It's important that this port be the same across all Deadline Client machines in your farm. If you have two Clients that are configured to use different ports, they will not be able to send Remote Commands to each other. Luckily, the Client Installer already sets the port number automatically during installation, so you should only change the port if there is a conflict.
WHY NOT HAVE THE DEADLINE APPLICATIONS LISTEN FOR REMOTE COMMANDS?
Having the Launcher handle the Remote Commands and pass them on when needed standardizes how the commands are handled, guaranteeing that they all must pass through a single point (the Launcher being the central point of Deadline Client on each machine). It also means the remote communication is only done over a single port (the Launcher Listening Port mentiond above).
In addition, commands like starting the Worker or Pulse need to be handled by something if those appliations aren't already running. When handling commands like shutting down or restarting the machine, the Launcher can also make sure the other Deadline applications shut down cleanly before the Launcher triggers the command.
Finally, the Launcher really needs this. Don't take this away from the Launcher!
AUTOMATIC UPGRADING AND DOWNGRADING
Automatic Upgrading and Downgrading, or Auto-Upgrading for short, is a helpful tool for rolling out new releases of Deadline across your farm. If Auto-Upgrading is enabled, you simply need to upgrade your Deadline Repository installation, and then you can trigger the Deadline Clients to automatically upgrade themselves. Note that Auto-Upgrading is only supported between versions that are part of the same major release cycle (for example, 8.0 to 8.0.1, or 7.1 to 7.2). More information on Auto-Upgrading can be found in the Upgrading or Downgrading Deadline documentation, or you can check out this previous blog entry.
To use Auto-Upgrading, you must first enable Automatic Upgrades / Downgrades in the Repository Configuration. This can be done from the Monitor by selecting Tools > Configure Repository Options, and then by selecting the Client Setup page from the list on the left. Note that you still must have Super User privileges to access the Repository Options.
TRIGGERING AN AUTO-UPGRADE
Auto-Upgrading checks are performed when the Launcher attempts to start a Deadline application, by comparing its version with the version stored in the Repository. So it can be triggered by starting a Deadline application from the Launcher's menu, or it can be triggered manually by running the following command from the command line:
If Auto-Upgrading is enabled and the Client version is different from the Repository version, then the Launcher will take note of all running Deadline applications, shut them all down, shut itself down, upgrade, relaunch all previously running Deadline applications, and finally perform whatever action triggered the Auto-Upgrade check in the first place.
Even better, you can trigger a restart of the Deadline Worker application across all of your render nodes by using a Remote Command. This allows you to upgrade your entire render farm without even getting up! See the Upgrading or Downgrading the Render Nodes documentation for more information.
KEEPING DEADLINE RUNNING WITH THE LAUNCHER
Look folks, sometimes bad things happen to good applications. When your Deadline application has a little hiccup and ends up crashing, you want that bad boy to start right back up don't you? Then you need to use this other handy feature of the Launcher!
The exposure for this feature is limited to the Worker currently, without manipulating your deadline.ini file. For the Worker, this feature will make sure that any Worker running on the Launcher machine that stalls gets restarted. The aptly named Restart Worker If It Stalls flag is accessible from the Launcher right-click menu.
When a Worker crashes, it will stop updating its status. During House Cleaning, Deadline will eventually detect that the Worker has not updated its status in a certain amount of time (minimum 5 minutes, configurable through the Repository Settings in the Monitor). When this happens, the Worker's status will be updated to Stalled, signifying that something went wrong with that Worker and it is no longer running. If the Restart Worker If It Stalls flag is set, then the Launcher will eventually start that Worker back up. This can take up to an additional 5 minutes after the Worker has been marked as Stalled, as the Launcher performs this check independently every 5 minutes.
You can also set this flag through the system-wide deadline.ini file. The flags you need to set to keep the various Deadline applications running are:
- RestartStalledWorker: If a Worker is marked as Stalled it will be restarted by the Launcher the next time the Launcher checks for dead applications.
- KeepPulseRunning: Ensures that Pulse is always running on this machine.
- KeepBalancerRunning: Ensures that the Balancer is always running on this machine.
- KeepWebServiceRunning: Ensures that the Web Service is always running on this machine.
- KeepProxyServerRunning: Ensures that the Proxy Server is always running on this machine.
- KeepLicenseForwarderRunning: Ensures that the License Forwarder is always running on this machine.
You can also specify in the per-user deadline.ini to start the various Deadline applications when the Launcher starts with the following flags:
BONUS LAUNCHER RESPONSIBILITIES
In addition to everything covered so far, the Launcher is also responsible for popup notifications, Deadline Worker scheduling and idle detection, and auto-configuration.
Popup notifications have already been covered in a previous blog post. If you did not follow either of those links, I'm about to blow this whole thing wide open: popup notifications are notifications that pop up on a user's machine when something happens to a Job. The Launcher needs to be running in order for popup messages to pleasantly pop up, and must not be running as a service.
Another feature the Launcher is responsible for is Worker Scheduling and Idle Detection. Worker Scheduling can be used to configure when the Deadline Worker should be started and stopped on specific machines, and Idle detection can be used to detect when there has been no keyboard or mouse activity for a certain period of time, and automatically start the Deadline Worker. This feature will only work if the Launcher is running on that machine, and note that Idle Detection only works when the Launcher isn't running as a service.
Finally, the Launcher can also make some timely changes to the machines local settings by keeping an eye on changes to Auto-Configuration, the process used to configure some Deadline Client settings such as region, licensing, Repository path and much more.
In addition to the specific responsibilities highlighted, the Launcher is also capable of doing the following:
- running scripts
- submitting jobs
- changing the current Repository
- changing the current region
- changing the current user
- changing the licensing settings
- setting the flag for caching the Repository files locally
- modifying the setting for Local Workers
- and of course, launching (pun still intended) other Deadline applications
While it may not be as flashy as other Deadline applications, it's happy to run in the background or hide in your system tray, and it's an essential piece of the Deadline render farm puzzle. If you still want to learn more, check out the Launcher documentation.