Improving Deadline Performance
Version: Deadline 8.0
Have you recently added more Workers to your farm and are finding that the Database has difficulty handling the load? Is your farm relatively small, but communication with the Database seems to take longer than it should? These issues and more can be resolved by adjusting your farm’s Performance Settings.
The default Performance Settings in Deadline are set for a farm with 200 Workers. If your farm has fewer Workers, then decreasing the different performance values will allow you to enjoy quicker response times from your farm! If you have more Workers, then increasing the time values will prevent Workers from competing for resources and will improve the response/lock time of your Repository and Database.
MANUALLY CHANGING PERFORMANCE SETTINGS
You can configure the Performance Settings to fit the needs of your farm from the Deadline Monitor while in Super User Mode. Just select Tools -> Configure Repository Options -> Performance Settings.
There are three tabs and a button on this page, as you can see. The first tab covers the data that the Monitor reads from the Database.
You can tweak the values (in seconds) for each collection that the Monitor tracks (Jobs, Workers, Balancers, etc). These collections are used for populating the panels in the Monitor, and the data is pulled from the Database (or the Proxy Server if you are running it). Since these retrievals can be time consuming on large data sets, trying to perform them too often can impact the Database.
For example, if you have 1500 Workers, perhaps you don’t need to refresh the Worker data every 20 seconds. Instead, you could bump this number up to 30 or 40 seconds to help reduce the load on the Database. In addition, if you aren’t running applications like Pulse or the Balancer, you can increase their update values. These kind of tweaks can allow you to improve the performance of the Database while still frequently updating the collections that are important to your workflow.
As mentioned above, most of the Monitor Performance Settings are tied explicitly to a specific panel. If there are panels you do not use, you can increase the times for those options:
- Number of seconds Between Job Updates -> The Jobs Panel
- Number of seconds Between Worker Updates -> The Workers Panel
- Number of seconds Between Pulse Updates -> The Pulses Panel
- Number of seconds Between Limit Updates -> The Limits Panel
- Number of seconds Between Cloud Updates -> The Cloud Panel
- Number of seconds Between Balancer Updates -> The Balancers Panel
- Number of seconds Between Proxy Server Updates -> The Proxy Servers Panel
The one exception is the Number of seconds Between Settings Updates option, which is not tied to any specific panel. This option is for retrieving changes to global Deadline settings, such as:
The second tab covers the data that the Deadline Worker reads from and writes to the Database.
The first option, Number of seconds Between Worker Information Updates, corresponds to how often the Worker updates its information in the Database. These updates are important, as other Deadline applications that rely on information about the Worker read that information from the Database. One of the places where this data is pulled is in the Monitor’s Worker panel, so there are effectively two different update intervals for the Worker panel:
- Number of seconds Between Worker Updates: How often the Monitor checks the Database to see if Workers have updated.
- Number of seconds Between Worker Information Updates: How often the Worker updates the information in the Database.
Number of seconds Between Queries For New Tasks While the Worker is Rendering is used when a Worker is rendering a Job and finishes a Task. Increasing this timeout could help prevent issues where too many Workers are trying to retrieve Tasks at the same time and are competing for access to the Database. Typically, this value can be left alone, or increased slightly if necessary. Note that changing this setting adds additional overhead to each Task, which can add up fast in the event that your renders are typically fast.
Multiplier to determine seconds between queries while the Worker is Idle does not explicitly refer to a time value, but rather a multiplier to be used when calculating how long a Worker should wait between checks for Jobs when it’s idle. The time that a Worker will wait is calculated by multiplying this value by the number of idle Workers in the farm. So if you have 200 idle Workers and the multiplier is 0.15, then the time each idle Worker will wait before checking for Jobs is 30 seconds (200 X 0.15). This timer is one of the only values not affected by Auto Adjust, which we will cover later.
The time idle Workers wait before checking for Jobs is also influenced by Maximum number of seconds between Job queries while the Worker is Idle and Minimum number of seconds between Job queries when the Worker is Idle. Regardless of the dynamic timeout calculate above, it will never be higher than the maximum value and will never be lower than the minimum value. This system can be very beneficial as it ensures the following:
- If your farm is idle, the multiplier with the minimum and maximum values ensures that the Workers are not all hammering the Database trying to find Jobs when there are none.
- Since the multiplier is based off of the number of idle Workers, it will dynamically decrease as the farm gets busier and increase as the farm becomes idle.
The third tab covers the data that Deadline’s Event Plugin system reads from the Database.
The only option here, Number of seconds between checking Event Plugins for updates, is used by the Deadline applications to determine when they need to update their Event information. This process is can be time consuming, as it involves checking files on the Repository machine to see if they’ve changed as well as information in the Database. Most Deadline applications start an Event Sandbox process, so keeping this value large will reduce how often these applications read this data from the Repository. Note that this value is also not affected by Auto Adjust (see below).
Managing all the above options can be time consuming itself! In order to improve your life performance, Deadline has the Auto Adjust button.
Pressing this wonderful button will present you with this delightful screen. Very neat.
You’ll notice that the number of Workers in the farm is set to 200. As mentioned earlier, the default Performance Settings are set for a farm of that size. If you have more Workers in your farm, try increasing that Worker count. Let’s assume my farm has 2000 Workers and go from there.
Look at those time values grow! The Previous Value column shows you what is currently set for each timeout and the Adjusted Value column shows what the settings will be changed to for a farm of your size. You can tinker with these updated values right in the Auto Adjust window. Let’s say you do not care about Pulse, Cloud, Proxy Server, Limit, Settings and Balancer updates in the Monitor but Job and Worker updates are important to you.
Will this perform better than our suggested defaults for 2000 Workers? It easily could! The default values are an educated guess, and your hardware setup may accommodate lower time values or may require higher time values. Deadline will make an initial estimate, but it is up to you to determine what is best for your farm.
Again, make sure to monitor your Database to determine how you can fine tune these values.
WRAPPING UP TIME
So there you have it! Performance Settings are just one tool you can use to improve your farm’s performance. For more information on the topics covered here, and other steps you can use to squeeze a bit more performance out of your farm, check out our Network Performance Documentation.