Concurrent Tasks vs. Multi-Worker Rendering
Version: Deadline 7.2, 8.0
OVERVIEW - INCREASING RENDER THROUGHPUT
As more and more jobs are submitted to your render farm, it becomes important to make use of the tools Deadline provides for maintaining a speedy render pipeline. These tools are both well documented and easy to use! This blog post examines two such tools in depth: Concurrent Tasks and Multi-Worker rendering. Taking advantage of concurrent tasks and making use of multiple Workers are two ways that Deadline provides a highly efficient, scalable environment for rendering large projects.
Concurrent Tasks and Multi-Worker rendering sound almost like the same concept, but it is important to outline their differences. Concurrent tasks allow a single Deadline Worker to pick up multiple tasks of a job at one time. On the other hand, having multiple Workers running on a single machine allows for multiple jobs or tasks (or concurrent tasks!) to be picked up at once. Generally speaking, Concurrent Tasks are a good way to increase the efficiency of a single job. Meanwhile, Multi-Worker rendering often improves the efficiency of a multi-job project (multiple Workers can be used for a single job too). Though there are some tradeoffs to discuss, putting these two tools together results in an incredibly efficient way to perform large amounts of Deadline jobs.
INSTRUCTIONS AND USE-CASES
Enabling Concurrent Tasks is a great way to improve the speed of a Deadline job in the context of a single Worker. The number of concurrent tasks corresponds to the number of tasks a single Worker may pick up at one time (from a single job). For example, if I have a job with multiple tasks and I set the Concurrent Task number to 3, then the next Worker to pick up the job will pick up 3 of the job’s tasks instead of just 1!
To enable Task Concurrency for a given job, complete the following:
Select your job in the Monitor, right click the job and select the “Modify Job Properties” option. Under “General”, a “Concurrent Tasks” option is available.
Please note that not every plugin supports concurrent tasks, but every plugin that does will have a “Concurrent Tasks” option in their In-app Submitter as well as in the above Monitor option. You can also choose to limit tasks to the Worker's task limit. This means that a Worker will not pick up more tasks from a job than its limit is set to. Regardless of whether a concurrent task limit is set, no more than 16 tasks can be concurrently performed by any given Worker.
Concurrent Tasks are a smart approach to increasing the efficiency of a single job. For example, if you are rendering a Nuke scene with 30 frames, it will be faster to allow a Worker to pick up multiple tasks at a time instead of only one. Conversely, 3dsMax is highly multithreaded at render time and prefers to exist as a single task on a single Worker on a machine.
Notice in the above screenshot that multiple tasks are being rendered at one time, but only one Worker is currently online. Concurrent Tasks allow your Workers to render multiple frames at once, thus decreasing the processing time of a Deadline job, assuming your machines have the available bandwidth (network and storage i/o infrastructure restrictions).
If you are using Traditional Floating Licensing, when a Worker runs multiple tasks concurrently, only 1 floating license will be checked out. This means that you can have a Worker render up to 16 tasks at the same time, and still only use 1 Deadline Worker license. If you are using Usage Based Licensing (UBL) for Deadline, the same rule applies. For example, if you render 4 tasks concurrently over a 5 minute period, only 5 minutes of Deadline render time is consumed.
Running multiple Workers on one machine is another great way to improve the speed of your Deadline workflow. The advantage of using multiple Workers is the fact that more than one job can be accomplished in a given timeframe. A multi-Worker approach makes up for the fact that a single Worker cannot pick up tasks from multiple jobs. Multiple Workers can also pick up tasks from the same job, increasing Deadline’s efficiency in the context of a single job.
For this blog I will demonstrate how to add an additional Worker to your machine, so you can try this out yourself.
To add a new Worker, you must make sure that permission to “Launch New Named Worker” is enabled for your user group. Note that user groups can only be configured in super-user mode.
Once you are able to do so, open your Deadline Launcher and hover your mouse over “Launch Worker By Name”. Select the “New Worker Instance” option and call your new Worker anything you like.
Now you can use your multi-Worker machine to tackle multiple Deadline jobs at once. Multiple Workers are particularly useful when you have a large batch of independent jobs. For example, if you have 2 Deadline Workers and 2 Maya render jobs, you can have each Worker render 1 Maya render job and (hypothetically) complete your work twice as fast!
Rendering with multiple Workers allows multiple jobs to be finished at once. Consider the example below:
In this screenshot, 4 different Python jobs are currently running in the Deadline repository. With a SWorkerave working on each job, the overall render time is greatly reduced! Multi-Worker rendering is a great choice when you have to deal with multiple independent jobs, or a single job with a large number of tasks to complete.
Sometimes your Workers will wind up converging on a single job. While this means that job will complete quickly, it also means that other jobs will be neglected. You can use Pools and Groups to ensure that specific Workers work on specific jobs. Check out this Deadline Feature Blog entry for more in-depth information on Pools and Groups.
Similar to how a single Worker running multiple concurrent tasks only uses 1 Deadline Worker license, multiple Workers running on the same machine will together only use 1 license, provided all the Workers are on the same operating system. This means that you can have a group of Workers on a machine work on a single job, and only 1 license will be checked out.
If you are using Traditional Floating Licensing, multiple Workers running on the same machine will together only use 1 floating license, provided all the Workers are on the same operating system. This means that you can have a group of Workers on a machine work on a single job, and only 1 license will be checked out. If you are using UBL for Deadline, this is NOT the case. Deadline UBL is consumed on a per-Worker basis, so if you have 2 Workers each render a task over a 5 minute period, you will consume 10 minutes of Deadline render time.
For more information on using multiple Worker, be sure to visit the documentation.
COMBINING MULTIPLE WORKERS WITH CONCURRENT TASKS
Once you have mastered Concurrent Tasks for a single job and multi-job rendering pipelines with Multi-Worker rendering, it’s time to combine both of these concepts and see what we can come up with. If I have a group of jobs which have concurrent tasks enabled, I can deploy multiple Workers to render multiple jobs at once, and each Worker in turn will pick up multiple tasks at the same time!
In the above screenshot, I have 2 Arnold jobs running and they both support 3 concurrent tasks. I also have 5 Workers running on one machine and ready to go. Notice that 3 Workers are now working on the first job and 2 are working on the second. Taking concurrent tasks into account, this means that 9 frames of the first job are being rendered at once, and 6 frames of the second job are also being rendered at the same time. When I combine multiple Workers with concurrent tasks, not only can I render multiple jobs at once, but I can also render a large number of tasks at once within each job.
When combining concurrent tasks with multi-Worker rendering, it is important to note that the licensing rules mentioned for concurrent tasks or multiple Workers still apply. For Traditional Floating Licensing, this means that multiple Workers on the same operating system can each pick up concurrent tasks of a job, and only 1 Deadline Worker license will be checked out overall! However, this rule is different when using Deadline UBL, as each Deadline Worker instance will consume Deadline render time. This is an important distinction when determining which option works best for you.
Concurrent Tasks and Multi-Worker Rendering are two easy approaches to improve the efficiency of your Deadline experience. However, both of these tools do have tradeoffs that need to be considered.
The first thing to consider is that having concurrent tasks in a job does not always mean the job will be rendered faster. Suppose a Worker picks up 3 concurrent tasks. In most cases, this means that your Worker will complete all 3 tasks in approximately the time it normally takes to complete 1. However, if your Worker is already at its capacity with 1 task, then taking on 3 tasks would offer no worthwhile performance gain over running 1 task at a time. Ensuring your Worker has the processing power to take on additional tasks will help you prevent this drawback. You can set your own limits for concurrent tasks on each of your Workers by right-clicking that Worker in your Monitor and editing the “Concurrent Task Limit Override” in the Worker's properties.
The second tradeoff to keep in mind is the fact that the performance of your Workers is highly dependent on available network bandwidth and storage. This applies both to concurrent tasks and multi-Worker renders. The performance of your machine and your network connection both factor in to the quality of your rendering pipeline. High amounts of multi-Worker rendering and task concurrency can lead to enormous streams of data passing through your network, testing the limits of your network service. Data caching both locally and on your network is one solution to avoid network bottlenecks. In order to get a proper understanding of your machine and network performance, it may be worthwhile to run benchmarks on the applications you will be using in your render workflow.
Even with these tradeoffs in mind, Concurrent Tasks and Multi-Worker rendering are still very strong approaches to keeping your Deadline pipeline moving fast. Provided that your Workers take advantage of concurrency limits, and are working within their performance capabilities, you can be confident Concurrent Tasks and Multi-Worker rendering together will maximize the efficiency of your Deadline Workers.
That covers the advantages and reasoning behind concurrent tasks and multi-Worker rendering. Concurrent tasks allow for Workers to pick up multiple tasks of a job, allowing for faster execution of a single job. Multi-Worker rendering particularly allows for more than 1 job to be picked up at once on the same machine. Multiple Workers can also be used in the context of a single job. Concurrent Tasks and Multi-Worker rendering work together to create an efficient, streamlined Deadline experience for you and your pipeline!