Deadline Groups and Pools: An Introduction
Version: Deadline 8.0
Deadline’s Pools and Groups are really nice ways of organizing the Deadline Workers in your farm. When Jobs are submitted, they're put into Pools and Groups. The Workers will select Jobs to render based on which Pools and Groups they’re assigned to. Today we’re going to learn how to manage both Pools and Groups, but before we get started, I suggest you read this great article that explains—in detail—the differences between Pools and Groups and how they’re intended to be used. This article also covers Limits but this post won’t be covering that.
If you don’t have enough time to read through all of that, here’s a very basic low down.
Workers with similar hardware and installed software should be placed in the same group. The order of the groups DOES NOT matter. Groups do not effect Job Scheduling, rather they are a way of quickly of filtering out a group of Workers.
For example, Workers with 16 cores and 32 GB of RAM might be grouped together. Or Workers with Maya installed will be in the ‘Maya’ Group. Workers with 3ds Max installed will be in the ‘Max’ Group. Workers that have both installed can be in both Groups.
The order of Pools DOES matter and they are typically used for prioritizing projects. Workers will prefer jobs from the Pool in the first position and then second, etc.
For example, let’s say you’ve got 30 Workers and three projects on the go. ‘Project A’ is really important and needs the most resources so we want two thirds of our farm to work on it. ‘Project B’ and ‘Project C’ are less important and only need five Workers at any one time. To accomplish this we’ll need to assign 20 Workers to the ‘project_a’ pool, and five Workers each to the ‘project_b’ and ‘project_c’ Pools respectively. To ensure that we don’t have any idle Workers, we can assign all of the Pools to the Workers and just make sure we have the correct Pool order set up. The 20 that will focus on ‘project_a’ might be assigned to Pools like this: ‘project_a’, ‘project_c’, ‘project_b’. Five Workers will have their assigned Pools look like this: ‘project_b’, ‘project_a’, ‘project_c’. Finally, five Workers will have their assigned Pools like so: ‘project_c’, ‘project_a’, ‘project_b’.
To create and assign Pools/Groups you’ll need to have Super User permissions. To do this, select Tools -> Super User Mode in the Monitor, and then enter your Super User password (if one is required).
CREATION AND ASSIGNMENT
The Manage Pools and Manage Groups dialogs are under Tools option in the Deadline Monitor, or you can right click on one or more Workers in the Workers List. Note that the right-click option will pre-filter the list of Workers in the dialog to only show the Workers that were originally selected.
On the left hand side of both dialogs you will find a list of all of your Pools/Groups, below the option to create new Pools/Groups. Click ‘New’ and enter a unique name for your Pool/Group.
The next step is to assign your newly created Pool/Group to some Workers. Select the newly created Pool/Group in the list on the left and then select the Worker(s) you wish to assign to that Pool/Group from the list on the right.
Below the list of Workers is a number of operations you can use to manipulate the Pools/Groups assigned to Workers. Click on ‘Add’ to assign the selected Pool/Group to the selected Workers. Note that Workers can be assigned to multiple Pools/Groups.
There are also options for removing a Pool/Group from a Worker, clearing all Pools/Groups assigned to a Worker, and copying and pasting a Worker's assigned Pool/Group to other Workers.
In the Manage Pools dialog, there are a couple Pool specific operations called ‘Promote’ and ‘Demote’. These operations allow you to change the priority of the selected Pool for the selected Workers (with top priority Pools on the left, and lowest on the right).
For example, let’s say the ‘mobile-024’ Workers has the ‘simproject’ Pool at top priority and the ‘compproject’ Pool at second priority.
If you select the ‘compproject’ Pool and the ‘mobile-34’ Workers, and then press the ‘Promote’ button, the ‘compproject’ Pool will be promoted to first, and ‘simproject’ will fall to second. After this change, ‘mobile-034’ will now prefer ‘compproject’ jobs first, and will only move on to ‘simproject’ jobs if there are no ‘compproject’ jobs left.
At some point you might find yourself in a situation where you no longer need a particular Pool/Group. Pools/Groups that have been deleted are not automatically removed from WorkersLuckily for us there’s an option to Purge obsolete Pools/Groups from all the Workers and from all the Jobs. To do this select the Purge checkbox in the bottom left corner of the Management dialog. Additionally, you can replace the deleted Pool/Group with another Pool/Group by selecting the new Pool or Group from the drop down below.
Here I’m Purging a deleted Group (referred to as an Obsolete Group in the dialog) and replacing all the existing assignments of the deleted Group with the ‘openstack’ Group.
The ‘none’ Pool/Group is a special case. It’s the default Pool/Group that all Jobs get put into if they aren’t assigned to an existing Pool/Group. Normally, any Worker can pick up Jobs in the ‘none’ Pool/Group but you can turn that behaviour off in Tools -> Repository Options -> Worker Settings and enabling Exclude Jobs in the ‘none’ Pool/Group. You might want to consider enabling this feature if you’re worried about a Worker picking up a Job that’s in the ‘none’ Pool/Group right before a Job is submitted into one of the Pools/Groups that Worker is assigned to. The Worker will have to finish that ‘none’ Job before it moves on to the new—probably more important—Job.
Having an ‘All’ Pool can be really helpful to avoid the hassle of assigning every Pool to all of your Workers. They’ll allow you to manage a lot of your farm very quickly with secondary Pools.
Then all of your Jobs can be submitted in the ‘All’ Pool as their Secondary Pool. A job’s Secondary Pool is a “fallback” Pool for the job. Workers will do an initial pass of the queue using each job’s Primary Pool. If the Worker doesn’t find anything then it will make a second pass using the Secondary Pool. In our example, Workers will prioritize jobs in the ‘ImportantPool’ but if there’s none in there, they’ll start picking up jobs whose Secondary Pool is All. What’s nice about this is that if you add new Workers you won’t have to assign all of the existing Pools to them, you only have to assign the ‘All’ Pool. Likewise, if you add a new Pool, you won’t have to assign it to all of your Workers.
Note: Secondary Pools only work with a Job Scheduling Order where Pools is the most important factor in scheduling. Deadline is set like this by default.