Written by Maram Ayari, DevOps Engineer
Configuring Deadline rendering farms entails navigating intricate JSON files, a process that can be daunting for many administrators without specialized expertise. This complexity often leads to time-consuming setups and potential errors, hindering the performance of render farms. Additionally, the need to optimize fleet configurations for specific software suites and AWS regions further complicates the process, requiring meticulous attention to detail and in-depth knowledge of both rendering requirements and cloud infrastructure.
The Spot Fleet Management Tool (SFMT) offers a solution by simplifying Spot Fleet setup with an intuitive platform tailored for Deadline render farm management. With SFMT, administrators can streamline the configuration process, enhancing efficiency and ensuring seamless rendering operations.
Contents
What is SFMT?
The Spot Fleet Management Tool (SFMT) is a user-friendly platform that simplifies the configuration process for Spot Fleet setups. It is designed for Deadline render farm administrators lacking access to specialized deployment frameworks such as the Thinkbox Deadline RFDK (Render Farm Deployment Kit).
The solution is open-source and can be tested directly on the website or deployed from GitHub.
Key Features of SFMT
- User-Friendly Interface: SFMT provides an easy-to-use platform that simplifies Spot Fleet configuration for render farm administrators.
- Simplified Setup: SFMT provides two methods for editing the Spot Fleet configuration. Users can make adjustments through the intuitive form interface with just a few clicks or edit the JSON preview directly for more advanced customization.
How SFMT Works
The TrackIt SFMT landing page contains a description of the tool, how to use it, and a blog post detailing how users can configure Spot Fleet settings to their specifications.
Tutorial – Using the Spot Fleet Management Tool (SFMT)
The tutorial below provides a step-by-step example of how to utilize the SFMT to streamline the spot fleet configuration process. Two popular software suites will be used: 3DS Max and Blender. These fleets will be configured to operate within the US West 2 region.
1. The setup is divided into four subnets, each corresponding to a different Availability Zone to increase fault tolerance and instance availability. The machine list includes specific instance types tailored to the requirements of each software suite. For 3DS Max, the c5.4xlarge and m5.4xlarge instances have been chosen, while Blender will be provisioned with m5.8xlarge and c5.2xlarge instances.
Worker allocation is another critical aspect of the setup. It has been identified that 20 workers are needed for 3DS Max and 15 for Blender to efficiently handle the workload.
Note: Users need to check their AWS account quotas to ensure that the fleets can request the desired number of VCPUs.
Additionally, Blender’s fleet allocation strategy has been set to capacityOptimizedPrioritized, ensuring that Blender’s fleet overrides received priority in resource allocation.
| { “farm-3dsmax”: { “AllocationStrategy”: “capacityOptimized”, “IamFleetRole”: “arn:aws:iam::450006745611:role/aws-ec2-spot-fleet-tagging-role”, “LaunchSpecifications”: [], “LaunchTemplateConfigs”: [ { “LaunchTemplateSpecification”: { “LaunchTemplateId”: “lt-x4r3bb9jhc53ddzx4”, “Version”: “$Latest” }, “Overrides”: [ { “InstanceType”: “c5.4xlarge”, “SubnetId”: “subnet-9knxxx5fjlh73esrg” } ] }, { “LaunchTemplateSpecification”: { “LaunchTemplateId”: “lt-x4r3bb9jhc53ddzx4”, “Version”: “$Latest” }, “Overrides”: [ { “InstanceType”: “c5.4xlarge”, “SubnetId”: “subnet-20onxs4fjl65revbh” } ] }, { “LaunchTemplateSpecification”: { “LaunchTemplateId”: “lt-x4r3bb9jhc53ddzx4”, “Version”: “$Latest” }, “Overrides”: [ { “InstanceType”: “m5.4xlarge”, “SubnetId”: “subnet-9knxxx5fjlh73esrg” } ] } ], “ReplaceUnhealthyInstances”: true, “TagSpecifications”: [ { “ResourceType”: “spot-fleet-request”, “Tags”: [ { “Key”: “aws-rfdk”, “Value”: “0.42.0:SpotEventPluginFleet” }, { “Key”: “deployedByStudioBuilder”, “Value”: “true” } ] } ], “TargetCapacity”: 20, “TerminateInstancesWithExpiration”: true, “Type”: “maintain” }, “farm-c4d”: { “AllocationStrategy”: “capacityOptimizedPrioritized”, “IamFleetRole”: “arn:aws:iam::450006745611:role/aws-ec2-spot-fleet-tagging-role”, “LaunchSpecifications”: [], “LaunchTemplateConfigs”: [ { “LaunchTemplateSpecification”: { “LaunchTemplateId”: “lt-x4r3bb9jhc53ddzx4”, “Version”: “$Latest” }, “Overrides”: [ { “InstanceType”: “m5.8xlarge”, “SubnetId”: “subnet-20onxs4fjl65revbh”, “Priority”: 1 } ] }, { “LaunchTemplateSpecification”: { “LaunchTemplateId”: “lt-x4r3bb9jhc53ddzx4”, “Version”: “$Latest” }, “Overrides”: [ { “InstanceType”: “c5.2xlarge”, “SubnetId”: “subnet-9knxxx5fjlh73esrg”, “Priority”: 2 } ] } ], “ReplaceUnhealthyInstances”: true, “TagSpecifications”: [ { “ResourceType”: “spot-fleet-request”, “Tags”: [ { “Key”: “aws-rfdk”, “Value”: “0.42.0:SpotEventPluginFleet” }, { “Key”: “deployedByStudioBuilder”, “Value”: “true” } ] } ], “TargetCapacity”: 15, “TerminateInstancesWithExpiration”: true, “Type”: “maintain” } } |
2. After uploading a Spot Fleet configuration JSON file to the SFMT, the interface displays two existing fleets from the JSON file by adding 2 forms and also displays the uploaded JSON file in the JSON Preview section.
If the user uploads an invalid Spot Fleet configuration file, indicated by its deviation from the expected format of a JSON file containing Spot Fleet configurations, the SFMT will prompt a notification alert informing the user of the invalid file format. As a result, the Spot Fleet configuration forms will not be generated until a valid file is uploaded.
A valid uploaded file should be in the following format:
| { “fleet_name_1”: <Replace with Spot Fleet Request>, “fleet_name_2”: <Replace with 2nd Spot Fleet Request>, … } |
If the user then clicks on the ‘Add Fleet’ button on the left panel, a new fleet named ‘fleet_0’ is created:
3. To delete a fleet, users can click the delete icon in the header of the corresponding fleet form and confirm the deletion. Once removed, the corresponding fleet configuration is also removed from the JSON file displayed in the JSON Preview.
In the example below, the ‘fleet_0’ has been deleted:
4. The following parameters can be modified to edit the fleet:
- Fleet name
- Allocation strategy
- IAM role
- Launch template configurations
- Unhealthy instance handling
- Target capacity
- Termination behavior
- Tagging specifications
The SFMT provides validation checks for each field, minimizing the likelihood of errors in the Spot Fleets configuration. This ensures the edited Spot Fleets configuration is deployed successfully through the Deadline Spot event plugin.
If the user mistypes a Launch Template ID and attempts to submit the form with an invalid value, SFMT prevents the submission until a valid value with the required pattern is entered.
5. Users can also make edits directly from the JSON preview. These edits are also subject to JSON validation checks, ensuring that the edited JSON file can be successfully deployed without errors.
For example, if the user tries to edit the allocation strategy for ‘farm-3dsmax’ but types an invalid allocation strategy, a notification error is displayed with the valid values, and the user is prevented from saving the file.
6. After making edits through the form interface or directly within the JSON preview, the resulting file is available for export. This functionality allows users to save their changes locally or share them with team members for review before deployment.
After exporting the JSON configuration file, users can deploy it in Deadline 10 Spot Event Plugins by copying and pasting the JSON content. Initially, users need to switch to Super User Mode by clicking on ‘Tools’ and then selecting ‘Super User Mode’.
If it’s the user’s first time creating a Spot Fleet request with new group names, they need to first create the groups.
After that, the user clicks on ‘Tools’ again and chooses ‘Configure Events’ to start creating a Spot Fleet request.
Conclusion
The Spot Fleet Management Tool (SFMT) offers Deadline farm administrators a streamlined solution to the challenges of configuring render fleets. By simplifying Spot Fleet setup with its user-friendly interface and validation checks, SFMT enables administrators to save time and minimize errors in their render farm configurations.
About TrackIt
TrackIt is an international AWS cloud consulting, systems integration, and software development firm headquartered in Marina del Rey, CA.
We have built our reputation on helping media companies architect and implement cost-effective, reliable, and scalable Media & Entertainment workflows in the cloud. These include streaming and on-demand video solutions, media asset management, and archiving, incorporating the latest AI technology to build bespoke media solutions tailored to customer requirements.
Cloud-native software development is at the foundation of what we do. We specialize in Application Modernization, Containerization, Infrastructure as Code and event-driven serverless architectures by leveraging the latest AWS services. Along with our Managed Services offerings which provide 24/7 cloud infrastructure maintenance and support, we are able to provide complete solutions for the media industry.