Welcome!

Buildcat is the lightweight, flexible render farm implemented using Python, RQ and Redis. It runs on MacOS, Linux, and Windows (WSL), requires almost no configuration, includes integrations for Houdini, Modo, and Redshift, and can be easily extended to support other tools.

Documentation

Installation

Buildcat

To install the latest stable version of Buildcat and its dependencies, use pip:

$ pip install buildcat

… once it completes, you’ll be able to use all of Buildcat’s core features.

Redis

To run a buildcat server, you’ll also need Redis which can’t be installed via pip. If you use Conda (which we strongly recommend), you can install it as follows:

$ conda install redis

Once you have Redis, you can install Buildcat and the rest of its dependencies:

$ pip install buildcat

Documentation

We assume that you’ll normally access this documentation online, but if you want a local copy on your own computer, do the following:

Install Buildcat along with all of the dependencies needed to build the docs:

$ pip install buildcat[doc]

Next, do the following to download a tarball to the current directory containing all of the Buildcat source code, which includes this documentation:

$ pip download buildcat --no-binary=:all: --no-deps

Now, you can extract the tarball contents and build the documentation (adjust the following for the version you downloaded):

$ tar xzvf buildcat-0.6.1.tar.gz
$ cd buildcat-0.6.1/docs
$ make html

Tutorial

We created Buildcat because we were frustrated with the complexity and builtin assumptions of other render farms. Buildcat is written in Python for portability, simplicity, and flexibility. It runs on MacOS, Linux, and Windows (using WSL), requires little-to-no configuration, and uses existing open source software wherever possible instead of reinventing wheels.

We strongly recommend you follow this tutorial to quickly setup a minimal-but-complete Buildcat render farm on a single machine. It’s quick and easy, and once you’ve done it on one machine, you’ll have a good feel for the process. Then, individual articles in the User Guide will discuss how to setup a full-fledged render farm deployed on multiple hosts.

Tip

While Buildcat is as simple as we could possibly make it, there are still many moving parts in a render farm - make sure you read and understand the following description of how the parts of a Buildcat farm work together before you start!

Your Buildcat render farm will have all of the following:

Server

The server keeps track of your render jobs, dispatching them to workers as the workers become available. In Buildcat, the server is actually an instance of the widely used open source Redis key-value store.

Workers

Workers are the processes that carry out the actual work of rendering. Idle workers receive jobs from the server and execute commands with your DCC tools, typically (but not necessarily) instructions to render an image from an animation. Buildcat workers are actually instances of RQ workers executing Buildcat integrations written for rendering.

Integrations

Integrations are the DCC-tool-specific code provided by Buildcat for rendering. Technically, integrations are Python functions that are called when a job is executed by a worker. You can easily use your own functions with Buildcat workers, written to work with your own tools for any purpose, not just rendering.

Clients

Clients submit jobs to the server to be rendered by workers. Any Python code can use the RQ API to submit a job to the server, and Buildcat provides API to make job submissions even easier. Buildcat also provides a command-line client that can be used to submit jobs using any of Buildcat’s builtin integrations. Using the API or the command line tool, jobs can be submitted from within your DCC tool using scripting.

Shared Storage.

Typically, a render job includes just the name of a scene file and a range of frames to render. To run the job, your workers must have access to the scene file and all of the assets it uses, including geometry, cached simulations, textures, and-so-on. Typically, this means setting up a network share that all of the workers can reach. It also means that artists will need to ensure that scene files and assets are moved to this location prior to job submission.

Network Connectivity.

To function, all clients and all workers must be able to communicate with the server.

Note

The server, workers, and clients in your Buildcat render farm can be run on any combination of hosts, including any mixture of platforms.

Platform Notes

Now you’re ready to run through the tutorial. In the sections that follow, we will put platform-specific information in tabs, as you see here … please choose the tab for your platform before proceeding:

LinuxMacOSWindows

You made the right choice! Proceed to the next section.

Shared Storage

The first step in setting-up your render farm will be to create or choose a filesystem location for the shared storage space. Throughout this documentation we will refer to this location as BUILDCAT_ROOT. Keep in mind that BUILDCAT_ROOT must have enough storage for your projects, their assets (geometry, textures, simulations, etc), and all of their rendered outputs, so you’ll want to pick a location on a disk with as much free space as possible.

LinuxMacOSWindows

For these examples, we’ll assume that BUILDCAT_ROOT is located in an external drive mounted on the path /mnt/farm.

Network Communication

The next step in setting-up any render farm is knowing how your server, workers, and clients will communicate with one another. In particular, you need to know the network address of the machine running your server.

Since our sample render farm will be running on one machine, we’ll use the loopback address - 127.0.0.1 - as our server address throughout the rest of this section.

Anaconda

Now that we’ve selected BUILDCAT_ROOT and our network address, it’s time to start installing software. We’ll need Python and all of Buildcat’s other dependencies. Getting all of the pieces in place (and dealing with different platforms) is one of the things that makes installing render farm software difficult.

Instead, whether you’re new to Python or an old hand, we strongly recommend installing Miniconda, a minimalist subset of Anaconda. Anaconda is a portable (MacOS, Linux, and Windows) Python distribution that you can use to conveniently install Python in your home directory. This is incredibly useful because installing the rest of Buildcat’s dependencies is easy and consistent across platforms, you get access to the latest versions of Python and related software, and your separate Anaconda install leaves your system-provided Python in pristine condition.

The remainder of this documentation will assume that you have Anaconda installed. You can still obtain Buildcat’s dependencies using other methods, but you’ll need to handle those details on your own.

LinuxMacOSWindows

Use the “Python 3.9 Miniconda3 Linux 64-bit” installer from https://docs.conda.io/en/latest/miniconda.html to install Anaconda in your home directory.

Server

Now we’re ready to install the Buildcat (Redis) server from a command line, using Anaconda:

$ conda install redis

Next, install the Buildcat software using pip:

$ pip install buildcat

Next, start the server:

$ buildcat server

The server will print some startup information to the console and begin waiting for connections:

15455:C 05 Jul 2019 15:50:57.378 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
15455:C 05 Jul 2019 15:50:57.378 # Redis version=5.0.3, bits=64, commit=00000000, modified=0, pid=15455, just started
15455:C 05 Jul 2019 15:50:57.378 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
15455:M 05 Jul 2019 15:50:57.379 * Increased maximum number of open files to 10032 (it was originally set to 256).
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 5.0.3 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 15455
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

15455:M 05 Jul 2019 15:50:57.380 # Server initialized
15455:M 05 Jul 2019 15:50:57.380 * Ready to accept connections

Note that the server listens for connections on the loopback address - 127.0.0.1 - by default, so we don’t have to specify it explicitly. Leave the server running, and open another command line console to run the next set of commands.

Worker

Now we’re ready to run a worker. Since we already installed Buildcat in the previous step, there’s nothing to do except fire it up:

LinuxMacOSWindows

$ cd /mnt/farm
$ buildcat worker

The worker will print a startup message, begin communicating with the server, and wait for jobs to work on:

13:23:51 Worker rq:worker:87138a93131c4b33a2ebe7d8a3c9c528: started, version 1.7.0
13:23:51 Subscribing to channel rq:pubsub:87138a93131c4b33a2ebe7d8a3c9c528
13:23:51 *** Listening on default...
13:23:51 Cleaning registries for queue: default

Note that we didn’t have to specify the server address because it defaults to to 127.0.0.1 for the worker, too. Also, before starting the worker we used the cd command to change the working directory to BUILDCAT_ROOT. This is how the worker knows where BUILDCAT_ROOT is located without having to configure it. Leave the worker running and open another command line for the following steps.

Client

Now it’s time to test the farm, by submitting a job to the server. To keep things simple, we’re going to use Buildcat’s builtin command line client, as this is the easiest way to confirm that everything’s working:

$ buildcat worker-info

This command submits a buildcat.worker.info job to the server, which hands it off to any available worker. If you check the console where we left our worker running, you’ll see that it accepts the job and runs it:

13:25:53 default: buildcat.worker.info() (b8de2065-9fd7-4018-b77c-dd930f388880)
13:25:53 default: Job OK (b8de2065-9fd7-4018-b77c-dd930f388880)
13:25:53 Result is kept for 500 seconds

… and in the console where you submitted the job, information about the worker is printed out:

{'os': {'host': 'tim-aurora',
        'machine': 'x86_64',
        'processor': 'x86_64',
        'release': '4.4.0-19041-Microsoft',
        'system': 'Linux',
        'version': '#488-Microsoft Mon Sep 01 13:43:00 PST 2020'},
 'python': {'prefix': '/home/tshead/miniconda3',
            'version': '3.8.5 (default, Sep  4 2020, 07:30:14) \n[GCC 7.3.0]'},
 'worker': {'pid': 224,
            'root': '/home/tshead',
            'user': 'tshead',
            'version': '0.4.0-dev'}}

This confirms that the client, server, and worker are all communicating and ready to go to work!

Summary

That’s it! Your single-machine render farm is up-and-running. Of course, there are many details we’ve skipped in this section, such as how to run workers on multiple machines, submit real render jobs, and how to secure your farm’s network connections. The articles in the User Guide will address these problems in detail.

User Guide

The User Guide includes detailed individual subjects covering how to setup and use Buildcat effectively.

Houdini

To integrate Buildcat with SideFX Houdini, we provide integrations to return information about a worker’s local Houdini installation, render a range of frames from a Houdini .hip file, and render an .ifd file.

Installation

To use the Houdini integrations, each worker must be able to run the hython and mantra executables provided by Houdini. Typically, this means that you will have to install Houdini on the worker host, locate the hython and mantra (or hython.exe and mantra.exe on Windows) executables, and add that location to the PATH environment variable before starting the worker. See the Houdini Documentation for details on where hython and mantra are located for specific platforms.

To submit a Houdini scene for rendering, save the scene file and all of the scene assets to BUILDCAT_ROOT, and ensure that all scene assets are accessed using relative paths (for example, relative to $HIP), including output files. Use Houdini’s Render > Pre-Flight Scene … menu item to double check your asset locations. It’s important that every input or output for your project is contained within BUILDCAT_ROOT.

Command Line Integration

To query for information about a worker’s Houdini installation from the command line, use the buildcat command:

$ buildcat houdini-info
{'houdini': {'executable': 'hython.exe', 'version': '18.5.408'},
 'os': {'host': 'tim-aurora',
    'machine': 'x86_64',
    'processor': 'x86_64',
    'release': '4.4.0-19041-Microsoft',
    'system': 'Linux',
    'version': '#488-Microsoft Mon Sep 01 13:43:00 PST 2020'},
 'python': {'prefix': '/home/tshead/miniconda3',
            'version': '3.8.5 (default, Sep  4 2020, 07:30:14) \n[GCC 7.3.0]'},
 'worker': {'pid': 225,
            'root': '/home/tshead',
            'user': 'tshead',
            'version': '0.4.0-dev'}}

Keep in mind that houdini-info will block until a worker handles the request, and that the results will vary depending on which worker does so.

Next, you can use the buildcat command to render a .hip file:

$ buildcat houdini-render-hip path/to/scene.hip

Note that the path to the .hip file must be relative to BUILDCAT_ROOT … so if your BUILDCAT_ROOT is on a network disk //aurora/farm and your scene is stored as //aurora/farm/projectfoo/scene24/take13.hip, your buildcat command line would be:

$ buildcat houdini-render-hip projectfoo/scene24/take13.hip

By default, the houdini-render-hip subcommand will render frame 1 of your scene. To render a different frame, just specify it after the hip file:

$ buildcat houdini-render-hip projectfoo/scene24/take13.hip --frames 8

Or to render all of the frames between 1-100 (inclusive):

$ buildcat houdini-render-hip projectfoo/scene24/take13.hip --frames 1-100

If you want to render every 3rd frame, you can do that too:

$ buildcat houdini-render-hip projectfoo/scene24/Take13.hip --frames 1-100:3

You can combine as many individual frames and ranges as you like, using commas:

$ buildcat houdini-render-hip projectfoo/scene24/Take13.hip --frames 1,7,9,10-25,75-100:2

Be aware that houdini-render assumes that your .hip file has a ROP node named /out/mantra_ipr that it will use for the render. If you want to render with some other ROP, you’ll need to specify that too:

$ buildcat houdini-render-hip projectfoo/scene24/take13.hip --frames 1-100 --rop /out/mantra1

After starting the render, keep an eye on the contents of BUILDCAT_ROOT, and you will see rendered frames begin to appear.

Parallelism

A limitation of houdini-render-hip is that it renders frames one-after-the-other as a single job. This is what you want if you’re caching the results of a simulation or other geometry to a disk, but won’t help speed up rendering if you want to render frames simultaneously on multiple machines. To do that, there are several possible alternatives:

On one hand, you could submit multiple jobs with houdini-render-hip that each render a subset of frames. For example, if you had two machines with similar resources, you might split your job into two:

$ buildcat houdini-render-hip projectfoo/scene24/take13.hip --frames 1-50 --rop /out/mantra1
$ buildcat houdini-render-hip projectfoo/scene24/take13.hip --frames 51-100 --rop /out/mantra1

As long as you have a worker waiting on each machine, both jobs will run simultaneously, and the render should complete in roughly half the time. Note that this can be an efficient approach because the hython processes that do the work only have to be started and stopped once.

On the other hand, the above approach can be suboptimal if your machines have mismatched capabilities - if you split the job into equal halves, but one machine is faster than the other, it will quickly complete its half and sit idle until the slower machine eventually finishes its half; in this case, it would be better if the render is load balanced, so both machines can work at their full capacity, and finish at roughly the same time. To accomplish this, you can submit one job per frame:

$ buildcat houdini-render-hip projectfoo/scene24/take13.hip --frames 1 --rop /out/mantra1
$ buildcat houdini-render-hip projectfoo/scene24/take13.hip --frames 2 --rop /out/mantra1
$ buildcat houdini-render-hip projectfoo/scene24/take13.hip --frames 3 --rop /out/mantra1

...

$ buildcat houdini-render-hip projectfoo/scene24/take13.hip --frames 100 --rop /out/mantra1

Because this approach creates many small jobs, there is more overhead from hython processes starting and stopping, but both machines can complete as many jobs as their resources allow.

The downside to either approach is that every worker that’s rendering a .hip file consumes a full Houdini license while it runs. Depending on how many workers you have, you may be prevented from using Houdini during rendering, or you may run out of licenses altogether, and overcoming this requires a more complicated approach:

First, create a Mantra ROP in your .hip file that writes .ifd files to disk, instead of rendering. As always, make sure the .ifd files and temp storage directories are located within $BUILDCAT_ROOT.

Second, start a job that renders the .ifd-generating ROP in your scene:

$ buildcat houdini-render-hip projectfoo/scene24/take13.hip --frames 1-100 --rop /out/generate_ifds

Finally, submit jobs to render each of the .ifd files:

$ buildcat houdini-render-ifd projectfoo/scene24/ifds/take13-0001.ifd
$ buildcat houdini-render-ifd projectfoo/scene24/ifds/take13-0002.ifd
$ buildcat houdini-render-ifd projectfoo/scene24/ifds/take13-0003.ifd

...

$ buildcat houdini-render-ifd projectfoo/scene24/ifds/take13-0100.ifd

Now, a Houdini license is only consumed while the .ifd files are generated by the first job. The remaining jobs render a single .ifd file apiece and each consume a single mantra license while rendering, allowing you to render with as many workers as you have render licenses, and continue using Houdini while they work.

Scripted Integration

Instead of using the command line, you might want Houdini to automatically submit Buildcat jobs for you. For example, if you have a ROP that generates .ifd files as described above, it’s useful to have the ROP submit a job for each .ifd file as soon as it’s written to disk, so rendering can begin immediately, instead of idling workers while .ifd generation completes. To do so, add a Python Post-Frame Script to your .ifd-generating ROP, with contents along the lines of the following:

import os

import hou
import redis
import rq

filename = os.path.relpath(hou.ch("soho_diskfile"), hou.getenv("HIP"))

queue = rq.Queue(connection=redis.Redis())
queue.enqueue("buildcat.hou.render_ifd", filename)

Now when you generate the .ifd files, render jobs will be automatically created. Note that this script assumes that you use the $HIP environment variable to anchor the .ifd and temporary storage paths, and that $HIP refers to the same location as BUILDCAT_ROOT. It also assumes that you’ve configured Houdini to load the redis and rq Python modules from an external location (these modules are not provided by Houdini).

Modo

To integrate Buildcat with Foundry Modo, we provide a pair of integrations that can return information about a worker’s local Modo installation, and render a range of frames from a Modo .lxo file.

Installation

To use the Modo integrations, each worker must be able to run the modo_cl executable provided by Modo. Typically, this means that you will have to install Modo on the worker host, locate the modo_cl (or modo_cl.exe on Windows) executable, and add that location to the PATH environment variable before starting the worker. See the Modo Documentation for details on where modo_cl is located for specific platforms.

Production

To query for information about a worker’s Modo installation from the command line, use the buildcat command:

$ buildcat modo-info
{'modo': {'executable': 'modo_cl.exe', 'version': '1321'},
 'os': {'host': 'tim-aurora',
        'machine': 'x86_64',
        'processor': 'x86_64',
        'release': '4.4.0-19041-Microsoft',
        'system': 'Linux',
        'version': '#488-Microsoft Mon Sep 01 13:43:00 PST 2020'},
 'python': {'prefix': '/home/tshead/miniconda3',
            'version': '3.8.5 (default, Sep  4 2020, 07:30:14) \n[GCC 7.3.0]'},
 'worker': {'pid': 237,
            'root': '/home/tshead',
            'user': 'tshead',
            'version': '0.4.0-dev'}}

Note that modo-info will block until the results are received, and that the results could vary depending on which worker handled the request.

To submit a Modo scene for rendering save the scene file and all of the scene assets to BUILDCAT_ROOT, and ensure that all scene assets are accessed with relative paths.

Then you can use the buildcat command to start rendering:

$ buildcat modo-render path/to/scene.lxo

Note that the path to the scene file must be relative to BUILDCAT_ROOT … so if your BUILDCAT_ROOT is on a network disk //Aurora/Farm and your scene is stored as //Aurora/Farm/ProjectFoo/Scene24/Take13.lxo, your buildcat command line would be:

$ buildcat modo-render ProjectFoo/Scene24/Take13.lxo

By default, modo-render will render frame 0 of your scene. To render a different frame, or multiple frames, you need to specify the start (first frame to render) and stop (one past the last frame to render) of the desired range. So, to render frame 8:

$ buildcat modo-render projectfoo/scene24/take13.lxo --start 8 --stop 9

Or to render frames 1-100:

$ buildcat modo-render projectfoo/scene24/take13.lxo --start 1 --stop 101

If you wanted to render every 3rd frame, you can do that too:

$ buildcat modo-render ProjectFoo/Scene24/Take13.lxo --start 1 --stop 101 --step 3

After starting the render, keep an eye on the contents of BUILDCAT ROOT, and you will see rendered frames begin to appear.

Monitoring

Buildcat is based on RQ, which already has a variety of builtin and third party monitoring tools. You can read about them in detail at http://python-rq.org/docs/monitoring, but in brief, you can see a quick overview of your farm from the command line with:

$ rq info

… which will print information about your current queues, jobs, and workers. To see continuously-updating information, specify a poll interval in seconds:

$ rq info --interval 5

For a more sophisticated, web-based interface to your farm, try RQ dashboard, which provides continuously-updated information, plus the ability to delete and reschedule jobs.

Finally, the buildcat tool provides a command to estimate how long it might take for a collection of jobs to finish:

$ buildcat eta

… the eta command will monitor the number of jobs submitted to your server over time; as long as the number of jobs is decreasing, it will estimate when the number of jobs will become zero.

Multiple Worker Setup

This section covers the additional details you’ll need to setup a real Buildcat render farm on multiple machines. If you haven’t already, we strongly recommend that you create a single-machine farm using the Tutorial documentation before proceeding.

Shared Storage

You’ll need to ensure that your shared storage is accessible to every worker that will be running jobs, and every client that will be submitting jobs. To do so, you’ll have to enable network file sharing for the shared storage directory on your platform. After doing so, we recommend that you manually verify that all of your worker and client hosts can access the shared storage before proceeding.

Note

Keep in mind that BUILDCAT_ROOT will vary from host-to-host depending on the platform and whether it’s being accessed locally or across the network. For example, on a Mac named Aurora hosting shared storage on an external drive named Farm, BUILDCAT_ROOT would be /Volumes/Farm. On the other Mac and Linux hosts on the network, BUILDCAT_ROOT would be //Aurora/Farm. On Windows hosts, it would be \\Aurora\Farm.

Network Communication

All of your workers and clients will need to communicate with the server, so you’ll need to identify a public network address on the server host that they can access. For example, instead of 127.0.0.1, which can only be accessed by processes running on the local host, you’ll need the address of an ethernet or wireless interface on the machine where the server is running. You should verify that all of your workers and clients can ping the server at that address.

We’ll use the address 192.168.2.1 for our server throughout the rest of this section.

Server

When we ran the Buildcat server in Tutorial, it defaulted to the loopback address 127.0.0.1, which meant that it would only accept connections from workers and clients running on the same host. Now, we need the server to accept connections from other machines, using a public server address. To do so, just specify the server’s network address at startup:

$ buildcat server --bind 192.168.2.1

Now, the server will be listening at the given address, allowing workers and clients to connect from other machines.

Workers

Now that we’ve changed the address that the server is listening on, we have to tell workers to contact it at that address (and we have to point them to the correct BUILDCAT_ROOT directory for their platform):

$ cd <local BUILDCAT_ROOT>
$ buildcat worker --host 192.168.2.1

There is no secret to running multiple workers for your farm - simply start as many workers as you need, on multiple hosts or even the same host, pointing them to the server’s address.

Clients

Similarly, clients will need to specify the server address to submit jobs:

$ buildcat worker-info --host 192.168.2.1

Keep in mind that, now that we’re running multiple workers, the job may be run on any one of them, so that the information returned by the worker-info job in this example will change depending on the host and worker where it was handled.

Redshift

To integrate Buildcat with Redshift, we provide a pair of integrations that can return information about a worker’s local Redshift installation, and render an image from a Redshift .rs file.

Installation

To use the Redshift integrations, each worker must be able to run the redshiftCmdLine executable provided by Redshift. Typically, this means that you will have to install Redshift on the worker host, locate the redshiftCmdLine (or redshiftCmdLine.exe on Windows) executable, and add that location to the PATH environment variable before starting the worker. See the Redshift Documentation for details on where the redshiftCmdLine executable is located on specific platforms.

Production

To query for information about a worker’s Redshift installation from the command line, use the buildcat command:

$ buildcat redshift-info
{'os': {'host': 'tim-aurora',
        'machine': 'x86_64',
        'processor': 'x86_64',
        'release': '4.4.0-19041-Microsoft',
        'system': 'Linux',
        'version': '#488-Microsoft Mon Sep 01 13:43:00 PST 2020'},
 'python': {'prefix': '/home/tshead/miniconda3',
            'version': '3.8.5 (default, Sep  4 2020, 07:30:14) \n[GCC 7.3.0]'},
 'redshift': {'executable': 'redshiftCmdLine.exe',
              'version': b'Redshift Command-Line Renderer (version 3.0.45 - API'
                         b': 3027) Copyright 2021 Redshift Rendering Technologi'
                         b'es '},
 'worker': {'pid': 302,
            'root': '/home/tshead',
            'user': 'tshead',
            'version': '0.4.0-dev'}}

Note that redshift-info will block until the results are received, and that the results could vary depending on which worker handled the request.

To submit a Redshift scene for rendering save the scene file and all of the scene assets to BUILDCAT_ROOT, and ensure that all scene assets can be accessed with relative paths.

Then you can use the buildcat command to render an image:

$ buildcat redshift-render path/to/scene.rs

Note that the path to the scene file must be relative to BUILDCAT_ROOT … so if your BUILDCAT_ROOT is on a network disk //Aurora/Farm and your scene is stored as //Aurora/Farm/ProjectFoo/Scene24/Take13.rs, your buildcat command line would be:

$ buildcat redshift-render ProjectFoo/Scene24/Take13.rs

After starting the render, keep an eye on the contents of BUILDCAT ROOT, and you should see the rendered frame appear.

Command Reference

Most of Buildcat’s functionality can be accessed at the command line, using the following:

buildcat

Command line client for Buildcat: the portable, lightweight render farm.

usage: buildcat [-h] [--debug]
                {certificate-info,client-keygen,client-tunnel,eta,houdini-info,houdini-render-ifd,houdini-render-hip,modo-info,modo-render,redshift-info,redshift-render,server,server-keygen,server-tunnel,worker,workers,worker-info,version}
                ...

Named Arguments

--debug

Verbose logging output.

Default: False

commands (choose one)

command

Possible choices: certificate-info, client-keygen, client-tunnel, eta, houdini-info, houdini-render-ifd, houdini-render-hip, modo-info, modo-render, redshift-info, redshift-render, server, server-keygen, server-tunnel, worker, workers, worker-info, version

Sub-commands

certificate-info

Display information about a TLS certificate.

buildcat certificate-info [-h] path

Positional Arguments

path

Path to the certificate file.

client-keygen

Generate client key and certificate for TLS encryption.

buildcat client-keygen [-h] [--country COUNTRY] [--days DAYS] [--email EMAIL]
                       [--locality LOCALITY] [--name NAME]
                       [--organization ORGANIZATION] [--state STATE]
                       [--unit UNIT]

Named Arguments

--country

Certificate country. Default: “US”

Default: “US”

--days

Length of time the certificate will be valid. Default: 365

Default: 365

--email

Certificate email. Default: None

--locality

Certificate locality. Default: “Albuquerque”

Default: “Albuquerque”

--name

Client name. Default: “*”

Default: “*”

--organization

Certificate organization. Default: “Buildcat”

Default: “Buildcat”

--state

Certificate state. Default: “New Mexico”

Default: “New Mexico”

--unit

Certificate organizational unit. Default: None

client-tunnel

Create a forwarding TLS tunnel for Buildcat workers and clients.

buildcat client-tunnel [-h] [--host HOST] [--identity IDENTITY] [--peer PEER]
                       [--port PORT] [--tunnel-port TUNNEL_PORT]

Named Arguments

--host

Server address. Default: “127.0.0.1”

Default: “127.0.0.1”

--identity

Client private key and certificate. Default: “/home/docs/.buildcat/client.pem”

Default: “/home/docs/.buildcat/client.pem”

--peer

Server certificate. Default: “/home/docs/.buildcat/server.cert”

Default: “/home/docs/.buildcat/server.cert”

--port

Server port. Default: 4443

Default: 4443

--tunnel-port

Listening port. Default: 6379

Default: 6379

eta

Estimate when a queue will empty.

buildcat eta [-h] [--count COUNT] [--host HOST] [--interval INTERVAL]
             [--port PORT] [--queue QUEUE]

Named Arguments

--count

Number of intervals to retain for the moving average. Default: 60

Default: 60

--host

Server address. Default: “127.0.0.1”

Default: “127.0.0.1”

--interval

Update interval in seconds. Default: 5

Default: 5

--port

Server port. Default: 6379

Default: 6379

--queue

Server queue to ping. Default: “default”

Default: “default”

houdini-info

Retrieve Houdini information from a worker.

buildcat houdini-info [-h] [--host HOST] [--port PORT] [--queue QUEUE]

Named Arguments

--host

Server address. Default: “127.0.0.1”

Default: “127.0.0.1”

--port

Server port. Default: 6379

Default: 6379

--queue

Server queue to ping. Default: “default”

Default: “default”

houdini-render-ifd

Submit Houdini .ifd render jobs.

buildcat houdini-render-ifd [-h] [--host HOST] [--port PORT] [--queue QUEUE]
                            [--timeout TIMEOUT]
                            ifdfile [ifdfile ...]

Positional Arguments

ifdfile

Houdini .ifd file(s) to render with mantra.

Named Arguments

--host

Server address. Default: “127.0.0.1”

Default: “127.0.0.1”

--port

Server port. Default: 6379

Default: 6379

--queue

server queue to use for rendering. default: “default”

Default: “default”

--timeout

Job timeout in seconds. default: 900

Default: 900

houdini-render-hip

Submit a Houdini .hip render job.

buildcat houdini-render-hip [-h] [--frames FRAMES] [--host HOST] [--port PORT]
                            [--queue QUEUE] [--rop ROP] [--timeout TIMEOUT]
                            hipfile

Positional Arguments

hipfile

Houdini .hip file to render.

Named Arguments

--frames

Frame(s) to render. Default: “1”

Default: “1”

--host

Server address. Default: “127.0.0.1”

Default: “127.0.0.1”

--port

Server port. Default: 6379

Default: 6379

--queue

Server queue to use for rendering. Default: “default”

Default: “default”

--rop

ROP to use for rendering. Default: “/out/mantra_ipr”

Default: “/out/mantra_ipr”

--timeout

Job timeout in seconds. default: 900

Default: 900

modo-info

Retrieve Modo information from a worker.

buildcat modo-info [-h] [--host HOST] [--port PORT] [--queue QUEUE]

Named Arguments

--host

Server address. Default: “127.0.0.1”

Default: “127.0.0.1”

--port

Server port. Default: 6379

Default: 6379

--queue

Server queue to ping. Default: “default”

Default: “default”

modo-render

Submit a Modo render job.

buildcat modo-render [-h] [--host HOST] [--port PORT] [--queue QUEUE]
                     [--start START] [--step STEP] [--stop STOP]
                     [--timeout TIMEOUT]
                     lxofile

Positional Arguments

lxofile

Modo .lxo file to render.

Named Arguments

--host

Server address. Default: “127.0.0.1”

Default: “127.0.0.1”

--port

Server port. Default: 6379

Default: 6379

--queue

Server queue to ping. Default: “default”

Default: “default”

--start

First frame to render. Default: 0

Default: 0

--step

Interval between rendered frames. Default: 1.

Default: 1

--stop

One past the last frame to render. Default: 1

Default: 1

--timeout

Job timeout in seconds. default: 900

Default: 900

redshift-info

Retrieve Redshift information from a worker.

buildcat redshift-info [-h] [--host HOST] [--port PORT] [--queue QUEUE]

Named Arguments

--host

Server address. Default: “127.0.0.1”

Default: “127.0.0.1”

--port

Server port. Default: 6379

Default: 6379

--queue

Server queue to ping. Default: “default”

Default: “default”

redshift-render

Submit a Redshift render job.

buildcat redshift-render [-h] [--host HOST] [--port PORT] [--queue QUEUE]
                         [--timeout TIMEOUT]
                         rsfile

Positional Arguments

rsfile

Redshift .rs file to render.

Named Arguments

--host

Server address. Default: “127.0.0.1”

Default: “127.0.0.1”

--port

Server port. Default: 6379

Default: 6379

--queue

Server queue to use for rendering. Default: “default”

Default: “default”

--timeout

Job timeout in seconds. default: 900

Default: 900

server

Start a Buildcat server.

buildcat server [-h] [--bind BIND] [--port PORT] [--storage STORAGE]

Named Arguments

--bind

Server address. Default: “127.0.0.1”

Default: “127.0.0.1”

--port

Server port. Default: 6379

Default: 6379

--storage

Persistent storage location. Default: “buildcat.aof”

Default: “buildcat.aof”

server-keygen

Generate server key and certificate for TLS encryption.

buildcat server-keygen [-h] [--country COUNTRY] [--days DAYS] [--email EMAIL]
                       [--locality LOCALITY] [--name NAME]
                       [--organization ORGANIZATION] [--state STATE]
                       [--unit UNIT]

Named Arguments

--country

Certificate country. Default: “US”

Default: “US”

--days

Length of time the certificate will be valid. Default: 365

Default: 365

--email

Certificate email. Default: None

--locality

Certificate locality. Default: “Albuquerque”

Default: “Albuquerque”

--name

Server name. Default: “127.0.0.1”

Default: “127.0.0.1”

--organization

Certificate organization. Default: “Buildcat”

Default: “Buildcat”

--state

Certificate state. Default: “New Mexico”

Default: “New Mexico”

--unit

Certificate organizational unit. Default: None

server-tunnel

Create a listening TLS tunnel for the Buildcat server.

buildcat server-tunnel [-h] [--identity IDENTITY] [--peers PEERS]
                       [--port PORT] [--tunnel-port TUNNEL_PORT]

Named Arguments

--identity

Server private key and certificate. Default: “/home/docs/.buildcat/server.pem”

Default: “/home/docs/.buildcat/server.pem”

--peers

Allowed client certificates. Default: “/home/docs/.buildcat/client.cert”

Default: “/home/docs/.buildcat/client.cert”

--port

Server port. Default: 6379

Default: 6379

--tunnel-port

Listening port. Default: 4443

Default: 4443

worker

Start a Buildcat worker.

buildcat worker [-h] [--host HOST] [--max-jobs MAX_JOBS] [--no-fork]
                [--port PORT] [--redshift-gpu REDSHIFT_GPU]
                [queues [queues ...]]

Positional Arguments

queues

Server queues to handle. Default: [‘default’]

Default: [‘default’]

Named Arguments

--host

Server address. Default: “127.0.0.1”

Default: “127.0.0.1”

--max-jobs

Maximum number of jobs to execute. Default: unlimited

--no-fork

Use a non-forking worker for collecting code coverage.

Default: False

--port

Server port. Default: 6379

Default: 6379

--redshift-gpu

Specify GPU indices to use for Redshift rendering. Default: use all GPUs.

workers

Start multiple Buildcat workers.

buildcat workers [-h] [--count COUNT] [--host HOST] [--max-jobs MAX_JOBS]
                 [--no-fork] [--port PORT] [--redshift-gpu REDSHIFT_GPU]
                 [--session-name SESSION_NAME]
                 [queues [queues ...]]

Positional Arguments

queues

Server queues to handle. Default: [‘default’]

Default: [‘default’]

Named Arguments

--count, -n

Number of workers to start. Default: 1

Default: 1

--host

Server address. Default: “127.0.0.1”

Default: “127.0.0.1”

--max-jobs

Maximum number of jobs to execute. Default: unlimited

--no-fork

Use non-forking workers for collecting code coverage.

Default: False

--port

Server port. Default: 6379

Default: 6379

--redshift-gpu

Specify GPU indices to use for Redshift rendering. Default: use all GPUs.

--session-name, -s

tmux session name. Default: “buildcat-workers”

Default: “buildcat-workers”

worker-info

Retrieve information about a worker.

buildcat worker-info [-h] [--host HOST] [--port PORT] [--queue QUEUE]

Named Arguments

--host

Server address. Default: “127.0.0.1”

Default: “127.0.0.1”

--port

Server port. Default: 6379

Default: 6379

--queue

Server queue. Default: “default”

Default: “default”

version

Print the version of this client.

buildcat version [-h]

Development

Getting Started

If you haven’t already, you’ll want to get familiar with the Buildcat repository at http://github.com/shead-custom-design/buildcat … there, you’ll find the Buildcat sources, issue tracker, discussions, and wiki.

Next, you’ll need to install all of the extra dependencies needed for Buildcat development:

$ pip install buildcat[all]

Then, you’ll be ready to obtain Buildcat’s source code and install it using “editable mode”. Editable mode is a feature provided by pip that links the Buildcat source code into the install directory instead of copying it … that way you can edit the source code in your git sandbox, and you don’t have to keep re-installing it to test your changes:

$ git clone https://github.com/shead-custom-design/buildcat.git
$ cd buildcat
$ pip install --editable .

Versioning

Buildcat version numbers follow the Semantic Versioning standard.

Coding Style

The Buildcat source code follows the PEP-8 Style Guide for Python Code.

Running Regression Tests

To run the Buildcat test suite, simply run regression.py from the top-level source directory:

$ cd buildcat
$ python regression.py

The tests will run, providing feedback on successes / failures.

Test Coverage

When you run the test suite with regression.py, it also automatically generates code coverage statistics. To see the coverage results, open buildcat/.cover/index.html in a web browser.

Building the Documentation

To build the documentation, run:

$ cd buildcat/docs
$ make html

Once the documentation is built, you can view it by opening buildcat/docs/_build/html/index.html in a web browser.

API Reference

Contents:

buildcat module

Provides the Buildcat public API, for use by clients and integrations.

exception buildcat.Error(message, description)

Bases: Exception

Base class for all Buildcat exceptions.

Parameters:

• message (str required) – Short message describing the failure.

• description (str required) – Detailed description of the failure, including possible remediations.

class buildcat.Serializer

Bases: object

RQ serializer that uses Python pickle version 2.

We use this serializer with workers / queues, so Python 2 clients can be used with Python 3 workers.

dumps(*, protocol=2, fix_imports=True)

Return the pickled representation of the object as a bytes object.

The optional protocol argument tells the pickler to use the given protocol; supported protocols are 0, 1, 2, 3 and 4. The default protocol is 3; a backward-incompatible protocol designed for Python 3.

Specifying a negative protocol version selects the highest protocol version supported. The higher the protocol used, the more recent the version of Python needed to read the pickle produced.

If fix_imports is True and protocol is less than 3, pickle will try to map the new Python 3 names to the old module names used in Python 2, so that the pickle data stream is readable with Python 2.

loads(*, fix_imports=True, encoding='ASCII', errors='strict')

Read and return an object from the given pickle data.

The protocol version of the pickle is detected automatically, so no protocol argument is needed. Bytes past the pickled object’s representation are ignored.

Optional keyword arguments are fix_imports, encoding and errors, which are used to control compatibility support for pickle stream generated by Python 2. If fix_imports is True, pickle will try to map the old Python 2 names to the new names used in Python 3. The encoding and errors tell pickle how to decode 8-bit string instances pickled by Python 2; these default to ‘ASCII’ and ‘strict’, respectively. The encoding can be ‘bytes’ to read these 8-bit string instances as bytes objects.

buildcat.check_call(command)

Run a command using subprocess.check_call().

buildcat.check_output(command)

Run a command using subprocess.check_output().

buildcat.connect(*, host='127.0.0.1', port=6379, timeout=None)

Connect to a listening Buildcat server.

Parameters:

• host (str optional) – IP address or hostname of the Buildcat server. Defaults to the local loopback adapter.

• port (int, optional) – Port number of the Buildcat server. Defaults to 6379.

• timeout (number, optional) – Maximum time to spend waiting for a connection, in seconds. Default: never timeout.

Returns:

connection – Persistent connection to the listening server.

Return type:

redis.Redis instance

Raises:

Error – If there are any problems connecting to the server.

buildcat.executable(name)

Return the platform-specific name of an executable.

Parameters:

name (str, required) – Name of the executable.

Returns:

name – The executable name, with platform-specific additions (such as .exe on Windows).

Return type:

str

buildcat.info()

Returns information about the current process.

The result is intended for use by integrations that return worker information.

Returns:

metadata – A collection of key-value pairs containing information describing the current process.

Return type:

dict

buildcat.queue(*, queue='default', host='127.0.0.1', port=6379, timeout=5)

Connect to a Buildcat server queue.

Parameters:

• queue (str, optional) – Name of the queue to connect.

• host (str, optional) – IP address or hostname of the Buildcat server. Defaults to the local loopback adapter.

• port (int, optional) – Port number of the Buildcat server. Defaults to 6379.

• timeout (number, optional) – Maximum time to spend waiting for a connection, in seconds. Default: 5 seconds.

Returns:

• connection (redis.Redis instance) – Persistent connection to the listening server.

• queue (rq.Queue instance) – Queue object.

Raises:

Error – If there are any problems connecting to the server.

buildcat.require_relative_path(path, description)

Raise an exception if a path isn’t relative.

Parameters:

• path (str, required) – Path to check.

• description (str, required) – Description of the path, used for raised exceptions.

Returns:

path – The path.

Return type:

str instance

Raises:

Error – If the path isn’t relative.

buildcat.root()

Return the worker root directory.

Returns:

root – The worker’s root directory.

Return type:

str

buildcat.submit(queue, command, *args, **kwargs)

Warning

function ‘buildcat.submit’ undocumented

buildcat.wait(*, connection, job)

Warning

function ‘buildcat.wait’ undocumented

buildcat.cli module

Warning

module ‘buildcat.cli’ undocumented

class buildcat.cli.FramesAction(option_strings, dest, nargs=None, **kwargs)

Bases: Action

Warning

class ‘buildcat.cli.FramesAction’ undocumented

buildcat.cli.main()

Warning

function ‘buildcat.cli.main’ undocumented

buildcat.hou module

Integration with SideFX Houdini, https://sidefx.com.

buildcat.hou.info()

Return information describing the worker’s local Houdini installation.

Environment Variables:

PATH (required) – Your PATH environment variable must be configured so that the worker can run the hython executable.

Returns:

metadata – A collection of key-value pairs containing a description of the Houdini installation on the machine where the job was run.

Return type:

dict

buildcat.hou.render_hip(hipfile, rop, frames)

Render a range of frames from a Houdini .hip file.

Environment Variables:

PATH (required) – Your PATH environment variable must be configured so that the worker can run the hython executable.

Parameters:

• hipfile (str, required) – Path to the file to be rendered.

• rop (str, required) – Absolute path of the ROP node to use for rendering.

• frames (sequence of tuple of three integers, required) – Contains one-to-many half-open (start, stop, step) ranges of frames to be rendered.

buildcat.hou.render_ifd(ifdfile)

Render a Houdini .ifd file.

Environment Variables:

PATH (required) – Your PATH environment variable must be configured so that the worker can run the mantra executable.

Parameters:

ifdfile (str, required) – Path to the file to be rendered.

buildcat.modo module

Integration with Modo, https://www.foundry.com/products/modo.

buildcat.modo.info()

Return information describing the worker’s local Modo installation.

Environment Variables:

PATH (required) – Your PATH environment variable must be configured so that the worker can run the modo_cl executable.

Returns:

metadata – A collection of key-value pairs containing a description of the Modo installation on the machine where the job was run.

Return type:

dict

buildcat.modo.render_frames(lxofile, frames)

Render a half-open range of frames from a Modo .lxo file.

Environment Variables:

PATH (required) – Your PATH environment variable must be configured so that the worker can run the modo_cl executable.

Parameters:

• lxofile (str, required) – Path to the file to be rendered.

• frames (tuple of three integers, required) – Contains the half-open (start, stop, step) range of frames to be rendered.

buildcat.modo.split_frames(lxofile, frames)

Render a range of frames from a Modo .lxo file as individual jobs.

Parameters:

• lxofile (str, required) – Path to the file to be rendered.

• frames (tuple of three integers, required) – Contains the half-open (start, end, step) of frames to be rendered.

buildcat.redshift module

Integration with Redshift, https://redshift3d.com.

buildcat.redshift.info()

Return information describing the worker’s local Redshift installation.

Environment Variables:

PATH (required) – Your PATH environment variable must be configured so that the worker can run the redshiftCmdLine executable.

Returns:

metadata – A collection of key-value pairs containing a description of the Redshift installation on the machine where the job was run.

Return type:

dict

buildcat.redshift.render(rsfile)

Render a Redshift archive (.rs) file.

Parameters:

rsfile (str, required) – Relative path of the file to be rendered.

Environment Variables:

• BUILDCAT_REDSHIFT_GPU (optional) – Whitespace-delimited list of GPU indices to use for rendering. Default: use all GPUs.

• PATH (required) – Your PATH environment variable must be configured so that the worker can run the redshiftCmdLine executable.

buildcat.worker module

Functionality for retrieving information about workers.

buildcat.worker.info()

Returns information about a worker.

Useful for testing that the system is functioning.

Returns:

metadata – A collection of key-value pairs containing information describing the local worker.

Return type:

dict

buildcat.worker.logtree()

Warning

function ‘buildcat.worker.logtree’ undocumented

Compatibility

A quick disclaimer on backwards-compatibility for Buildcat users:

Buildcat follows the Semantic Versioning standard for assigning version numbers in a way that has specific meaning. As of this writing Buildcat releases are still in the 0.y.z development phase, which means (among other things) that the API may change at any time. We try not to be abusive about it, but you should be prepared for the occasional bump on the road to the 1.0 release.

Release Notes

Buildcat 0.5.0 - December 9th, 2022

• Added certificate-info command to show certificate file details.

• Added workers command to start more than one worker at a time.

• Updated houdini-render-ifd command can submit multiple jobs at once.

• Set a reasonable default timeout for the houdini-render-ifd command.

• Added timeout arguments for houdini-render-hip, modo-render, and redshift-render commands.

• Allow newer Redis versions.

• Switched to pyproject.toml and flit for packaging.

Buildcat 0.4.1 - October 22nd, 2021

• Updated the way we collect code coverage data.

• Switched from Zulip to Github Discussions for support.

Buildcat 0.4.0 - October 14th, 2021

• Removed DCC-specific files from the install.

• Improved consistency among subcommand arguments.

• Added subcommands to secure communication channels with TLS.

• Allow clients to contact the server using a nonstandard port.

• Organized and streamlined the documentation.

• Switched from Travis-CI to Github Actions for continuous integration.

• Buildcat server can serialize tasks to an arbitrary storage path.

• Added an option to run workers without forking, for Windows.

• Added an integration to render Houdini .ifd files.

• Added support for rendering arbitrary collections of frames and frame ranges from a Houdini .hip file.

• Added an option to specify the maximum number of jobs executed by a worker.

Buildcat 0.3.0 - March 30th, 2021

• Create a single command-line client with subcommands for starting the server, starting workers, and submitting jobs.

• Added integrations for Modo and Redshift.

• Removed support for Python 2 in the Buildcat module.

• Removed support for Windows, use WSL instead.

• Eliminated the need for custom worker code, so we can use vanilla RQ workers instead.

• Completely reorganized and harmonized behavior among integrations.

• Added a command to estimate when a queue will be empty.

• Explicitly use pickle protocol 2 for serialization, so (non-Buildcat) Python 2 clients can submit jobs.

Buildcat 0.2.0 - July 8th, 2019

• Greatly expanded documentation.

• Introduced support for multiple render queues in Houdini.

• Added support for rendering multiple frames at a time from DOP ROPs in Houdini.

• Improved error checking and troubleshooting messages in Houdini.

• Support rendering with step sizes other than one in Houdini.

Buildcat 0.1.0 - March 20th, 2019

• Initial release, including basic Houdini integration.

Buildcat 0.0.1 - May 18th, 2018

• Setting up the project.

Support

The Buildcat documentation:

• http://buildcat.readthedocs.io

Visit our GitHub repository for access to source code, test results, issue tracker, and the wiki:

• http://github.com/shead-custom-design/buildcat

Our coverage statistics are updated automatically when modifications are committed:

• https://coveralls.io/r/shead-custom-design/buildcat

For Buildcat questions, comments, or suggestions, get in touch with the team using chat at:

• https://github.com/shead-custom-design/buildcat/discussions

Otherwise, you can contact Tim directly:

• Timothy M. Shead - tim@shead-custom-design

Indices and tables

• Index

• Module Index

• Search Page