New to Shotgun or Toolkit? Check out Our Story, learn about Our Product, or get a Quick Toolkit Overview. If you have specific questions, please visit our Support Page!
Hi everyone! Josh here. Welcome back for part two of our series dedicated to building a simple pipeline using Toolkit.
Last week we gave a quick introduction of the project and talked about the early phases of design and set up. This week I want to go a little deeper into the setup and management of a Toolkit project and talk about the configuration system. Since I’ve been responsible for the pipeline configuration for our little pipeline, this will be a brain dump of my thoughts and experiences regarding the configs. I’d love to find out what you all have done as far as config customizations go and how you manage them on a daily basis across multiple projects and/or locations. We got some great feedback in the comments last week, so let’s keep the conversation going!
Here are the links I used to get up to speed with the configuration system and to better understand how it works within Toolkit:
Managing your Project Configuration
Also, if you are interested in following along on the development side, you can find our simple pipeline config repository here. We don’t recommend anyone actually use our config yet, but we’ll be referencing it throughout the series.
As we mentioned last week, getting the initial configuration installed and working for the pipeline was relatively painless. Using Shotgun Desktop was very convenient and we were able to launch DCCs in no time.
The next step was actually configuring the pipeline to behave the way we wanted. After reading the docs, I decided to just dive into the config directory structure, take a look at all the files, and try to get the “lay of the land,” so to speak.
In the core directory I found the templates.yml file and the schema directory tree, which are used to map Shotgun entities into the desired file-system structure. There was a lot to take in as far as the documentation for this portion of the config. To keep things simple for us, I cleaned out some of the references to DCCs we wouldn’t be using and left the rest of the structure largely unchanged.
The one thing I did add in the schema hierarchy was deferred folder creation files for the work area DCC directories. The default config does not include these files by default, so when you create the directories for an asset or shot on disk, you get many more directories than you likely need. Adding these little configuration files in there defers creation of the DCC directories until the engine is launched within that context. My preference would be to have this as the default behavior, but I’m curious what you guys think.
Deferred folder creation file
The other big chunk of the configuration structure is the env directory, which houses the various environment configs that drive what engines and apps are available for a given context. A core hook called pick_environment is used by Toolkit to map a context to one of these files (with the exception of the shotgun_*.yml files which are mapped by convention from within shotgun).
One of the things I’ve found really convenient about working with the environment configurations is the location setting available on each of the engine, app, and framework bundles. The location setting makes it really easy to pull the bundle from a local directory, a github repo, or the Toolkit app store. This makes development and testing changes to an environment configuration very convenient.
After browsing through these environment files some more, the thing that stuck out the most was the amount of repetition in the settings. Take look at the asset_step.yml and shot_step.yml files from the default config, for example.
Repetition in the config files
You’ll notice quite a bit of repetition in the engine configurations of each file. You’ll also see the tk-multi-* apps being used by several engines in each file, again with a very repetitive list of settings. After looking through these, I had a some thoughts about how to make managing these files a better experience.
My first thought was that I’d much prefer an inheritance scheme where I could define common settings in one file that I could include into an asset or shot environment. From there I could override the default settings with only the settings that differed for a given context. The config system already has an include mechanism, but it updates at the key level of the config dictionary rather than doing a merge with value-level overrides. So I decided to see if I could hack into the Toolkit core (which sounds really dangerous and awesome) to see if I could get the includes to do what I wanted. After fiddling around with it for a bit, I was able to get it working. You can see the changes in this branch of the tk-core github repo.
I didn't take the time to map the entire default config into a new organizational structure, but I did run some tests on a simple inheritance-based config, and it seems to work as expected. The changes allow you to define an engine and its apps with default values in one file, and then include that into an environment config that only overrides the settings that are unique to that context.
You can browse these modified environment configs in this branch of the tk-config-simple repository.
I think this type of organization could make environment configs easier to manage. For example, when you want a specialized environment for one sequence on your project or for an animation step on one shot, you wouldn’t need to duplicate an entire config.
I should also note that I haven’t fully tested these ideas, and I’m not exactly sure how bad this would break the tank command. It was an interesting experiment, and it helped me learn more about how the includes worked and what goes on inside of core. I’m curious if any of you have done similar experiments or if you have an environment config setup that works well for you.
Management UI and API
Having looked through the configs, and played around with their organization, it seems to me that the best long-term solution would be to not have to manage them by hand or via the tank command at all. It seems what Toolkit really needs is a slick configuration management interface. It would be neat to have features like drag and drop installation of engines/apps/frameworks from the app store, push-button configuration deployment across multiple projects or locations, and visual indications when apps/engines/frameworks are out of date, just to name a few.
I’d also like to think about an environment configuration API that wasn’t file-centric. From a conceptual standpoint, how the data is stored under-the-hood shouldn’t really matter and we should be interacting with an API rather than files on disk. I’m guessing that if there was such an API, and perhaps some modifications to the pick_environment hook to accept an environment object, some of you might build your own slick UIs or come up with clever ways to organize and store your environment configurations. Then again, maybe you’ve already done that. If so, I’d love to hear about it!
Taking things a bit farther, in the long term I’d really like it if we could all share the bundled configurations (and other pipeline-related tools, scripts, widgets, gizmos, etc) that we’re building in a pipeline marketplace. Shotgun has the Toolkit App Store, and Autodesk owns Creative Market; what if we could build a community around pipeline-related content? Imagine working in Maya and being able to install a toolkit app, try it out, comment on it, rate it, or even suggest improvements for it, all within a single session!
This seems like a good place to wrap up this week’s post. It was fun getting the config into a semi-working state with the stripped down engines and apps, and the deferred directory creation. Tinkering with the Toolkit core was also an interesting experiment that we may end up fully fleshing out in the coming weeks.
Right now, I’m still down in the guts of the config system, making changes and trying to learn how some of the more subtle bits and pieces work. Feel free to watch the tk-config-simple repo if you want to follow along with the development over the coming weeks.
I hope this week’s post gave you some insight or perhaps sparked an idea in your mind about how we might move forward. I’m excited to hear your thoughts on and experiences with Toolkit configurations and to learn how you manage them at your facility.
I know last week we teased our studio code integration, but that’ll have to wait for a future post. Next week, Jeff will be giving you the low down on his experiences with file publishing and how data flows down the pipe from one step to another.
Updates from Last Week
Hey gang, this is Jeff. Last week’s post elicited a handful of comments in the blog post itself, but also a bit of feedback in support ticket conversations that revolved around some of the same things that we are trying to achieve with this project.
In last week’s post, I think we could have provided a little more detail in the section concerning cloning a pipeline configuration to create a development environment. It is a very useful feature and one that we are using daily, but there is some information that we failed to mention. After cloning the primary pipeline configuration, things should be up and running. However, when I set this up for myself, I immediately removed the cloned config directory and created a config symlink to a clone of our tk-config-simple repo in my development area. This mostly worked, but there were autogenerated files that were part of the project’s primary config that were created on project setup.. These files had to be copied across from the primary config into the repo that I had linked to, and we added them to our .gitignore file since they shouldn’t be kept under revision control. This step is not immediately made obvious by the documentation, and so it could be a bit of a gotcha for new users.
Below is the list of ignored files that I copied across for reference. Hopefully it will help some of you get past that gotcha with a minimum amount of frustration.
Our .gitignore which includes the auto-generated files from the primary config
About Jeff & Josh
Jeff was a Pipeline TD, Lead Pipeline TD, and Pipeline Supervisor at Rhythm and Hues Studios over a span of 9 years. After that, he spent 2+ years at Blur Studio as Pipeline Supervisor. Between R&H and Blur, he has experienced studios large and small, and a wide variety of both proprietary and third-party software. He also really enjoys writing about himself in the third person.
Josh followed the same career path as Jeff in the Pipeline department at R&H, going back to 2003. In 2010 he migrated to the Software group at R&H and helped develop the studio’s proprietary toolset. In 2014 he took a job as Senior Pipeline Engineer in the Digital Production Arts MFA program at Clemson University where he worked with students to develop an open source production pipeline framework.
Jeff & Josh joined the Toolkit team in August of 2015.