New to Shotgun or Toolkit? Check out Our Story, learn about Our Product, or get a Quick Toolkit Overview. If you have specific questions, please visit our Support Page!
Hi everyone! We are Jeff and Josh, the new guys on the Toolkit team, and we’d like to welcome you to part one of a ten part, weekly series where we discuss our experiences building a simple pipeline using Toolkit. Each week we’re going to walk you through our thought process and how we approach some of the common aspects of building a pipeline using the framework. We will tell you what bits of Toolkit worked well for us, what roadblocks we hit, and how we think things might be improved.
One thing we recognize is that many of you have been down this road before. You’ve built a pipeline with Toolkit, and you’ve come up with some really cool solutions at your studio. That’s why we don’t want these posts to just be a lot of us telling you what we’re doing. We want to know how and why you built your pipeline the way you did. Any input you (the Toolkit veterans) can provide will be tremendous insight for us. We'll do our absolute best to learn from what you all have to say in the comments each week and apply what we hear to this little pipeline we're putting together. We’re brand new to Toolkit, so tell us what we’re doing wrong. The more we know about how you all work, the kinds of solutions you’ve built, and some of the Toolkit best practices, the better we’ll be able to support you in the long run as members of the Toolkit team.
On the flip side, if you’re someone who is considering using Toolkit in your facility and you have questions about how certain pieces fit together, let us know that as well. If we can help answer those questions or address any concerns you may have, we’d love to do so. There’s a good chance that other readers have been, or are, in a similar situation, so please be vocal in the comments and we can get a conversation going.
We do expect that if you’re reading this you have some experience with Shotgun, or at least a general understanding of the role it serves in production. Throughout these posts we’ll do our best to link to pre-existing Shotgun or Toolkit documentation that we’ve used or that we think might be relevant to the discussion. Along those lines, here are some links that we found useful as we were getting up to speed.
Getting Started with Shotgun
Shotgun Community Forums
Toolkit Example Videos
Other technologies we’ll be talking about and using during this series include Python, Github, Maya, Nuke, and Alembic.
Still with us? Good! Let’s get to work!
The first thing on our to-do list was to map out how we wanted our pipeline to work. We both came from the visual effects side of production and have had success in the past building subscription-based, pull pipelines. That kind of workflow requires features that are outside of the scope of Toolkit as it exists out of the box, but we would like to explore some of those ideas in the coming weeks. With that in mind, our initial goal has been to implement the broad strokes of our pipeline in such a way that we have a solid foundation upon which to build more complex features in the coming weeks.
We also wanted to minimize the number of content-creation packages we would be dealing with. So far we’re using Maya for everything except for compositing where we’re using Nuke. The reason for minimizing the number of DCCs is that we wanted to put the focus on learning and exploring Toolkit, not the content-creation software. We realize this isn’t a luxury many of you have in the real world, but the focus here is on Pipeline. And while the concepts and features we implement will often be DCC-specific, they should translate well, from a design standpoint, to other software packages.
|A very simple representation of the pipeline|
We decided to stick pretty closely to the default Shotgun configuration for assets and shots. On the asset side we wanted to pull the model into rigging (if needed) and surfacing. On the shot side, we wanted to pull either the model (prop) or rig into layout, which would then be pulled into animation. Lighting would pull an Alembic cache from animation and it would then render out frames to be pulled in by comp. Our shot camera would be generated in layout and pulled by each of animation, lighting, and compositing. Surfacing was a bit of a question mark as neither of us are Maya experts. We decided to see if we could get shader networks pulled into lighting (not pictured) and attached to the Alembic caches as a first pass. Another requirement was that we needed to be able to generate reviewable material at each step in the pipeline.
The first step in making this all work was to get Shotgun and Toolkit up and running. We got our Shotgun site set up and started customizing it based on our pipeline design. Jeff set up a couple of assets and a shot and once we had the tasks created for each, we were ready to get rolling with Toolkit.
It should be mentioned that we are in different physical locations, on opposite sides of the U.S. In the coming weeks we might formally explore how our pipeline would work with users in different locations, but for now we’re simulating a shared file system using an auto-synced folder via a free cloud-storage service. It’s not ideal, but suitable for our needs.
Getting toolkit set up for our project was fairly straight forward. There is some excellent documentation for installing and running Shotgun Desktop which walks you through setting up your project’s Toolkit configuration.
One small issue we hit during the process was getting locked out of the being able to define the primary storage location for our project.
|Locked paths in the Setup Wizard|
What we didn’t realize was that these paths are shared across all projects and had already been set in the Shotgun site preferences. The docs state this clearly enough (we got a little ahead of ourselves), but this particular page of the setup wizard was confusing. We’ve created an internal ticket for this one to add a little more polish to the UI to clarify why all the paths are locked and to not suggest you can enter new paths.
We ended up creating a github repo (which we’ll be opening up to you guys in the coming weeks) based off of the default configuration and pointing Desktop to that during the install. We knew we were going to be making many changes, so being able to track those in a git repo was a must. We noticed that after Desktop finished with the setup process, there was a working clone of our repo in the newly created configuration directory. This seemed really nice! The next thing we did was go to the Pipeline Configurations page in Shotgun and clone the Primary configuration so that we could each have our own sandbox to play in. As a bonus, the cloned configuration directories also retained the necessary git repo structure, so we were ready to start independent development.
We noticed that the cloned configurations you have access to in Shotgun show up as additional items in the action menus.
|Repeated configurations in the Action Menus|
It would be nice if you could customize the Shotgun action menus to some extent or just have cloned configurations show up as sub menus by default. We can see cases where some users, especially support folks, may clone configurations often which could make the menu really long. We created an internal ticket for this as well.
Next up was customizing the launchers for the content-creation software we’d make available to the users of our pipeline. This is typically managed within the pipeline configuration in these two files:
In the paths.yml file we entered the paths to the software we’re making available in the pipeline and cleaned out anything we weren’t using. In the app_launchers.yml file we configured instances of the tk-multi-launchapp with additional launch settings and again cleaned out the configurations for software we weren’t planning on using.
One thing that might be a bit confusing or misleading to folks is the use of the word app. In Toolkit, there’s the concept of an app which is a functional bit of a workflow that runs within one or more engines. The word app is also being used to refer to the content-creation packages. The app_launchers.yml file drives the launching of the content-creation software, but houses various configuration blocks for the tk-multi-launch Toolkit app. There’s another file in the includes directory called common_apps.yml which defines common configurations of Toolkit apps. Long term it might be nice to be more consistent with the terminology. Perhaps it would make sense for the file to be named dcc_launchers.yml and for the Toolkit app to be named tk-multi-launchdcc. What do you guys think?
Next we found the toggle in Desktop that allows you to set the pipeline configuration you want to use when launching content creation software.
|Set the configuration to use within Desktop|
This was really convenient, but it did make us realize that desktop only allows launching the content creation software within a project context. It would be nice to be able to set an asset or shot context via Desktop and launch Maya or Nuke within that context; maybe even on a workfile displayed in Desktop. From our internal discussions, it sounds like the Toolkit team anticipated this as well; see all the extra space next to Apps at the to of the Desktop UI? There didn't seem to be a ticket for this internally, so we created one.
Next week we’re going to talk quite a bit about the configuration system and how we think it might be improved. We’ll look at how we integrated our “studio” code with Toolkit and how that could be bundled and referenced with pipeline configurations.
We hope you all have a fantastic week!
- Jeff & Josh
About Jeff & Josh
Jeff was a Pipeline TD, Lead Pipeline TD, and Pipeline Supervisor at Rhythm and Hues Studios over a span of 9 years. After that, he spent 2+ years at Blur Studio as Pipeline Supervisor. Between R&H and Blur, he has experienced studios large and small, and a wide variety of both proprietary and third-party software. He also really enjoys writing about himself in the third person.
Josh followed the same career path as Jeff in the Pipeline department at R&H, going back to 2003. In 2010 he migrated to the Software group at R&H and helped develop the studio’s proprietary toolset. In 2014 he took a job as Senior Pipeline Engineer in the Digital Production Arts MFA program at Clemson University where he worked with students to develop an open source production pipeline framework.
Jeff & Josh joined the Toolkit team in August of 2015.