← Back to blog
pipelineproductionpython

Building a Production Pipeline That Doesn't Make You Want to Quit

Lessons learned from years of building and maintaining creative production pipelines — what works, what doesn't, and why simplicity wins.

There’s a particular kind of pain that comes from inheriting a production pipeline built by someone who left the company two years ago. No documentation, a folder called utils_FINAL_v3_USE_THIS, and a cron job that emails the CEO every time it fails at 3am.

I’ve been on both sides of this. Here’s what I’ve learned.

Start With the Boring Parts

Every pipeline I’ve seen fail started with someone building the exciting parts first — the fancy UI, the automated render submission, the Slack integration. Meanwhile, the file naming convention was “whatever felt right at the time.”

Start here instead:

  • Naming conventions — Agree on them. Write them down. Enforce them programmatically.
  • Directory structure — One source of truth. No “oh but for this project we did it differently.”
  • Configuration — Centralize it. If you have config in more than three places, you have config in zero places.

The 80/20 of Pipeline Tools

Most production problems can be solved with:

# Seriously, this covers more than you'd think
import os
import json
import subprocess
import logging

You don’t need a framework. You don’t need a plugin system. You need a function that reliably does one thing, with good error messages when it doesn’t.

Automation Should Be Boring

The best automation is the kind nobody notices. If your automated system requires a Slack channel dedicated to discussing why it broke, it’s not automated — it’s a distributed manual process with extra steps.

Rules I follow:

  1. If it fails, it should fail loudly and clearly. No silent failures. Ever.
  2. If it succeeds, it should be silent. Nobody needs a “SUCCESS” notification for routine operations.
  3. Idempotency is everything. Running it twice should produce the same result as running it once.

The Human Pipeline

The hardest part of any pipeline isn’t the code. It’s getting people to use it consistently. The best technical solution that nobody uses is worse than a mediocre solution everyone follows.

Build for the least technical person on your team. If they can’t figure it out in under two minutes, simplify it.