It worked on my machine. Then I opened a PR.

Last Tuesday the training run looked fine. Thursday it didn’t. Same repo. Same script. Slightly different numbers. I spent an afternoon blaming random seeds before I noticed someone (me) had upgraded a package “just to clear a warning.”

Environment drift is the silent killer of ML side projects. Also of some production ones, but those at least have people paid to notice.

What’s not in your repo

Your import list is the tip. Underneath:

• Python minor version
• CUDA / driver mess if GPUs are involved
• system libs that got installed once and never documented
• a default argument that changed two releases ago

None of that is in git. All of it changes results.

Dockerfile as the README that fails

I used to write setup steps in markdown. People (including future me) skipped half of them. A Dockerfile doesn’t skip. It dies mid-build and tells you which line.

FROM python:3.11-slim

WORKDIR /app

# deps first so editing train.py doesn't reinstall the world
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "train.py"]

That layer-order trick is small and saves hours. Cache is a feature. Use it.

Stuff I learned the hard way

Pin versions. pandas is a mood. pandas==2.2.3 is a fact. Same for torch, numpy, everything that touches numerics.

Slim images. Dropped hundreds of MB switching to -slim. I didn’t miss the extra junk.

Don’t bake data into the image. Mount datasets as volumes. Nobody wants to pull your 4GB “hello world” image.

.dockerignore. Without it I was shipping .git, venv/, and old checkpoints into every build. Builds got slow. I got sad. One file fixed it.

What actually changed

“Run the experiment” became one command that behaved the same on my laptop and on a random cloud box. Reproducibility stopped being a hope and started being a property of the setup.

That’s most of what I want from MLOps at this stage of my career: make the boring parts boring enough that I can spend time on the model instead of on “why is sklearn a different version here.”