MODULE 12 OF 12 · RELEASE 3
Deployment & Production Operations
Shipping changes calmly, and undoing them faster than you shipped them.
Theatres rehearse the fire drill when there is no fire. Not because they expect one, but because the middle of an emergency is the worst possible time to be reading instructions for the first time.
So what is this really about?
Releasing software is routine right up until it is not. The teams that stay calm are the ones who decided in advance what has to pass before a release goes out, and who have actually practised undoing one — including the database, which is the part everyone forgets.
QUICK CHECK
A release passes every check but breaks under real traffic. What matters most in the next ten minutes?
The words you will hear
- Pipeline
- The automatic checks a change must pass before it can go live.
- Gate
- A check with real authority to stop a release. A check that cannot block is decoration.
- Rollback
- Putting the previous version back, quickly and without losing data.
- Migration
- A change to the database shape. Additive ones let the old version keep working — which is what makes rollback possible.
By the end you will be able to
- Build checks that can genuinely stop a bad release.
- Keep secrets and settings out of your code.
- Practise undoing a release before you have to do it for real.
Going deeper
Gates that can actually stop a release
A check that reports a problem but cannot block anything is decoration. It gets ignored within a month, usually by someone in a hurry who is right that the change is urgent.
For an AI system, the pipeline has more to run than usual. Types and tests, as always. The evaluation suite from Module 8, so quality regressions block the same way broken tests do. The injection corpus from Module 10, so a prompt restructure cannot silently reopen an attack. And a secret scan, because credentials get pasted into code by accident and the only reliable fix is a machine that looks every time.
Any one failing stops the deploy. That is the whole point of calling it a gate.
Configuration and secrets
Configuration comes from the environment. Secrets come from a managed store. Neither is ever in the repository, including in a file you intend to delete before committing.
Two habits pay for themselves. Keep an example file listing every variable the application reads, with fake values — it is the only documentation of configuration that stays accurate, because the app breaks when it drifts. And fail loudly at startup if something required is missing, rather than discovering it at the first request that needs it.
Migrations decide whether rollback is possible
This is the part that turns a five-minute recovery into a bad evening.
If a deploy renames a column and you then roll back the code, the old version meets a schema it does not understand and fails — so rollback is no longer available, and the only way out is forward, under pressure.
Additive migrations avoid the trap. Add the new column, deploy code that writes both and reads the old, then deploy code that reads the new, and only much later remove the old one. More steps, and each step is individually reversible.
The rule: after any migration, the previous version of the application must still run. If it cannot, you have given up rollback, and you should know you are doing that rather than discover it.
Rehearse the rollback
Everyone has a rollback procedure. Far fewer have run it.
Do it on a normal afternoon. Deploy something harmless, roll it back, and time it. You will find something: a step that needs credentials nobody has, a cache that has to be cleared, a queue that has to drain. Every one of those is much cheaper to find now.
Write down the measured time. "Rollback takes about four minutes" is a fact you can plan around during an incident. "We can roll back" is a hope.
The runbook is four pages
Deploy. Roll back. Rotate a secret. Handle a provider outage.
Written plainly enough that someone who did not build the system can follow them at 3am — which, in a small team, is a realistic description of you on very little sleep.
What good looks like
Deploying is uneventful enough to do on a Friday. Every gate has genuinely blocked something at least once, which is how you know it works. Rollback has been rehearsed and timed. And the runbook was written before it was needed rather than during.
How it is put together
CI runs typecheck, tests, the evaluation baseline and the security corpus; any failure blocks the deploy. Configuration comes from the environment, secrets from a managed store, and neither is ever in the repository. Migrations are additive and deployed before the code that needs them, so the previous version keeps running and rollback stays possible.
Where you start
A project deployed by running a command on a laptop.
The lab
- Build the pipeline with every gate, including the evaluation and security suites.
- Move configuration to the environment and secrets to a managed store.
- Make migrations additive and deploy them ahead of the dependent code.
- Write and rehearse a rollback procedure, including the database.
- Write the runbook: deploy, roll back, rotate a secret, handle an outage.
What goes wrong, on purpose
The scenario. A deploy passes CI but causes errors in production that only appear under real traffic.
What should happen. Alerts fire within minutes, rollback restores the previous version without a destructive migration, and the runbook is followed rather than improvised.
How you prove it works
- A pipeline run proving each gate can block a release.
- A rehearsed rollback, timed, including the database step.
- A secret-scanning check over the repository history.
What you walk away with
A pipeline whose gates genuinely block, a rehearsed and timed rollback, and a runbook written before it was needed.