MODULE 11 OF 12 · RELEASE 3

Production Reliability

Staying useful when something you depend on is slow, broken, or quietly wrong.

So what is this really about?

Something you depend on will fail. The worst kind of failure is not "down" — it is "slow", because everything piles up behind it while it pretends to be fine. Decide in advance what your system does when that happens, and rehearse it.

The words you will hear

Timeout

How long you will wait before giving up. Without one, you wait forever.

Retry

Trying again — but a limited number of times, and only for things that are safe to repeat.

Circuit breaker

After enough failures, stop calling for a while. Like not queuing at a till you can see is broken.

Degraded mode

The cash-only sign. Reduced service, honestly explained, rather than collapse.

By the end you will be able to

• Set limits on waiting, retrying and giving up.
• Decide what reduced service looks like, before you need it.
• Prove it by deliberately breaking things.

Going deeper

Slow is worse than down

This is the part that surprises people, so it is worth being concrete.

A dependency that is down fails in milliseconds. Your code gets an error immediately, handles it, and moves on. Throughput barely changes.

A dependency that takes forty-five seconds and then succeeds is far more destructive. Every request waits. Connections stay open. Your pool exhausts. Requests that have nothing to do with that dependency start queuing behind the ones that do — and now your entire system is unavailable because one service got slow.

This is why timeouts matter more than retries, and why most people add retries first.

Timeouts come from the budget, not a habit

A single global timeout is a guess applied everywhere. Derive it instead.

Start from what the caller can tolerate. A person waiting on a page: two seconds total. That budget is then divided among the calls needed to serve it — and each call gets a timeout that fits inside what remains.

The rule that follows is simple and frequently violated: an inner timeout must be shorter than the outer one. If your handler gives up after two seconds but the call inside it waits ten, the timeout on the inner call does nothing — the request was abandoned eight seconds earlier while the connection stayed open.

Retries, and the harm they can do

Retrying is the obvious response to failure and it can make an outage worse.

A struggling service that receives three times its normal traffic because everyone is retrying will not recover. You have converted a degradation into an outage, with the best intentions.

Three rules keep retries helpful. Only retry what is safe to repeat — anything that changes state elsewhere needs an idempotency key first. Cap the attempts, because a call that has failed three times is unlikely to succeed on the fourth. And add jitter, so a thousand clients do not synchronise into a wave that arrives every second on the second.

The breaker stops you making it worse

When a dependency is clearly failing, continuing to call it wastes your resources and prolongs its recovery.

A circuit breaker watches the failure rate. Past a threshold it opens, and calls fail immediately without touching the network. After a pause it allows one request through — half-open — and if that succeeds it closes again.

The half-open state is the part that is easy to get wrong. Letting all traffic back at once will knock over a service that has only just stood up. One request, then a few, then normal.

Decide what degraded means

Every dependency needs an answer to: what do we do when this is unavailable? Decide it in advance, because deciding during an incident produces whatever occurs to someone at the time.

There are only a few honest options. Serve a cached answer and say it may be stale. Accept the work and queue it for later. Or refuse clearly — "this feature is temporarily unavailable, your data is safe" — which is a perfectly respectable answer and infinitely better than a spinner.

What is not acceptable is hanging, or failing in a way that loses work the user believed was saved.

Prove it by breaking things

All of the above is theory until you inject failure deliberately. Add latency. Return errors. Return half a response and close the connection. Make a dependency slow rather than absent, because that is the case everyone gets wrong.

Do it in tests so it runs every time. The first run is usually humbling — timeouts that were never reached, a breaker that never opened, a degraded path that throws. Better to find that in CI than at 2am.

What good looks like

A provider outage costs you one feature, not the system. Every external call has a timeout you can justify from a budget. And you have watched your own system survive a dependency getting slow, rather than assuming it would.

How it is put together

Every external call has a timeout shorter than the caller’s own budget. Retries are bounded, jittered, and applied only to idempotent operations. A circuit breaker stops calling a dependency that is clearly down. Each dependency has a declared degraded mode — a cached answer, a queued task, or an honest refusal.

Where you start

A system where every dependency is assumed available and no call has a timeout.

The lab

1. Add timeouts derived from the caller’s budget, not from a guessed constant.
2. Add bounded jittered retries, restricted to idempotent operations.
3. Add circuit breakers with a defined half-open recovery path.
4. Define and implement a degraded mode per dependency.
5. Inject failure — latency, errors, partial responses — and observe the result.

What goes wrong, on purpose

The scenario. The model provider becomes slow rather than unavailable: every call takes 45 seconds and eventually succeeds.

What should happen. Timeouts fire before the caller’s budget expires, the breaker opens under sustained slowness, and the degraded mode serves users rather than a queue of hanging requests.

How you prove it works

• A fault-injection suite covering latency, errors and partial responses.
• A test asserting the breaker opens and later recovers through half-open.
• A test asserting the degraded path returns something useful and honest.

What you walk away with

A system that stays responsive during a provider outage, with a rehearsed degraded mode and fault-injection tests in CI.