MODULE 02 OF 12 · RELEASE 1

Event-Driven Agent Infrastructure

Take the order first, cook it after — so a slow kitchen never turns customers away at the door.

So what is this really about?

Software that receives work from elsewhere should do the same. Write the order down straight away, say "got it", and do the slow part afterwards. If the slow part breaks, the order is still on the rail. The alternative — making the sender wait while you do everything — means a slow step turns into a lost order.

The words you will hear

Event

A message from somewhere else saying something happened. Your order ticket.

Ingress

The counter where orders are taken. Fast, and it only writes things down.

Worker

The kitchen. Picks tickets off the rail and does the actual work.

Idempotent

Handing in the same ticket twice still gets you one coffee, not two.

Dead letter

The tray for tickets that failed repeatedly, so someone can look at them instead of them vanishing.

By the end you will be able to

• Separate taking work in from doing the work.
• Make repeated deliveries harmless.
• Recover cleanly when the slow part fails after you already said "got it".

Going deeper

The mistake almost everyone makes first

You write an endpoint that receives an event. Inside it you validate the payload, call a model, write a record, send an email, and return a response. It works, and it is the obvious way to write it.

Then one of three things happens. The model call takes eleven seconds and the sender gives up at ten, so it retries — and now you have done the work twice. Or the email provider is down, your handler throws, and the enquiry that was about to be saved is gone. Or traffic doubles and every request is holding a connection open for eight seconds, so you run out of connections and start refusing work you could have accepted.

All three come from the same decision: doing slow, unreliable work while someone is waiting for you to say yes.

Two jobs, not one

Split the endpoint into two things that fail independently.

The first job is to accept. It checks the message is genuine and well formed, writes it down, and returns. It touches nothing slow and nothing that can be unavailable. It should finish in single-digit milliseconds, because all it does is a validation and one insert.

The second job is to process. It picks up what was written down and does the expensive part. It can be slow. It can fail. It can be restarted. None of that reaches the sender, because the sender already got its answer.

The two are joined by a row in a database, not by a function call. That is the whole trick: a function call means the second job failing takes the first one with it, and a row means it does not.

The state column is the design

Each recorded event carries a state, and the set of states you choose is the design of the whole system:

pending      just written down, nobody has picked it up
claimed      a worker has taken it, with a timestamp
done         finished successfully
failed       failed in a way worth retrying
dead         failed too many times, a human should look

Two details in there matter more than they look.

Claimed carries a timestamp, not just a flag. If a worker takes an event and then the process dies, a flag leaves that row claimed forever and the work is silently lost. A timestamp lets another worker say "this was claimed four minutes ago and nothing has happened, I will take it".

Dead is a real state, not an error log. Something that fails repeatedly should stop being retried and start being visible. A queue of things that failed permanently is a queue you can look at, fix, and replay. Errors scrolling past in a log are not.

Claiming without two workers colliding

If you run more than one worker — and you will — two of them can read the same pending row at the same moment and both start processing it. The fix is to make claiming a single atomic update rather than a read followed by a write:

UPDATE events
   SET state = 'claimed', claimed_at = now()
 WHERE id = (SELECT id FROM events
              WHERE state = 'pending'
              ORDER BY created_at
              LIMIT 1)
   AND state = 'pending'

The repeated state = 'pending' is the important part. Whoever runs the update first changes the row; the second worker’s update matches nothing and affects zero rows, so it knows it lost and moves on. Checking first and updating afterwards has a gap between the two where both can win.

Duplicates are normal, not exceptional

Every serious sender retries. If they do not get a clean response quickly, they send again — and they are right to, because the alternative is losing your data when your server has a bad second.

So duplicates are not an edge case to handle if you get time. They are routine traffic, and the system has to be built so they cost nothing.

The mechanism is boring and reliable: take the sender’s own identifier for the event, put a uniqueness constraint on it, and insert. If the insert succeeds you have a new event. If it violates the constraint you have seen this one before, and the correct response is to return success without doing anything, because the work either is done or is already queued.

Let the database do the rejecting. Application code that checks whether a row exists and then inserts has a window between the check and the insert, and under a retry storm that window is exactly where two requests land.

When processing fails halfway

Picture a worker that calls a model, then writes a result, then sends a notification. It dies between the model call and the write.

The event is still claimed, so nothing else has touched it. After the claim expires another worker picks it up and starts again — which means calling the model a second time. That costs money but does no damage, because nothing outside your system happened yet.

Now move the crash later: after the notification, before marking done. Reprocessing sends a second notification, and a customer gets two emails.

That is the real lesson of this module. Ordering decides what a crash costs. Put the irreversible, externally visible step as late as possible, and record that it happened before you do anything else — so a restart can see it and skip it.

What you should be able to do afterwards

Kill a worker mid-run and watch the system recover on its own. Deliver the same event three times and find one record and one action. Look at a list of permanently failed events, fix the cause, and replay them.

None of that is possible in the version where everything happens inside the request handler, which is why this is the module where the shape of the system actually changes.

How it is put together

An ingress endpoint verifies, validates and durably records an event, then returns. A separate worker picks the record up and does the expensive work. The two are joined by state in a database rather than by a function call, which is what lets the second one fail and retry without the first one knowing.

Where you start

The Free Foundations webhook, which does everything inline in the request handler.

The lab

1. Split acceptance from processing, with an explicit state column on the event record.
2. Add an idempotency key with a uniqueness constraint so duplicate delivery is a no-op.
3. Add a claim step so two workers cannot process the same record.
4. Add a dead-letter state for records that fail repeatedly, with the failure reason attached.
5. Add a replay path that reprocesses a dead-lettered record after a fix.

What goes wrong, on purpose

The scenario. The worker crashes mid-processing, after the model call and before the result is written.

What should happen. The record remains claimed but unfinished, is reclaimed after a visibility timeout, and reprocessing does not duplicate the external effect.

How you prove it works

• A test delivering the same event twice concurrently, asserting one record and one effect.
• A test killing a worker mid-run and asserting recovery on the next pass.
• A test asserting a repeatedly failing record lands in the dead-letter state rather than retrying forever.

What you walk away with

An ingress that responds in single-digit milliseconds, a worker that survives being killed, and a dead-letter queue you can inspect and replay.