The transactional outbox
Solve the dual-write problem: update your database and publish events reliably with an outbox, change data capture and idempotent consumers.
- Explain the dual-write problem when saving data and publishing an event
- Implement the transactional outbox with a relay or change data capture
- Version events and upgrade old ones safely
Here’s the most common bug in event-driven systems. Ordering saves an order, then publishes OrderPlaced:
db.save(order) # 1. commit to the database
broker.publish("OrderPlaced", ...) # 2. publish the eventIf the service crashes between 1 and 2, the order exists but nobody is told: the kitchen never cooks it. Swap the order and a crash leaves an event for an order that doesn’t exist. This is the dual-write problem: you can’t atomically write to two different systems.
Try it
Watch the dual write fail
Step through a crash at the worst moment, then see how the outbox fixes it.
Transactional outbox: in the same database transaction as the business change, insert the event into an outbox table. A separate relay publishes unpublished rows and marks them sent. Since the event is committed atomically with the data, it can’t be lost - but it can be published more than once (a crash between publishing and marking), so consumers stay idempotent, often with an inbox table of processed event IDs.
Instead of polling the outbox, change data capture (CDC) tools such as Debezium tail the database’s transaction log and turn new outbox rows into messages - no polling, and in commit order.
1{
2 "outbox_row": {
3 "id": 981,
4 "aggregate": "order",
5 "aggregate_id": 42,
6 "type": "OrderPlaced",
7 "version": 2,
8 "payload": {"order_id": 42, "dishes": ["spicy miso"], "total_cents": 1450, "currency": "USD"},
9 "created_at": "2026-10-02T18:30:00Z",
10 "published": false
11 }
12}Versioning events
Events live a long time - in broker topics, in event stores, in replayed history - so their shape must evolve carefully. Include a version in every event. Additive changes are safe for tolerant readers; for real changes, publish a new version and write an upcaster: a function that converts old versions to the current one, so consumers only handle the newest shape.
Key takeaways
Saving data and publishing an event are two writes that can’t be atomic: the dual-write problem.
The transactional outbox stores events in the same transaction; a relay or CDC publishes them.
Delivery is at-least-once, so consumers deduplicate (an inbox of processed event IDs).
Version events and upcast old versions so consumers handle one shape.
Lesson quiz
7 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: simulate microservice patterns in Python
Build small Python simulations of the patterns - routers, sagas, outboxes, circuit breakers, traces - and run them against sample inputs. They run locally in your browser; no servers or containers needed.
Run the outbox relay
The input has outbox rows id event, then a line crash after publishing N (or no crash). The relay publishes unsent rows in id order and marks each sent. If it crashes after publishing row N but before marking it sent, it restarts and starts again from the first unsent row.
The consumer keeps an inbox of processed IDs. Print each delivery as deliver 3 DishesCooked: processed or : duplicate ignored, then published: 5, processed: 4.
- Crash mid-way
- Smooth run
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Upcast old events
Each input line is an OrderPlaced event as JSON, in version 1, 2 or 3. Upcast every event to version 3 and print it with json.dumps(event, sort_keys=True):
- v1 → v2:
total(a string of dollars like"14.50") becomestotal_cents(an integer, 1450), andcurrency"USD"is added. - v2 → v3:
customer(a full name string) becomescustomer={"name": <the name>, "planet": "Earth"}- early customers all lived on Earth.
Set version to 3. End with upcast: 2 from v1, 1 from v2, 1 already v3.
- Mixed history
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…