The outbox pattern stops an integration from losing messages when an application updates its database and sends a message in the same operation. You write the message to an outbox table inside the same database transaction. A separate process reads that table and publishes the messages to your broker. A message only leaves when the change it describes has actually been saved.
The problem shows up in every integration that keeps another system informed. An order is marked as shipped and the invoicing system needs to know. A customer changes their address and the CRM has to follow. Nobody notices until a short outage reveals an invoice that was never created, or one that went out twice.
The dual write problem
A dual write happens when an application writes to two systems that share no transaction, such as a database and a message broker. AWS Prescriptive Guidance on the transactional outbox pattern describes the two ways this goes wrong:
- The database write succeeds and the message fails. The receiving system never hears about the change, and the data drifts apart.
- The message goes out and the database write fails. The receiving system acts on a change that doesn't exist, such as charging for a booking that was rolled back.
The obvious fixes don't hold. Sending first and committing second creates the second failure. Committing first and sending second creates the first, because the process can crash between the two steps. A distributed transaction using two phase commit (2PC) across database and broker is ruled out as well. Chris Richardson of microservices.io points out that the database or broker might not support it, and that coupling a service to both is often undesirable anyway.
The MuleSoft 2026 Connectivity Benchmark Report, based on 1,050 IT leaders in nine countries, counts an average of 957 applications per organization. Only 27% of them are integrated. Every integration that passes on events is a potential dual write.
How the transactional outbox pattern works
The outbox pattern turns two writes into one. The application stores the business change and the message in the same local transaction, in its own database. The message goes into a separate table, the outbox. If the transaction commits, both are there. If it fails, neither is. Richardson sums up the idea in one sentence:
"The solution is for the service that sends the message to first store the message in the database as part of the transaction that updates the business entities."
A second component then takes over: the message relay. It reads new rows from the outbox, publishes them to the broker, and marks them as sent. If the relay crashes, it picks up the unsent rows after a restart. No message gets lost. It can arrive twice, though.
In pseudocode, the write side looks like this:
BEGIN TRANSACTION
UPDATE orders SET status = 'shipped' WHERE id = :order_id
INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload, created_at)
VALUES (:new_uuid, 'order', :order_id, 'OrderShipped', :json, now())
COMMIT
In its simplest form, the relay does this:
every few seconds:
rows = SELECT * FROM outbox WHERE published_at IS NULL ORDER BY created_at LIMIT 100
for each row in rows:
broker.publish(topic = row.aggregate_type, key = row.aggregate_id, message_id = row.id, body = row.payload)
UPDATE outbox SET published_at = now() WHERE id = row.id
Each message gets a fixed id that travels with it to the receiver, for deduplication. The aggregate_id, here the order number, is the key that protects ordering.
Polling publisher or change data capture
There are two common ways to build the relay. A polling publisher queries the outbox at intervals, as in the pseudocode above. With transaction log tailing, also known as change data capture (CDC), a tool follows the database's transaction log, such as the MySQL binlog or the PostgreSQL WAL.
| Polling publisher | Change data capture | |
|---|---|---|
| Works with | Any SQL database | Databases with a readable transaction log |
| Extra components | A background process in your own code | A CDC platform such as Debezium, often with Kafka |
| Latency | Depends on the polling interval | Short, the relay follows the log |
| Ordering | Tricky with concurrent transactions | Follows the commit order of the log |
Debezium ships a ready made outbox event router for this pattern. By default it expects an outbox table with the columns id, aggregatetype, aggregateid, type, and payload. The aggregatetype value decides the Kafka topic, and aggregateid becomes the message key.
Our rule of thumb: start with a polling publisher when you have one database and a modest message volume. Choose CDC when you already run a streaming platform, handle high volumes, or want several services to publish events the same way.
Idempotent consumers and deduplication
The outbox pattern guarantees that a message arrives at least once, and after a failure sometimes more often. If the relay crashes after publishing a message but before marking the row as sent, it publishes that message again after the restart. The receiver must handle that.
The matching pattern is the idempotent consumer. The receiver records the id of every processed message in a processed messages table, in the same transaction as the processing itself. The combination of receiver and message id is the primary key. When the same message arrives again, the insert fails, the transaction rolls back, and the message is discarded.
Brokers add their own deduplication on top:
- Amazon SQS FIFO queues ignore a message with the same deduplication id within a 5 minute window.
- Azure Service Bus tracks the MessageId during a configurable window of 10 minutes by default, with a minimum of 20 seconds and a maximum of 7 days. The basic tier doesn't support it.
- Kafka with Debezium passes the outbox id along in the message headers, so the receiver can check it.
Those windows are finite. A relay that resumes after an hour long outage falls outside a 5 minute window. Broker deduplication is an extra safety net, and the idempotent consumer stays necessary.
Preserving message order
Order matters for many integrations. An OrderCancelled message that arrives before OrderCreated leaves the receiver with an error or the wrong status. Ordering per object is usually enough: every message about order 4711 arrives in sequence.
You get that by using the aggregate_id as the message key. Kafka puts messages with the same key in the same partition, and a partition preserves order. Gunnar Morling of Debezium describes exactly this mechanism. In SQS FIFO the equivalent is the message group id, in Service Bus the session id.
A polling publisher has a trap inside the outbox itself. A sequence number or timestamp is assigned at insert time, while the commit comes later. Two concurrent transactions can therefore become visible in reverse order. A relay that remembers the highest number it read will skip rows. Use a column such as published_at and read everything that hasn't been sent yet. Let only one relay publish for a given key at a time.
Cleaning up and monitoring the outbox
An outbox grows with every message, so it needs a cleanup policy. Three approaches are common:
- Delete after publishing. The relay removes the row once the broker has confirmed the message.
- Delete immediately with CDC. Because Debezium reads the log, you can insert and delete the row in the same transaction. The outbox table stays empty, and the message still goes out through the log.
- Retention period. Sent rows are kept for analysis, then purged. In its Cosmos DB example, Microsoft Learn recommends a retention of several days, like 10 days, so a stalled relay has time to catch up.
An outbox fails silently while messages pile up. At a minimum, measure these signals:
- The age of the oldest unsent message.
- The number of unsent rows in the outbox.
- With CDC, the lag of the replication slot and the size of the transaction log.
- The number of messages in the receiver's dead letter queue.
The third point needs attention on PostgreSQL. The documentation warns that a replication slot can retain so much WAL that it fills the disk. A stalled CDC pipeline can take down your primary database that way. The max_slot_wal_keep_size setting puts a limit on it.
When you don't need the outbox pattern
The pattern adds a table, a background process, and monitoring. You can skip it in these cases:
- There is no second write. A synchronous API integration where the caller waits for the response and retries on failure needs no outbox. The receiver does need to be idempotent.
- A missed message is harmless. A signal to refresh a cache or update a statistic can be lost now and then.
- The broker is the source of truth. With event sourcing, events are the primary data. Microservices.io lists it as an alternative to an outbox.
- You can't change the database. With packaged software, as in many ERP integrations, CDC on the existing tables or the vendor's own webhooks are often the better route.
With several systems involved, an integration layer can handle the outbox and message distribution centrally, as explained in our article on what middleware is.
Isatis has been building integrations for more than 30 years, including work on aviation MRO software, a subsidy administration platform, and RFID in the supply chain. With 30+ engineers in Nijmegen and Sarajevo, 100+ projects, and ISO 9001 and ISO 27001 certification, we know where messages go missing along the way. See how we approach software integration and API development, or get in touch to talk through your own integrations.
Frequently asked questions
What is the outbox pattern?
The outbox pattern is a design pattern in which an application stores an outgoing message in the same database transaction as the change it describes. A separate process then publishes the messages to a message broker. No message gets lost, and none goes out for a change that wasn't saved.
Does the outbox pattern prevent duplicate messages?
No. The pattern guarantees that a message arrives at least once, and after a failure sometimes more often. The receiver therefore has to be idempotent: it stores the id of every processed message and ignores any message it has seen before. Broker deduplication is an extra safety net with a limited time window.
What is the difference between polling and change data capture?
A polling publisher queries the outbox table at intervals and works with any SQL database. Change data capture follows the database's transaction log, for example with Debezium. CDC has lower latency and follows the commit order, but it needs an extra platform and database specific configuration.
Do I need the outbox pattern for a REST API integration?
Only when your system has to notify another system after its own change, and that notification must never go missing. For a synchronous call where the caller waits for the response and retries on failure, an idempotent receiver is enough.
