EventForge
Order processing with an outbox and idempotent consumers.
Interactive architecture
Order delivery and recovery.
Orders + Outbox: one PostgreSQL transaction
Orders + Outbox: one PostgreSQL transaction
A request enters
The client submits an order. Follow its order ID and event ID through the transaction and delivery path.
- Order
- not created
- Outbox
- not written
- Business effect
- 0 changes
- Kafka offset
- not committed
Accepts the order request and assigns identities to the order and its event.
Read every step as text
Commit the order, publish its event, apply one business update, then commit the offset.
The client submits an order. Follow its order ID and event ID through the transaction and delivery path.
The API writes an order and an outbox event inside the same open PostgreSQL transaction. Neither is durable yet.
Commit makes both rows durable. A rollback would make neither visible to the relay.
The relay reads an eligible committed event. A production relay also needs a safe claim/concurrency strategy.
The relay publishes event-42 with order-42 as its key. In this model the key selects partition p1.
After broker acknowledgement, the relay marks the row sent. This confirmation is not atomic with publishing.
Worker A owns partition p1. Another member may own p0 or p2, but does not process this partition concurrently in the same group.
The handler attempts to insert a durable processed-event ID. A new ID allows the business update in the same transaction.
The event ID and order update commit together. This creates one business effect; the Kafka offset is still uncommitted.
Only after the database succeeds does the consumer commit its offset. This is at-least-once delivery with idempotent effects, not a distributed exactly-once transaction.
Reliability in practice.
Explore the intended recovery behavior in the interactive model. Choose a failure, then run or step through the scenario.
One event, delivered twice
Deterministic demo- Logical events
- 0
- Deliveries
- 0
- Business updates
- 0
Overview
I am designing an order-processing learning project with a transactional outbox, Kafka and idempotent consumers.
The problem
Writing an order to a database and publishing its event are two independent operations. A failure between them can leave downstream services unaware of a committed order.
My contribution
I am working through the delivery contract, crash windows and recovery strategy. The backend is in progress; this walkthrough captures the design decisions.
Decisions & trade-offs
- Write the order and outbox row in one PostgreSQL transaction. A relay publishes committed outbox entries to Kafka.
- Use an order identifier as the partition key to preserve order within a partition. Consumer groups distribute partitions across workers.
- Record a processed event ID atomically with the business update. Retry transient failures; isolate exhausted retries in a dead-letter queue.
At-least-once delivery simplifies recovery but permits duplicates. Idempotency needs durable state, and the outbox introduces cleanup and relay operations. Kafka ordering applies within a partition, not globally.
Two writes. One failure window.
PostgreSQL and Kafka do not share an application-level transaction in this design. Writing the order and then publishing directly leaves a gap: a process can die after the database commit, before the event reaches the broker. Publishing first reverses the inconsistency rather than eliminating it.
I am designing for durable order state, recoverable publication, partition-local ordering and safe redelivery. The target is at-least-once delivery with an idempotent business effect.
The outbox closes a specific gap.
The order and outbox insert share one local transaction. A relay reads only committed records, publishes the original event ID and order key, then records acknowledgement. If it dies after publication, the still-eligible row is published again. The duplication is an intentional recovery trade-off.
For concurrent relay workers, I need a row-claim strategy, bounded in-flight work and outbox cleanup. I am comparing polling with row locks against CDC: simpler deployment versus less polling and more infrastructure.
-- Illustrative SQL, not extracted production code
BEGIN;
INSERT INTO orders (id, status) VALUES ($1, 'created');
INSERT INTO outbox (event_id, order_id, payload)
VALUES ($2, $1, $3);
COMMIT;
-- Publish only committed rows. Mark sent after broker ack.One database commit protects two related facts. Kafka publication happens afterwards.
A committed update can be delivered again.
The most revealing crash happens after the business transaction commits but before the consumer commits its Kafka offset. A replacement worker receives the record again. A durable uniqueness constraint on (consumer, event_id) prevents a second business effect; an in-memory set would lose that protection on restart.
Insert the processed-event marker and perform the business update in the same transaction. On uniqueness conflict, do not run the update. Commit the broker offset only after this decision is durable. An offset is not an atomic extension of the PostgreSQL commit.
-- Illustrative handler transaction
BEGIN;
-- Requires UNIQUE (consumer, event_id).
WITH accepted AS (
INSERT INTO processed_events (consumer, event_id)
VALUES ($1, $2) ON CONFLICT DO NOTHING
RETURNING event_id
)
UPDATE orders SET status = 'processed'
WHERE id = $3 AND EXISTS (SELECT 1 FROM accepted);
COMMIT;
-- Then commit Kafka offset; never before DB success.The INSERT outcome gates the update. If an expected order row is missing, roll back the transaction rather than retain a processed marker for an unapplied effect.
Partition ordering and retry strategy.
A stable order key co-locates related events in a Kafka partition. One consumer-group member owns that partition at a time. This is not global ordering; adding consumers beyond the partition count does not create more parallelism for that topic.
Inline retries pause a partition and preserve its order, but a slow record blocks later ones. Retry topics can avoid that blockage while changing ordering semantics. Moving an exhausted record to a DLQ also requires acknowledgement before advancing the source offset; later replay can overtake subsequent events.
Test the crash windows.
The proposed backend test matrix kills the relay after broker acknowledgement and before marking sent; kills a consumer after DB commit and before offset commit; redelivers a duplicate concurrently; and forces retry exhaustion. Assert durable rows and offsets, not only successful HTTP responses.
I plan to check the durable business state after each injected failure. Concurrent duplicate delivery must leave one processed-event marker and one business update; restart must preserve that outcome.
Backlog and recovery signals.
I plan to track oldest outbox age, publication failures, consumer lag, idempotency conflicts, retries and DLQ age. Order and event IDs belong in correlated traces, not high-cardinality metric labels.
A recovery procedure should answer who owns DLQ triage, how replay preserves identity, and which business states are safe to revisit. Without that procedure, a DLQ is storage, not a recovery mechanism.
Results & next steps
The design handles redelivery by preserving event identity and committing the processing marker with the business update. Next, I plan to validate the crash windows with backend integration tests.
The takeaway
A delivery guarantee is a contract with operational consequences. Make duplicate handling, retries, observability, and recovery part of the design from the beginning.