This is the first chapter of a series where I build a saga orchestrator from scratch — the pattern that keeps data consistent across services when a single business action touches several of them. Before any code, this chapter is only about the problem. Because the problem is the whole reason the pattern exists, and if it doesn't feel real, none of the machinery in later chapters will make sense.
The task that looks trivial
Here's the setup, and it's deliberately mundane. You run the backend for an online store. A merchant uploads a CSV of orders to fulfill — say 500 rows. For each order you have to do four things:
create the order in your system
reserve the stock so you don't sell what you don't have
record the payment
hand it to shipping
Four steps. It sounds like an afternoon's work — a loop over rows, four function calls each. And in one specific world, it is that easy.
In a monolith, this is one transaction
If your whole application is one service talking to one database, those four steps are a single transaction. You wrap them in BEGIN and COMMIT, and the database gives you a guarantee so strong it's easy to forget it's there: all four happen, or none of them do.
If the settlement is declined at step three, you ROLLBACK — and the order and the stock reservation vanish as if you never touched them. There is no in-between. The database's atomicity (the A in ACID) does the hard part for you, silently, on every request. You have probably never once thought about what it would take to build that guarantee yourself.
You're about to.
But production isn't one database
Real systems of any size are not one service and one database. Orders live in an Order service. Stock lives in an Inventory service. Payments go through a Payment service that keeps the books — a chart of accounts and the ledger that posts to them. Shipping is its own service too. Each owns its own data, deliberately, so teams can move independently.
And here is the sentence the whole problem hangs on: there is no BEGIN…COMMIT that spans four services. You cannot wrap a transaction around a call to your Order database, a call to your Inventory database, and a call to your Payment database. There is no coordinator that will roll all three back together. The single guarantee your monolith handed you for free — all or nothing — simply does not exist here.
So the question becomes very concrete: what happens when you're halfway through and something fails?
What happens when step three fails
Walk one order through. You create it — committed to the Order database. You reserve the stock — committed to the Inventory database. You ask the Payment service to post the payment, and it fails.
You now hold a real order and a real stock reservation for a payment that never happened. Nothing crashed. No bug. Every service did exactly what it was told. And yet the system is sitting in a state that a monolith could never have produced — half-applied — and there is no ROLLBACK to walk it back, because the two committed steps live in two different databases that have already said "done."
This is the entire problem in one picture. Once work spans services, you are guaranteed to sometimes be caught half-done, and undoing it is now your job, by hand.
The failure modes, and what they cost
"Payment declined" is just the friendly one. Flip the order of the steps, or fail at a different point, and each leaves a different mess — and each mess is a real support ticket or real money:
What went wrong | The state it leaves | What it costs |
|---|---|---|
Settled, then out of stock | Money taken for an item you can't ship | Refund + an angry customer + a support ticket |
Reserved, then never settled | Stock locked for a dead order | Phantom out-of-stock; you can't sell real inventory |
Shipped, then payment failed | Goods gone, no money | Pure loss — you can't un-ship a package |
Timed out, so you retried | Did the first settlement happen? You can't tell | Double settlement — the account debited twice, and a reconciliation mess |
That last row deserves a stare. When a network call times out, you cannot know whether the work happened and the reply was lost, or the request never arrived. The two are indistinguishable. So you retry to be safe — and risk settling the same order twice. "Just retry" is not free; it's a second, sneakier way to corrupt the system.
Now do it five hundred times at once
Everything above was one order. The CSV has 500, and you won't process them one-at-a-time — that would take forever — so they run concurrently. Which stacks new problems on top of the old ones:
They share stock. Order #12 and order #390 both want the last unit. Two concurrent reservations, one item — who wins, and does the loser find out cleanly?
They don't share fate. Order #12 failing must not roll back order #390. Each order is its own all-or-nothing unit, independent of the other 499.
The process can die mid-batch. A crash at row 247 leaves you with no idea which orders completed, which are half-done, and which will double-settle when you re-run the file.
Someone re-uploads the file. To retry the failures, a merchant uploads the same CSV again. Now every already-processed order is at risk of being processed a second time.
So the real task was never "loop over a CSV and call four functions." It was: keep an entire system consistent across services it doesn't control, through partial failures, retries, crashes, concurrency, and duplicate submissions — without the one tool (a transaction) that would have made it trivial.
Why not just use distributed transactions?
There is a textbook answer to "make several databases commit together": two-phase commit (2PC). A coordinator asks every participant "can you commit?", waits for all of them to say yes, then tells them all "commit." It does give you atomicity across services.
Nobody uses it for this, for good reasons. It's blocking — every participant holds its locks open across the network while it waits for the coordinator, so one slow service stalls all of them. It scales badly and fails badly (a coordinator that dies mid-round can leave everyone stuck). And most decisively: A system you don't control will not join your two-phase commit. You cannot ask a third-party payment API to hold a transaction open on your behalf. The moment one of your "services" is someone else's system, 2PC is off the table.
The real question
So you can't make it atomic. That's not a gap to engineer around — it's the ground truth. Which means the question you're actually facing is not:
"how do I roll back across services?"
It's:
"how do I keep the system consistent when I know I'll sometimes be caught half-done — and I can only ever fix one service at a time?"
That reframing is the doorway to the saga. Instead of one atomic transaction, you model the work as a sequence of local transactions, each in one service — and for each step, a compensating action that semantically undoes it (a refund, not an un-charge; a release, not an un-reserve). When something fails partway, you don't roll back — you run the compensations in reverse and walk the system back to a consistent state on purpose.
That's the next chapter. Here I only want one thing to have landed: the problem is real, it's unavoidable the moment you have more than one service, and the comfortable guarantee you've relied on your whole career quietly disappears at exactly the wrong moment.
What this series builds
Over the next chapters I build a working orchestrator for exactly this scenario — a CSV batch of orders, processed across Order, Inventory, Payment, and Shipping — that:
runs each order as a saga with compensations, so a partial failure unwinds cleanly
survives a crash mid-batch and resumes exactly where it left off
is safe to retry and safe to re-upload — no double settlements, no duplicate orders
keeps every report (stock, revenue, fulfillment) correct while orders are mid-flight
and does it for 500 concurrent orders sharing the same inventory
Next chapter: the saga itself — local transactions, compensations, and the one idea (the pivot) that decides when "undo" stops being possible and you have to roll forward instead.