How do you safely move money at massive scale without charging the customer twice?
Imagine this simple scenario:
You have ₹500 in your account and try to pay a merchant ₹400.
You tap Pay.
The network is slow.
You tap again.
Now two identical requests reach the backend almost simultaneously.
If the system isn’t designed correctly, both requests might see the ₹500 balance, both might authorize the payment, and you could end up with:
₹500 − ₹400 − ₹400 = −₹300
Obviously, a payment system cannot work this way.
At UPI-scale volumes—tens of billions of transactions over a month—the solution isn’t a single database lock or a simple if balance >= amount check.
It requires multiple layers of protection working together.
1. Where Does Double-Spending Come From?
There are two common problems.
Problem 1: The user retries the same payment
The user clicks Pay, but the response doesn’t arrive.
The app assumes the request failed and retries.
The backend receives:
Request #1 → Pay ₹400
Request #2 → Pay ₹400Both requests may represent the same business transaction.
If the backend treats them as two independent transactions, the customer could potentially be charged twice.
This is why payment systems need idempotency.
Problem 2: Two different transactions race with each other
Consider:
Balance = ₹500
Transaction A → ₹400
Transaction B → ₹400Without concurrency control, both transactions can execute:
Read balance
↓
₹500 >= ₹400
↓
Approve
↓
Deduct ₹400Both transactions saw the same old balance.
The result can be incorrect.
This is a classic race condition.
So we need two different protections:
Idempotency prevents the same transaction from being processed twice.
Concurrency control prevents different transactions from incorrectly modifying the same account at the same time.
2. Idempotency Key — The First Line of Defense
Every payment request should carry a unique transaction or idempotency key.
For example:
transactionId = 7f3c9b2e-...The client may retry the request multiple times, but all retries carry the same key.
The backend maintains an idempotency record:
IDEMPOTENCY_KEY
-------------------------------
key
status
response
created_at
expires_atA fast distributed store such as Redis can be used for short-lived request coordination.
For example:
SETNX transactionIdIf the key doesn’t exist:
SETNX → SUCCESS
↓
Process paymentIf the key already exists:
SETNX → FAILURE
↓
Duplicate request
↓
Return existing resultThe important principle is:
The same business transaction must produce the same outcome, regardless of how many times the client retries it.
But there is an important caveat
Simply doing:
SETNX → process paymentis not enough for a financial system.
What happens if the service crashes after SETNX but before the payment is committed?
You could end up with:
Idempotency key = exists
Payment = not completedTherefore, production systems generally need a durable transaction record and a well-defined state machine rather than relying solely on a Redis key.
Redis is excellent for fast duplicate detection, but the payment’s durable source of truth should be persistent.
3. Preventing the Race Condition
Idempotency doesn’t solve this problem:
Transaction A → ₹400
Transaction B → ₹400These are two legitimate transactions.
They simply happen concurrently.
The balance update must therefore be atomic.
One traditional approach is a database row lock:
SELECT balance
FROM accounts
WHERE user_id = 123
FOR UPDATE;The database locks the account row.
Then:
Check balance
↓
balance >= amount?
↓
Update balance
↓
Write transaction/ledger entry
↓
COMMITWhile Transaction A owns the lock, Transaction B waits.
After A commits, B sees the new balance.
For example:
Initial balance = ₹500
Transaction A
Lock account
Check ₹500 >= ₹400
Deduct ₹400
Commit
Transaction B
Acquire lock
See balance = ₹100
Check ₹100 >= ₹400
RejectThe race condition disappears.
4. Don’t Treat the Balance as the Only Source of Truth
A common beginner implementation is:
UPDATE accounts
SET balance = balance - 400
WHERE user_id = 123;That isn’t enough for a financial system.
You also need a ledger.
A simplified double-entry model looks like:
Payment ₹400
Debit → Customer account ₹400
Credit → Merchant account ₹400Both entries belong to the same business transaction.
Conceptually:
Payment
│
┌────────┴────────┐
↓ ↓
Debit Customer Credit Merchant
₹400 ₹400
The ledger should ideally be append-only and immutable.
Instead of modifying historical transactions, new entries represent subsequent adjustments, reversals, refunds, etc.
This provides an important capability:
The system can reconstruct and reconcile an account’s financial position from the ledger.
The balance can then be treated as a derived/current representation that must remain consistent with the ledger.
5. Double Entry Makes Reconciliation Possible
Suppose the system reports:
Customer balance = ₹10,000But the ledger says:
Opening balance ₹8,000
+ Credits ₹5,000
- Debits ₹3,000
-----------------------------
Calculated balance ₹10,000Everything matches.
Now imagine the database balance says:
₹10,000but the ledger calculates:
₹9,600That’s a red flag.
Something went wrong.
This is why financial systems need reconciliation, not just successful API responses.
6. Scaling to Billions of Transactions
At massive transaction volumes, putting everything into one database becomes a bottleneck.
A common scaling strategy is sharding.
For example:
hash(accountId) % Nmight determine which shard owns an account.
Conceptually:
Account 123
↓
Hash
↓
Shard 5All transactions involving that account are routed to the same shard.
This has another major benefit:
Concurrency control becomes easier when the account’s authoritative state lives on one shard.
Instead of trying to coordinate locks across multiple database instances, transactions affecting the same account can be serialized at its owning partition.
7. Where Does Kafka Fit?
Kafka is excellent for asynchronous processing and decoupling, but it shouldn’t be treated as the mechanism that guarantees financial correctness by itself.
A simplified architecture might look like:
API Gateway
│
▼
Payment Service
│
┌───────┴────────┐
│ │
▼ ▼
Idempotency Account/Payment
Store Database
│
▼
Outbox
│
▼
Kafka
│
┌────────────┼────────────┐
▼ ▼ ▼
Bank/PSP Notifications Analytics
Workers
The payment can first be validated and durably recorded.
After that, downstream processing can happen asynchronously.
Kafka provides:
- Decoupling
- Buffering
- Horizontal scalability
- Retry capabilities
- Consumer parallelism
- Replayability
But consumers must still be idempotent.
Kafka delivering a message twice should not result in money being moved twice.
8. What Happens When the Bank Doesn’t Respond?
This is where payment systems become significantly more interesting.
Suppose the request is sent to the downstream bank.
The connection times out.
Did the bank reject the transaction?
Or did the bank successfully process it but the response was lost?
You cannot safely assume failure.
A simplified state machine can be:
INIT
│
▼
PROCESSING
│
├──────────────► SUCCESS
│
├──────────────► FAILED
│
└──────────────► PENDINGIf the downstream system times out:
PROCESSING
↓
PENDINGAn asynchronous retry process can then query the downstream system using the same transaction/idempotency key.
Retry
↓
Check transaction status
↓
SUCCESS / FAILED / PENDINGThis is much safer than:
Timeout → assume failed → retry as a new paymentbecause that could create a duplicate debit.
9. Reconciliation Is the Final Safety Net
Even a highly reliable distributed system can experience:
- Network failures
- Timeouts
- Duplicate messages
- Partial failures
- Database failures
- Consumer crashes
- Downstream inconsistencies
That’s why payment systems need reconciliation.
A reconciliation process can periodically compare:
Internal Ledger
vs
Bank/Network Settlement DataFor example:
Every hour
↓
Read internal transactions
↓
Compare with external settlement
↓
Match transaction IDs
↓
Identify discrepancies
↓
Alert / investigateA simple duplicate detector can group transactions by their idempotency key:
transaction_key
↓
GROUP BY
↓
COUNT(*) > 1
↓
InvestigateReconciliation doesn’t prevent the original problem.
It detects problems that escaped the primary controls.
10. Putting the Pieces Together
A robust payment architecture therefore looks something like:
┌──────────────┐
│ API Gateway │
└──────┬───────┘
│
▼
┌──────────────────┐
│ Payment Service │
└────────┬─────────┘
│
┌───────────┴───────────┐
│ │
▼ ▼
Idempotency Account/Payment
Store Database
│ │
│ ┌─────┴─────┐
│ │ │
│ ▼ ▼
│ Balance Ledger
│
│
└──────────────┐
▼
Outbox
│
▼
Kafka
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Settlement Notification Analytics
│
▼
Bank / Network
+ Reconciliation
+ Monitoring
+ Audit
Each component has a specific responsibility.
| Problem | Primary mechanism |
|---|---|
| Client retry | Idempotency key |
| Duplicate request | Idempotency record |
| Concurrent balance updates | DB locking / atomic update |
| Financial auditability | Immutable ledger |
| Large-scale data | Sharding |
| Async processing | Kafka |
| Downstream timeout | Pending state + status inquiry |
| Consumer retry | Consumer idempotency |
| Data inconsistency | Reconciliation |
| Operational visibility | Monitoring + audit trails |
The Most Important Lesson
The key insight is that there isn’t one magic mechanism that prevents double-spending.
It is a combination of controls:
Idempotency
+
Concurrency Control
+
Atomic Financial Updates
+
Double-Entry Ledger
+
Durable State Machine
+
Async Processing
+
ReconciliationEach solves a different failure mode.
A junior implementation might look like:
Check balance
↓
Deduct money
↓
Retry if something failsThat’s dangerous.
A production-grade design thinks more like:
Identify transaction
↓
Guarantee idempotency
↓
Serialize conflicting updates
↓
Atomically record financial state
↓
Write immutable ledger
↓
Publish durable events
↓
Process downstream asynchronously
↓
Handle uncertainty with PENDING
↓
Reconcile continuouslyThat’s the difference between “an API that moves money” and “a financial system that can be trusted.”
Final Thought
When someone says:
“UPI handles billions of transactions. How does it make sure a customer isn’t charged twice?”
The answer isn’t simply Redis, Kafka, database locks, or idempotency.
The real answer is defense in depth.
Every transaction passes through multiple layers designed around one fundamental principle:
A payment may be retried, delayed, duplicated, or partially failed—but the financial outcome must remain correct exactly once.
