Engineering Reliable Payment & Fulfillment Workflows

Challenge

3D Secure 2 sends its result back through the cardholder's browser, which is something an attacker controls. A forged or replayed callback, or two requests racing each other, could mark an order paid or fulfil it twice. For gift cards that's unrecoverable: once a redeemable code is delivered, it's cash.

Approach

• Treat the browser as advisory. Only the server-to-server payment response is accepted as proof of payment.
• Bind the response to the transaction. It must be approved and match the transaction identifier, the amount (compared as fixed-point, never floating-point) and the ISO 4217 currency. Anything else fails closed with a machine-readable reason.
• Let the database pick one winner. Every state change is a single conditional update, so exactly one request can move a transaction forward across threads and instances.
• Check again before acting. Fulfillment re-reads payment status from the database, and a configurable kill switch holds paid gift cards without stopping checkout.

Implementation

• An explicit transaction state machine: initiated → verifying → pending → completed / failed / expired.
• A verification layer for payment responses and atomic compare-and-set state transitions.
• Hardened 3DS2 callback handling: first write wins, the amount never comes from the browser, and a transaction can only complete through the flow that created it.
• A fulfillment guard and fail-closed issuance switch; provider payloads containing redeemable links are never logged.
• Failure tracking, blocklists and background tasks that expire abandoned payments.

Outcome

Forged, replayed and concurrent completions cannot produce a payment or a fulfillment. Operations can pause gift card issuance instantly while customers keep paying bills and shopping. The changes shipped as a single audited containment release.

Technical perspective

The weakness wasn't a missing feature; it was trust in data that only looked authoritative, combined with read-then-write races. The fix had to be retrofitted into a live checkout serving several flows (shop, bills, reservations) without breaking any of them.