Digital Wallet (Venmo, Cash App)

Overview

Introduction
Requirements
Data Model
API Design
High Level Design
Deep Dive 1: The Immutable Ledger
Deep Dive 2: Cross-Shard Transactions
Deep Dive 3: Asynchronous Settlement & Reconciliation
Complete Architecture
Additional Discussion Points

Introduction

A digital wallet system provides users with a virtual account to store funds, execute peer-to-peer (P2P) transfers, and deposit/withdraw money to external bank accounts.
Designing a robust financial system hinges on absolute data integrity. A system that drops a social feed update is annoying; a system that drops money is a catastrophic failure. Therefore, the primary technical challenges involve maintaining strict ACID compliance under massive concurrency, preventing "double-spending" during network failures, and executing reliable distributed transactions across sharded databases.

Requirements

Functional Requirements
- Internal Transfers: Users can send money instantly to other users within the ecosystem.
- Fund/Withdraw: Users can load money from an external bank or cash out to it.
- Balance Inquiry: Users can view their current available funds.
- Transaction Feed: Users can view a chronological feed of their past activity and social transactions.
Non Functional Requirements
- Strict Consistency (ACID): "No lost money." The system must strictly enforce CAP theorem Consistency for financial movements.
- Idempotency: Network retries must safely absorb duplicate client requests without duplicate fund movement.
- Scalability: Support 100M+ users and handle peaks of 10,000+ Transactions Per Second (TPS).
- High Availability: 99.99% uptime for the core API.
- Low Latency: < 200ms for internal wallet-to-wallet transfers.

Data Model

Financial systems strictly require Polyglot Persistence, leveraging the ACID guarantees of relational databases for active balances and the high-throughput capabilities of NoSQL for immutable audit trails and social feeds.

Relational (PostgreSQL / CockroachDB): The absolute "Source of Truth" for the financial ledger. Standard row-level locking capabilities are mandatory to prevent race conditions during concurrent transfers.
In-Memory Cache (Redis Cluster): Crucial for transient state management. It stores high-speed Idempotency Keys to block duplicate requests at the edge, enforces rate limits, and caches user profiles.
Message Broker (Kafka): The durable nervous system of the platform. It decouples the core ledger from secondary systems (notifications, fraud detection, social feed updates).
Wide-Column Store (Cassandra / ScyllaDB): Serves as the historical transaction log and social feed. Since user transaction histories are append-only and accessed via chronological pagination, Cassandra provides massive horizontal write scalability without bogging down the relational ledger.

API Design

The API must enforce secure, idempotent, RESTful interactions.

Create Transfer:
- POST /v1/transfers
- Headers: Idempotency-Key, Authorization
- Body: to_user_id, amount, currency, note
- Returns 201 Created or 409 Conflict.
Fund Account:
- POST /v1/funding/deposit
- Initiates async pull from external bank.
Get Balance:
- GET /v1/accounts/me/balance
Get Social Feed:
- GET /v1/feed?limit=20&cursor={token}

High Level Design

At a high level, the system separates internal instantaneous transfers from asynchronous external money movements and social features.

1. The Internal P2P Path (Synchronous) (1-10)

This is the "fast path." It handles moving money from one digital wallet to another. The actual money movement is strictly synchronous, while the social and notification updates happen asynchronously immediately afterward.

Mobile Client: A user taps "Pay" in the app, generating a unique Idempotency-Key attached to the request payload.
API Gateway: Receives the ingress request, validates the auth token, terminates SSL, and routes the traffic directly to the core orchestrator.
Transfer Service (The Orchestrator): Acts as the central brain and optimized "front door." Before executing any heavy database transactions, it actively orchestrates pre-flight checks to protect system compute resources.
Redis Cluster (L1 Defense): Queried first by the Transfer Service. It checks the Idempotency-Key in O(1) time to immediately intercept and block duplicate network retries (e.g., if the user lost cell service), preventing wasted downstream processing.
Fraud Service: Queried synchronously by the Transfer Service only after a request safely passes the Redis check. It evaluates the sender's IP and velocity rules in milliseconds to ensure the transaction isn't a malicious account takeover.
Ledger DB (PostgreSQL): Engaged only after the request clears all pre-flight checks. The Transfer Service executes an ACID transaction here: securely locking the sender's row, deducting the balance, and atomically writing a "Credit Receiver" message to the local Outbox table.
Debezium (CDC): Detects the Outbox insert in the database's Write-Ahead Log in real-time.
Apache Kafka: Safely receives the published event. (A credit worker picks this up to execute the credit on the receiver's DB shard. Once finalized, a success event drops back into Kafka).
Feed Service: Consumes the success event and writes the denormalized social timeline update to Cassandra.
Notification Service: Independently consumes the exact same success event and formats a message to ping the receiver's phone via APNs/FCM.

2. External Funding (Asynchronous) (11-19)

This is the "slow path." It handles moving money between the digital wallet and the traditional banking system. Because banks are slow, this flow is broken into two distinct phases separated by days.

Mobile Client: The user requests to pull $100 from their linked bank account.
API Gateway: Receives and routes the ingress request.
Funding Service: Manages this long-running process and acts as the state machine tracker.
Ledger DB (PostgreSQL): Receives a transaction write with a status of PENDING. The user's spendable digital wallet balance does not increase yet.
External Bank Adapter: Formats the request (e.g., into an ISO 8583 message or a Stripe API call) and translates it for the outside world.
External Bank Network: Receives the handoff, where it will take 1-3 business days to clear the clearinghouse. Successfully finishes moving the money three days later and fires an HTTP POST webhook back to your system.
Webhook Listener: Verifies the cryptographic signature to ensure it's not a hacker spoofing a deposit.
Apache Kafka: Buffers the validated payload in a dedicated topic to ensure it isn't lost if downstream databases are busy.
Reconciliation Worker: Consumes the payload from Kafka and begins processing.
Ledger DB (PostgreSQL): Is queried by the worker to find the original PENDING transaction, transitions it to SETTLED, and officially adds the $100 to the user's spendable digital wallet balance

While this synchronous internal path works initially, it catastrophically fails at scale. Concurrent transactions cause database deadlocks, network flakes cause double-charging, and sharded databases make local ACID transactions impossible. We must dive deeper.

Deep Dive 1: The Immutable Ledger

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.