Designing High-Availability Payment Gateway Middleware

Payment middleware sits in the least forgiving part of a banking system: money is moving, regulators are watching, and downtime is measured in failed transactions rather than missed page views.

Why availability is non-negotiable

When a transfer request comes in over BI-FAST, SKN, or RTGS, the middleware has to either complete it or fail it cleanly. A half-processed transaction is far worse than a rejected one, so every design decision starts from a single question: what happens if this node dies right now?

Patterns that held up in production

• Idempotency keys on every request so retries never double-charge a customer.
• Stateless services behind a load balancer, with shared state pushed into Redis and the database.
• Circuit breakers around every downstream bank connection, so one slow partner cannot take the whole gateway down.
• Health checks that fail fast, so Kubernetes can pull a bad pod before it poisons live traffic.

A minimal idempotency guard in Go looks like this:

func (h *Handler) Transfer(w http.ResponseWriter, r *http.Request) {
    key := r.Header.Get("Idempotency-Key")
    if key == "" {
        http.Error(w, "missing idempotency key", http.StatusBadRequest)
        return
    }
    if prev, ok := h.store.Get(key); ok {
        writeJSON(w, prev) // replay stored result, never re-run
        return
    }
    res, err := h.process(r.Context(), r)
    if err != nil {
        http.Error(w, err.Error(), http.StatusBadGateway)
        return
    }
    h.store.Save(key, res)
    writeJSON(w, res)
}

Observability is part of availability

You cannot keep a system up if you cannot see it. Centralised logging with the ELK stack, plus Prometheus and Grafana dashboards, meant we usually spotted a degradation before the client did.

The goal was never zero incidents; it was making every incident boring, contained, and quick to recover from.