Designing High-Availability Identity Systems Processing Billions of API Calls
It’s Tuesday at 3 a.m., and the engineer on call receives a page: there’s an issue with the entire service authenticating. After an hour and a half, it turns out to be an incorrect time-to-live (TTL) for the cache, which allowed stale data about token information to “break” all the validation checks. There were no warnings. No circuit breaker tripped. It just didn’t scale. Identity is a “fragile” area of software systems. It is the conduit for all user authentication, token renewals, and authorization. With billions of calls per day, a 0.01% error rate means tens of thousands of authentication failures. The design of systems to handle this without performance or correctness penalties is a design trade-off which must be weighed in three aspects: observability, latency engineering, and failure isolation. Observability is needed to do latency engineering, and latency engineering is needed to do failure isolation.
Observability: You Cannot Fix What You Cannot See
Before we can do anything, we have to have one thing: you need to be able to see what’s happening in your system. Logging is not for heavy traffic. Logging systems are not built for that amount of traffic. The critical factor in a secure identity system is SNR. We need to have structured logging with trace context. The correlation ID should be included with every API request, from the edge to the token services and the database queries, so that you can understand the request’s path through the application.
But, as well as logs, we can also have spans from distributed tracing with OpenTelemetry. You can see if it’s a performance issue in your JWT signing service, Redis token cache, or database queries. If not, engineers will guess. With billions of calls per day, you can’t guess.
We need special metrics for identity workloads. Normal percentiles (p50, p95, p99) miss the tail. A separate error rate for each client ID, error rate for each grant type (for OAuth), and cache hit rate should be tracked. You could have a 0.02% error rate for all requests, but 40% errors for one OAuth client-and that’s what you get paged about.
Latency Engineering: Where Milliseconds Become Business Outcomes
Once you have observability of the system, next is performance, and it’s important. Identity systems are in the critical path for most user requests. A 50 ms increase in the time to validate a token doesn’t just affect the identity service. It impacts several services in a microservice architecture, and this results in a noticeable difference in the user experience for a marginal increase in system response time. Hence, managing the latency budget is an important engineering consideration in identity.
The best solution is caching, but it raises the problem of correctness for an authentication system. Revocation is the rub. If a user revokes a session, or an administrator disables an account, then having an in-memory cache that contains a valid representation of the token (or the account) is a security problem. So we have a low time-to-live and publish/subscribe invalidation of the cache entries. This is a fast-by-default, consistent-on-demand cache (not a fresh-by-default cache).
We need to be careful with cryptography. JWTs are not parallelizable and are CPU-expensive for RSA signing. Signing templates-signing token templates for a set of common claims-elliptic-curve cryptography (if appropriate), and a compute pool for signing are all possible solutions. It’s not about making one operation faster but ensuring it’s not slow.
Failure Isolation: Designing for Degradation, Not Just Uptime
For all your efforts to get a system to run fast, it will eventually hit something it can’t cope with. Not “if” any dependency goes down, but “when.” High availability is seen as a problem of redundancy. Systems scale horizontally, have multiple availability zones, etc. But, when it comes to identity systems, most outages don’t involve hardware. They are dependency outages, as a downstream API call slows and the connection pool is exhausted, or a misconfigured rate limit turns into an authentication epidemic.
Enter the bulkhead pattern. By dividing up the thread pools (or async task queues) by type of traffic, password change requests don’t exhaust (for example) the threads used for token refreshes. This should be done at the network level by service mesh settings and not at the application level. The point is to ensure the failure of one type of identity service request doesn’t take down the whole authentication service.
Reads from the database and external directory (LDAP, Active Directory) services should use circuit breakers. A fallback state should be provided, which for identity services is returning the latest cached results for non-critical reads and queuing writes until the service is back. Chaos engineering in the staging environment (where faults can be injected) helps ensure that these fallback states are correct.
Building Toward Genuine Resilience
All of the systems that survive millions and billions of API calls have one thing in common: they are designed to fail. Observability telemetry that delivers the signals, latency budgets at each layer of the system, and failure isolation in the design are the difference between identity infrastructure that works and identity infrastructure that is trusted.
The engineering practices we’ve described are continuous. They include periodic load testing, constant review of the observability telemetry, and validation of the failure isolation boundaries as the system evolves. Complacency is a common precursor to outages at scale. The systems that are successful keep reliability as an attribute of the system, rather than something they achieved on their first release.
