[P] Provider outages are more common than you’d think – here’s how we handle them

I Work on Bifrost (been posting a lot here lol) and wanted to share what we learned building multi-provider routing, since it’s messier than it seems.

Github: https://github.com/maximhq/bifrost

Initially thought weighted routing would be the main thing – like send 80% of traffic to Azure, 20% to OpenAI. Pretty straightforward. Configure weights, distribute requests proportionally, done.

But production is messier. Providers go down regionally. Rate limits hit unexpectedly. Azure might be healthy in US-East but degraded in EU-West. Or you hit your tier limit mid-day and everything starts timing out.

So we built automatic fallback chains. When you configure multiple providers on a virtual key, Bifrost sorts them by weight and creates fallbacks automatically. Primary request goes to Azure, fails, immediately retries with OpenAI. Happens transparently – your app doesn’t see it.

The health monitoring part was interesting. We track success rates, response times, error patterns per provider. When issues get detected, requests start routing to backup providers within milliseconds. No manual intervention needed.

Also handles rate limits differently now. If a provider hits TPM/RPM limits, it gets excluded from routing temporarily while other providers stay available. Prevents cascading failures.

One thing that surprised us – weighted routing alone isn’t enough. You need adaptive load balancing that actually looks at real-time metrics (latency, error rates, throughput) and adjusts on the fly. Static weights don’t account for degradation.

The tricky part was making failover fast enough that it doesn’t add noticeable latency. Had to optimize connection pooling, timeout handling, and how we track provider health.

how are you folks handling multi-provider routing in production. Static configs? Manual switching? Something else?