[D] Anyone running into KV cache / memory bandwidth limits with long-context inference?

Hey guys, I’m working on optimizing inference for transformer models and keep seeing memory bandwidth become the bottleneck well before compute, especially once context length gets past ~8k tokens.

A few questions for for teams running LLaMA / Mistral / similar models in production:

Is KV cache memory your limiting factor at longer context?

Do you hit HBM limits or throughput collapse first?

What have you tried so far (quantization, FlashAttention variants, batching tweaks, offloading, etc.)?

What tradeoffs were not acceptable (latency, accuracy, complexity)?

Just trying to understand how people are dealing with this in real systems vs benchmarks.

Curious to hear what’s actually painful in practice.