When evaluating migrating production LLM workloads off commercial cloud APIs, the conversation usually gets oversimplified into a trade-off between quality and infrastructure cost. To look past clean, isolated averages, I built a repeatable evaluation matrix using a real-world workload: cold outreach and contextual profile re-engineering for my resume generation platform.

I benchmarked an unquantized Gemma 2 9B against an optimized FP8 variant served via vLLM on a single commodity NVIDIA L4 GPU.

The dataset evaluates dynamic text generation across diverse recipient personas, varied complexity buckets (short to long contexts), and strict integer formality parameters. I captured client-side and server-side telemetry to audit how FP8 compression changes runtime reality.

The base evaluation set is public at rsher60/resume-gen-benchmark. Here is the raw telemetry and the infrastructure trade-offs I uncovered.

1. Time to First Token (TTFT): The Hidden Prefill Tax of Quantization

The dominant open-source narrative is that FP8 quantization makes everything faster. However, if your application is highly interactive and streaming to a UI, TTFT is the only metric that dictates perceived user speed.

My telemetry exposed a classic hardware-software trade-off:

• The Prefill Penalty: For complex, long-context prompts targeted at high-complexity personas, the unquantized model returned tokens to the server in 866.93ms. The FP8 variant spiked to 1372.12ms—a 58% latency penalty on the initial prefill.
• Why this happens: Quantization reduces memory bandwidth bottlenecks during token generation (the decoding phase). However, the matrix-multiplication de-quantization overhead during the heavy, compute-bound prefill phase introduces a noticeable tax on long input tokens when running on compute-bound commodity hardware like the L4.
• Production Edge Cases: I caught a massive TTFT spike on the FP8 model during short-context runs, hitting 1,740.34ms. This reflects live infrastructure realities under vLLM scheduling—such as a cold prefill or context block swapping. It proves you cannot evaluate architecture purely on clean, isolated averages.

2. End-to-End Latency: Where FP8 Wins the Generation War

While FP8 forces you to pay a tax on the prefill, it aggressively earns its keep during the steady-state decoding loop where the LLM is heavily memory-bandwidth bound.

• By dropping the weight precision down to 8-bit integers, the amount of data moving across the GPU memory bus is sliced roughly in half.
• For medium-length generation sequences, the average client total time dropped from 12,290.2ms to 11,526.2ms.
• If your application handles medium-to-short context sizes or runs entirely asynchronous/batch tasks, FP8 provides a clean, deterministic infrastructure win.

3. The Quality Ledger: Did 9B Parameters Hold the Line?

I verified the generated outputs of the raw unquantized runs against the FP8 model outputs (rsher60/resume-gen-benchmark-results).

• Schema & Persona Adherence: For targeted, single-turn tasks like tailoring text based on a fixed personal profile, a carefully designed system prompt ensures that the 9B architecture executes with near-identical formatting and persona fidelity as a frontier model.
• Semantic Drift: For narrow, domain-specific tasks, FP8 quantization introduced practically negligible semantic drift. The model successfully retained complex context keys—matching the tone for a cold outreach to an engineer versus a formal application letter—while executing within a significantly lower memory footprint.

Strategic Architectural Takeaways

1. Interactive/Low-Batching/Long Inputs: Unquantized weights or a highly aggressive, unchunked prefill strategy might be required to protect your TTFT and prevent user UI friction.
2. Asynchronous/Streaming/Short-to-Medium Context: FP8 is an absolute necessity.

The real reason to run FP8 on an L4 isn’t just saving a few hundred milliseconds of total latency—it’s the VRAM liberation. Shrinking the model footprint frees up massive amounts of memory for the KV Cache, allowing you to scale concurrency without throwing Out-Of-Memory (OOM) exceptions.

I put together the complete analysis, including the upcoming vLLM configurations and cache allocation strategies I used to sustain 92.7% KV Cache utilization under heavy concurrent load, in the full write-up here:

https://billionars.substack.com/p/benchmarking-my-self-hosted-gemma

HF datasets here:

1. https://huggingface.co/datasets/rsher60/resume-gen-benchmark
2. https://huggingface.co/datasets/rsher60/resume-gen-benchmark-results
3. https://huggingface.co/datasets/rsher60/resume-gen-benchmark-results-optimised