Why Bigger Context Windows Make AI Agents Worse, Not Better

digitado ⋅ 18 de July de 2026

The industry sells context length as pure progress — 200K, 1M, 2M tokens. So engineers stuff everything in “just in case.” That’s the bug, not the fix. Signal-to-noise beats raw volume, every time.

The day a bigger prompt made my agent dumber

I had a code-fixing agent that worked. Small, focused, fast. You handed it a bug, it pulled the two or three relevant files, and it shipped a patch.

Then I “helped.” The agent kept missing context from neighboring modules, so I did the obvious thing: I pasted in the whole repo. The full file tree. The entire conversation history from the debugging session. Every doc the retriever could find, ranked or not. Now it has everything, I thought. Now it can’t miss.

It got worse. Measurably worse.

Accuracy dropped. Latency roughly tripled — I went from a few seconds to a coffee-refill wait per step, because every call now chewed through ~90K tokens. And the answer? It confidently edited the wrong file. The one file that actually mattered was in there — sitting somewhere around token 90,000, in the middle of a wall of imports and unrelated helpers the model glanced at and forgot.

I had given it more information and less intelligence. That’s not a paradox. That’s how attention over long context actually behaves.

A model that can read a million tokens is not a model that understands a million tokens. Capacity is not comprehension. Right where it matters most, those two things come apart.

More context is not more intelligence

The marketing frames context length as a straight line of progress. 8K, then 128K, then 200K, then a million, then two. Bigger number, better model. So the engineering reflex becomes: fill it. If the window holds 200K tokens and I’m using 6K, I’m “wasting” the model, right?

Wrong, and it’s the most expensive instinct in agent design right now.

The window is not free storage you’re obligated to fill. It’s the model’s working attention for this one call. Everything you put in competes for that attention. A long context doesn’t give the model more focus — it spreads the same focus thinner across more tokens. You are not adding signal. You are diluting it.

Two failure modes turn that dilution into wrong answers. Both are established, repeatedly-observed behavior in long-context LLMs, not speculation.

Lost in the middle: present, and still ignored

Take a fact the model needs, and place it at the very start of a long prompt. The model uses it. Place the same fact at the very end. It uses it. Place it dead center, buried in thousands of tokens of other material, and accuracy on retrieving that fact drops — sometimes sharply.

This is the lost-in-the-middle effect, and once you’ve seen it in your own logs you stop trusting “it’s in the context” as a defense. Long-context models exhibit a U-shaped attention profile: strong at the edges, weak in the belly. Information at the start and end of the window gets used; information in the middle gets skimmed.

That changes the whole game. The question is no longer did I include the relevant fact? It’s where in the window did it land, and what did I bury it under? When I pasted my whole repo, the file that mattered didn’t fall out of context. It fell into the middle of it. Same thing, from the model’s point of view.

So position is a design decision, not an accident of whatever order your retriever happened to return. If a fact is load-bearing, it belongs at an edge.

Distractors: the model can be talked out of the right answer

Lost-in-the-middle is about where you put things. Distractors are about what else you put in.

A distractor is a passage that’s irrelevant to the question but plausible-looking — same topic, similar vocabulary, wrong answer. The intuition most engineers carry is that extra context is at worst harmless: if it’s not useful, the model will just ignore it. That intuition is false. Irrelevant-but-on-topic passages actively degrade accuracy. They pull the model toward confident wrong answers it would not have given on a clean prompt.

This is the part that should change how you build retrieval. Accuracy is driven by signal-to-noise, not raw volume. Ten tightly relevant chunks beat fifty chunks where forty are “kind of related.” Every low-relevance passage you add is not neutral ballast — it’s an active vote for the wrong answer.

Which means a retriever that returns more candidates to be safe is making your agent less safe. The job isn’t recall. The job is precision under a budget.

Treat the window as a budget you spend, not a tank you fill

Here’s the mental flip. Stop thinking of the context window as storage you must fill, and start thinking of it as a token budget you spend deliberately. Every token you spend should earn its place. The default state of a token is out.

So I build a context assembler that does four things on every call: retrieve candidates, rerank them by real relevance, keep only what fits the budget, and order survivors so the strongest sit at the edges. The middle gets the weakest of the keepers, because the middle is where attention goes to die.

def assemble_context(query, candidates, token_budget, count):
    # 1. Rerank by true relevance to THIS query (a cross-encoder beats#    raw vector similarity — it catches plausible distractors).
    ranked = sorted(candidates, key=lambda c: rerank_score(query, c),
                    reverse=True)

    # 2. Greedily keep the top items that fit the budget. Drop the rest.#    Dropping is the feature, not a failure.
    kept, used = [], 0for c in ranked:
        if used + count(c) <= token_budget:
            kept.append(c); used += count(c)

    # 3. Order so the highest-relevance items sit at the EDGES, and the#    weakest land in the middle — where the model attends least.
    head = kept[0::2]            # 1st, 3rd, 5th ... strongest outward
    tail = list(reversed(kept[1::2]))
    return head + tail           # strong | ... weak ... | strong

The interleave at the end is the part people skip. Sorting by relevance and pasting top-to-bottom puts your second-best item in the highest-attention slot and shoves your tenth-best into a strong-attention edge by accident. Fold the ranked list so #1 and #2 take the two edges and the long tail of weak-but-kept context settles into the low-attention middle. You’re matching importance to the model’s actual attention curve.

Context rot: the slow poison of long agent runs

There’s a fourth failure that only shows up in agents, not single prompts, and it’s the one that bit me hardest in production.

A long agent run accumulates garbage. Tool call #3 returned a 4,000-token API dump you needed for one field. Step #7 went down a dead end you backed out of. Step #11 retried a failed search three times, each result still sitting in history. By step #20, most of the context is exhaust — stale tool outputs, abandoned branches, dead ends — and the model is dragging all of it into every new decision. I call it context rot, and it compounds. Late steps in a long run are reasoning over a window that’s mostly noise the agent generated itself.

The fix isn’t a bigger window to hold more rot. It’s the opposite. Compress aggressively: summarize completed sub-tasks into a one-line result and evict the raw output. Give the agent an explicit tool to drop context it’s done with — a forget(scope) it can call when a sub-task closes. The agent that prunes its own working memory stays sharp at step 30. The one that hoards everything “just in case” is reasoning through fog. (I spend most of my research time on LLM orchestration, and this single discipline — bounded, self-pruning working context per step — does more for multi-step reliability than any prompt I’ve ever written.)

A tight, well-ordered 8K context beats a sloppy 200K one. Not sometimes. As a rule.

Takeaways

Stop filling the window. Start budgeting it. The default state of any token is out. Each one has to earn its slot against everything else competing for the model’s attention.
Position is a design choice. Load-bearing facts go at the start and the end. The middle is where attention goes to die — never bury what matters there.
Precision beats recall in retrieval. Rerank and prune. A handful of distractors will talk your model out of the right answer. Signal-to-noise drives accuracy, not token count.
Compress history and give the agent a forget. Long runs rot. Summarize closed sub-tasks, evict raw tool exhaust, and let the agent drop context it no longer needs.
A curated 8K beats a sloppy 200K. Bigger context is a capability, not a strategy. The engineering is in what you leave out.

The next time your agent gets something wrong, don’t reach for a bigger window. Ask what you buried in the middle, and what noise you let it vote with. The fix is almost never more context. It’s less, in the right order.

Like 0

Liked Liked