[D] Shower thought after 13hr coding session: Could physical filtration principles inform attention head design? (Claude wrote this, I just had the idea)

Full transparency upfront: I’m not an ML researcher. I’m a solutions architect who works with voice AI integrations. After a 13-hour coding marathon today, my brain started making weird connections and I asked Claude to help me write this up properly because I don’t have the background to formalize it myself.

I’m posting this because: (a) I genuinely want to know if this is interesting or stupid, (b) I don’t need credit for anything, and (c) if there’s signal here, someone smarter than me should do something with it.

The shower thought:

Physical substrate filtration (like building a road bed or water filtration) layers materials by particle size: fine sand → coarse sand → gravel → crushed stone. Each layer handles what it can and passes the rest up. Order matters. The system is subtractive.

Attention in transformers seems to have emergent granularity—early layers handle local patterns, later layers handle global dependencies. But this is learned, not constrained.

The question:

What if you explicitly constrained attention heads to specific receptive field sizes, like physical filter substrates?

Something like:

∙ Heads 1-4: only attend within 16 tokens (fine) ∙ Heads 5-8: attend within 64 tokens (medium) ∙ Heads 9-12: global attention (coarse) 

Why this might not be stupid:

∙ Longformer/BigBird already do binary local/global splits ∙ WaveNet uses dilated convolutions with exponential receptive fields ∙ Probing studies show heads naturally specialize by granularity anyway ∙ Could reduce compute (fine heads don’t need O(n²)) ∙ Adds interpretability (you know what each head is doing) 

Why this might be stupid (more likely):

∙ Maybe the flexibility of unconstrained heads is the whole point ∙ Maybe this has been tried and doesn’t work ∙ I literally don’t know what I don’t know 

Bonus weird idea:

What if attention was explicitly subtractive like physical filtration? Fine-grained heads “handle” local patterns and remove them from the residual stream, so coarse heads only see what’s ambiguous. No idea if gradient flow would survive this.

What I’m asking:

1. Is this a known research direction I just haven’t found? 2. Is the analogy fundamentally broken somewhere? 3. Is this interesting enough that someone should actually test it? 4. Please destroy this if it deserves destruction—I’d rather know 

Thanks for reading my 1am brain dump. For Clyde Tombaugh.