[P] Yet another garage model – Prisma: Interpretability-Inspired Architecture

digitado ⋅ 11 de March de 2026

Hey y’all! I think some of you might be interested in this creature.

Don’t roast me that much, as I really wanted to collect your feedback and ideas about this ~~crap~~ prototype.

At least it is not GPT/Llama/Mistral/Qwen architecture based, I based it on some ideas that I had while studying other models. The basic differences are:

Attention and output weight sharing (reduces parameters);
Additional weight set in the FFN (increases parameters, yay!);
Introduces Word-Relative Rotary Position Embedding;

The thing with the added weights, I think is the most interesting part of the architecture and I’d like many pinches of salt on that. This weight set is used as a nested gate, making the usual W2 @ (W1 @ x * silu(W3 @ x)) to be W2 @ (W1 @ x * silu(W3 @ x * silu(W4 @ x)))… I’ll leave it as this and wait for the stones to come.

Yes, it is a garage model but works. It is about 25% more data efficient than the “standard transformer architecture”, regarding trainging and gets pretty decent results in basic benchmarks (arc-e, arc-c, piqa, boolq, hellaswag…). Trained in a single H100 with 30B tokens (openwebtext and fineweb-edu).