LLMs have consistent response styles even without a system prompt. I measure these “behavioral fingerprints” by projecting hidden states onto contrastive axes and find that instruct fine-tuning is associated with reduced steerability on specific axes. (“Personality” = stable response style, not human-like inner states.)

https://preview.redd.it/bsz91zsyzuig1.png?width=800&format=png&auto=webp&s=b8204972794c46d48f6c596404000ca73f3abef7

Contributions:

• A contrastive probing method that extracts 7 behavioral axes (warm/cold, verbose/concise, etc.) from hidden states, with IQR normalization for cross-model comparison
• Stability and reproducibility metrics: test-retest ICC > 0.75 for all 42 model-axis pairs, cross-provider delta < 0.05, length confound control (6/7 axes clean)
• “Dead zones” — axes where models failed to reliably follow style instructions across 5 tested prompt formulations, validated by external judge (Claude Opus, pooled r = 0.38 [0.29, 0.47])

Findings:

• Each model has a distinct fingerprint. Llama 3.1 8B Instruct is the most constrained (benchmark pass rate 60%), DeepSeek LLM 7B Chat the most independent (eff. dim = 3.66 of 7)
• Base-vs-instruct comparison across 5 organizations shows instruct versions consistently have lower behavioral variability
• Dead zones are stable, not noisy — models reliably reproduce the same constrained behavior across seeds and the tested prompt variants

Code: github.com/yunoshev/mood-axis | Which models should I test next? Currently limited to 7-9B.

Details below. Extended discussion on r/LocalLLaMA*:* original post

Key Results

1. Distinct fingerprints

https://preview.redd.it/i884c3zmzuig1.png?width=2280&format=png&auto=webp&s=f2b96680b60b663c663593760cff8ec20dc716db

Each model’s default profile across 7 axes. No system prompt. Values = hidden-state projections normalized by calibration IQR.

• DeepSeek LLM 7B Chat: verbose (+1.00), confident (+0.97), proactive (+1.00) — ceiling on 3 axes
• Llama 3.1 8B Instruct: all |mean| < 0.10 — flattest profile (most constrained on benchmarks: pass rate 60%)
• Yi 1.5 9B Chat: slightly cold (−0.24), patient (+0.35), confident (+0.46), verbose (+0.48) — differentiated profile
• Qwen 2.5 7B Instruct: formal (+0.42), cautious (−0.36), proactive (+0.47)

2. Instruct models show reduced behavioral dimensionality

Observation. PCA on baseline projection matrices reveals a spectrum of behavioral dimensionality. Gemma 2 9B IT shows the highest concentration (PC1 = 87.9%), likely driven by variable response length rather than behavioral collapse. Axis vectors are geometrically near-orthogonal (low |cos|) but projections are behaviorally correlated (higher |r|).

Interpretation. This gap is consistent with fine-tuning constraining how models utilize their representation capacity — but alternative explanations exist: inherent semantic correlations between axes, SFT data distribution, chat template effects, or decoding strategy could all contribute. We observe the pattern across 6 models from 5 organizations, but cannot isolate which component of the instruct pipeline drives it.

Length confound control. Response length could drive spurious axis correlations. I computed per-model Pearson r between n_tokens and each axis projection across 30 baseline questions. Result: 6/7 axes are clean (mean |r| < 0.3 across models). Only verbose/concise is partially confounded (mean r = 0.50), which is expected — longer responses literally are more verbose. Cross-axis correlations drop only −7.7% after regressing out length, confirming behavioral bundling is not a length artifact.

Base versions of 5 models (Llama, Yi, Qwen, Mistral, Gemma) show higher variability on most axes than their instruct counterparts. Most extreme: verbose/concise std ratio = 0.13 (87% lower in instruct). All 5 organizations show the same direction, though this is observational — base and instruct models differ in many ways beyond alignment. Gemma base can’t distinguish empathetic/analytical or formal/casual at all (50% accuracy = chance), but the instruct version does — suggesting these particular axes may reflect distinctions introduced during fine-tuning rather than suppressed by it.

https://preview.redd.it/m56aq8aszuig1.png?width=2400&format=png&auto=webp&s=21e07f04f7891b565f087b0b5901b9942091ddd8

[IMAGE: pca_calibration_contrast — PCA scatter, Qwen vs Yi]

PCA of calibration hidden states. Left: Qwen 2.5 7B (d’ = 5.0–12.0) — diverse axis directions, poles clearly separated. Right: Yi 1.5 9B (d’ = 2.2–5.4) — lower separability but all axes still discriminate.

3. Dead zones and the ICC dissociation

I introduce a composite Dead Zone Severity metric (0 = healthy, 1 = dead) combining calibration accuracy (30%), d’ (30%), stability cosine (20%), and baseline SNR (20%). The weights are heuristic — I chose them to balance discrimination, stability, and effect size, but other weightings could shift individual model rankings. Three dead zone types: hard (fine-tuning suppresses differentiation), soft (unstable across calibration sets), and asymmetric (model follows instructions in only one direction — e.g., Llama achieves 100% for “be concise” but 0% for “be verbose”).

An interesting pattern is the dissociation between reliability and validity: mean ICC (test-retest, 5 seeds) is 0.91–0.99 across models, all 42 model-axis pairs exceed 0.75 — but Llama’s benchmark pass rate is 60%. This is partly expected (a model that always outputs neutral will have high ICC and low benchmark scores), but the degree of dissociation varies across models, suggesting it captures something beyond trivial low-variance cases.

Text-level validation. I computed text-level compliance metrics (token count, hedging markers, emotion words) between opposite calibration poles across all 6 models × 7 axes. Spearman correlation between calibration accuracy and text-level effect size (Cohen’s d): r = 0.47, p = 0.002 (n = 42). Caveat: text metrics and hidden states are not fully independent — both are derived from the same generated text, so this correlation partly reflects consistency between two views of the same data rather than independent validation. Still, it confirms dead zones manifest in observable text, not just internal representations.

External validation (Claude Opus 4.6 as independent judge). To address the circularity concern above, I had Claude Opus rate 48 baseline responses (8 per model, no system prompt) on all 7 axes using a −2 to +2 scale, based only on text — no access to hidden states or knowledge of our measurement method. Per-axis Spearman correlations with hidden-state projections:

3/7 axes reach p < 0.05, with 2 robust under bootstrap (warm/cold and formal/casual: 95% CI excludes 0). Pooled r = 0.38 [0.29, 0.47 bootstrap 95% CI]. Leave-one-model-out: pooled r ranges from +0.30 to +0.58 — no single model drives the result. The negative correlation on proactive_reluctant is informative: it’s driven by Llama (dead zone — hidden states say “reluctant” while text is structured and proactive) and DeepSeek (ceiling — projections saturate at +1.00 while Claude sees neutral text). This is exactly the dead zone phenomenon: hidden state projections and observable text diverge on constrained axes. verbose_concise shows no correlation — Claude rates “verbosity” qualitatively while our projection tracks length-correlated hidden state variation.

Prompt robustness test (5 formulations × 3 models × 3 axes) confirms dead zones persist across phrasings.

Method (4 steps)

1. Calibrate: Show neutral questions with contrastive instructions (“be warm” / “be cold”). Extract hidden states from last 4 layers of assistant-generated tokens only. Axis = normalize(tmean(warm) - tmean(cold)) (10%-trimmed mean, IQR normalization).
2. Measure: Project any response onto axis. IQR-normalized values in [-1, +1].
3. Validate: Calibration accuracy 93-100% (4/6 models). Axis stability: cosine 0.69 across 3 independent calibration sets. Test-retest: mean ICC 0.91–0.99 across models, all 42 pairs exceed 0.75 (5 seeds). Scaling curve: axis stabilizes at n ≈ 15 questions (cosine > 0.93 to full-30 reference), holdout accuracy flat across all n.
4. Reproduce: Two cloud providers (RunPod RTX 4090, Vast.ai RTX 3090), max delta < 0.05.

Config chosen for cross-model robustness via 150+ configuration ablation (layer selection × token aggregation × weighting). Not optimal per-model, but the only config that works 85-100% on all 5 ablated models.

Limitations

• AI-generated dataset: 310 English questions by Claude Opus 4.6, curated by author. No psychometric instruments or crowdsourcing
• Partial external validation: Claude Opus as independent judge — 2/7 axes robust under bootstrap (warm/cold, formal/casual; 95% CI excludes 0), 1 marginal (patient/irritated), 4 not validated. Pooled r = 0.38 [0.29, 0.47]. Text-level validation (r = 0.47) is internal consistency, not ground truth
• Length confound: 6/7 axes are clean (mean |r| < 0.3 with n_tokens), but verbose/concise is partially confounded (r = 0.50) and should be interpreted as partly a length proxy rather than a pure stylistic dimension. External validation confirms this: Claude’s qualitative verbosity ratings don’t correlate with our projection (r = 0.04). Gemma is an outlier with strong length correlations on multiple axes. Cross-correlations drop ~8% after length residualization
• Single chat template & decoding per model (temp=0.7, top_p=0.9 for all). Cross-model comparisons are fair within this regime, but absolute profiles could shift under different decoding — a temperature sweep is planned future work
• Full pipeline on 7–9B models only; one 14B model (Phi-4) evaluated with shortened pipeline. Thinking mode tested on one model only
• Axes are behaviorally correlated (eff. dim 1.3–3.7 across models). 4/7 axes highly stable (cosine > 0.7); 2 weaker (0.55-0.60)
• Dead Zone Severity weights (30/30/20/20) are heuristic. Different weights could shift model rankings
• DeepSeek has the highest effective dimensionality (3.66) but is fundamentally unstable across calibration sets (mean stability cosine 0.53). Independence ≠ stability: its axes capture diverse behavioral dimensions, but those dimensions shift between calibrations
• Gemma’s high PC1 (87.9%) likely driven by response length variation, not behavioral collapse

More details in the repo README: conflict drift (20 scenarios × 12 turns), cross-axis correlations, full methodology.

Follow-up: Phi-4, Qwen3, and Thinking Mode

After posting this work on r/LocalLLaMA, several people asked about newer models. I ran a shortened pipeline (calibration + baseline + benchmark, no drift/stability) on two additional models in ~30 min on 2×H100 (~$6):

Phi-4 (Microsoft, 14B) — first model outside the 7–9B range

The most extreme cautious/reluctant profile in the entire set: cold (−0.51), highly cautious (−0.85), strongly reluctant (−0.93). Polar opposite of DeepSeek on confidence and proactivity axes. Verbose/concise is in a dead zone (+0.01). Benchmark: 3/9 — Phi-4 can only decrease along axes (be cold, be cautious, be concise) but fails to shift in the positive direction, suggesting a strong “conservative” alignment prior.

Qwen3-8B vs Qwen 2.5 7B — generational fingerprint shift

Same family, one generation apart. Two axes invert: confident/cautious flips from −0.36 to +0.38 (Δ = +0.74), formal/casual flips from +0.42 to −0.26 (Δ = −0.67). Proactive/reluctant stays identical (+0.47 → +0.45). Qwen3 achieves the highest benchmark pass rate in the full set (7/9). Behavioral fingerprints are not stable across model generations, but some axes are more persistent than others within a family.

Thinking vs non-thinking mode (Qwen3-8B)

Same weights, same calibration axes — only difference is enable_thinking=True. Initial results (max_new_tokens=384) appeared to show a confidence drop (Δ = −0.26), but 28/30 responses were 100% <think> tokens — the model never finished reasoning. That comparison was effectively internal monologue vs actual response.

Control experiment (max_new_tokens=4096, n=10, 100% visible responses): comparing visible response after thinking vs non-thinking response on the same questions.

The original confidence drop reverses sign when properly controlled — thinking mode makes the model more confident, not less. The largest genuine shifts are on proactivity (less proactive) and verbosity (less verbose after thinking). This demonstrates the importance of separating <think> token artifacts from actual behavioral shifts.

Caveats: n=10 (PoC subset), single model, decay-weighted aggregation means only the last ~50 tokens of each segment contribute to projections.

Reproducing

git clone https://github.com/yunoshev/mood-axis.git cd mood-axis && pip install -r requirements.txt python scripts/run_app.py --model Qwen/Qwen2.5-7B-Instruct 

Pre-computed axes included — measure any model’s fingerprint without re-running calibration.

What I’d love feedback on:

• Is the geometric-vs-behavioral dissociation (low |cos|, high |r|) evidence for alignment-induced compression, or could it reflect inherent semantic correlations between the axes?
• External validation confirms 2/7 axes (bootstrap CI excludes 0) but 5 remain unvalidated. What would be a convincing validation for axes like confident/cautious or empathetic/analytical?
• The Dead Zone Severity metric weights are heuristic (30/30/20/20). What principled approach would you use to combine calibration accuracy, d’, stability, and SNR?
• Length confound: verbose/concise is the one axis clearly correlated with response length. Is this a problem or expected tautology?

P.S. I have a full paper version (LaTeX, ~20 pages with methodology, ablations, reproducibility details). Do you think this is worth putting on arXiv? If so, I’d be grateful for an endorsement for cs.CL or cs.LG — happy to share the draft via DM.