Every trained network produces two things: the weights (what it learned) and a spectral blueprint (how it arranged itself to learn it). Standard practice throws the blueprint away. Prism extracts it — the singular-value spectrum and singular directions of every weight matrix — and uses it to initialize and continuously regularize a new model. No parameters are copied. Only geometry.
The original idea, kept as the working metaphor. A spectrographic flashlight for data: shine it at information — here, a trained checkpoint — and it parses the light into attribute bands you can measure, store, and re-project. The left prism is extraction (SVD per weight group); the right prism runs backwards (initialization + regularization of a fresh model). What travels between them is geometry, never content.
| in the picture | in the math | what it is |
|---|---|---|
| white light | W ∈ ℝm×n | a trained weight matrix — content and organization mixed together, unreadable directly |
| the prism | W = U Σ Vᵀ | singular value decomposition — the one factorization that cleanly separates how much energy from which directions |
| the spectrum | σ₁ ≥ σ₂ ≥ … ≥ σᵣ | the singular values: how the model distributes representational energy. Trained models have heavy-tailed, decidedly non-random spectra |
| each color band | weight group | attention, FFN-up, FFN-down, embedding — matrices of the same role share a characteristic spectral shape, so their spectra are averaged per group |
| the photograph of the spectrum | 8 DCT coefficients | the group-averaged spectrum is a smooth decaying curve; a least-squares fit of 8 cosine terms (in inverse-softplus space) captures it to ~0.03 mean abs error. ~128 bytes total |
| polarization of each band | U, V | the singular directions: which subspaces the model decided were worth using. Large (~500 MB uncompressed), dimension-specific |
| the reversed prism | imprint · align · hold | the three write-side operations (§02): reshape the student's spectrum, rotate its directions toward the teacher's, and keep pulling it back as it trains |
Take any trained weight matrix and decompose it: W = UΣVᵀ.
The spectrum Σ says how much the matrix concentrates its energy
into few directions (trained matrices concentrate hard; random ones don't).
The directions U, V say where.
The spectrum compresses brutally well — it is a smooth curve, and eight cosine coefficients reconstruct it almost exactly. The directions do not compress; they are carried whole. That asymmetry is honest and load-bearing: "128 bytes" describes the spectral tier of the method, not all of it.
On the write side, three operations move the geometry into a fresh matrix
W′ — and the third one never stops running.
Why the third operation matters most. Overfitting, seen geometrically, is the weight matrices drifting out of the task-aligned subspace they should occupy. The mod wheel makes the transferred shape a standing constraint, not just a starting point — which is why the recipe holds its best loss for thousands of steps where the from-scratch baseline peaks and collapses.
Zoom out: a transformer is a graph of weight matrices. Prism treats every node the same way — but the spectrum is shared per color (weight group), while the directions stay per-node. Four spectral photographs describe the energy layout of the entire network; each node additionally receives its own directional rotation.
Look at what crosses the gap and what doesn't. The teacher's activations, logits, data, and learned associations stay home. What crosses is the organizational grammar: how sharply each kind of matrix concentrates energy, and which axes it concentrates it on.
This is what makes the measured results possible: a student trained on disjoint data — or a different corpus — inherits the full head start, because the head start was never about the data.
Reading the picture. Node color = weight group · ring thickness = spectral energy · dashed halo on the student = the mod wheel's standing pull. The four pooled spectrum chips are the "photograph" — the whole network's energy layout in ~128 bytes. Directions travel per-node (the heavy tier). Gradients on the right come from the student's own data only; the measured results below show it does not matter whether that data overlaps the teacher's at all.
nanoGPT Shakespeare-char, 10.65M
parameters, three seeds per run, NVIDIA L4. Artifacts live in
results/
with full loss curves, provenance, and censoring flags.
The student never sees what the teacher knew. It inherits how the teacher was shaped — and that shape turns out to be most of what the first thousand steps of training laboriously rediscover, largely independent of the data used to discover it.
And the same instrument works the other way: kept on while you finetune a trained model — anchored to its own weights — it cuts catastrophic forgetting up to ~10×. The twist is that that protection rides the singular directions, not the spectrum — so the geometry splits in two: the spectrum is transferable structure, the directions are retained content. Structure transfers; content is retained. The other direction →
method: Prism — a trained model's geometry, used two ways: (1) TRANSFER it into a
fresh model (from-scratch spectral transfer), and (2) ANCHOR a trained model
with it during FINETUNING (finetune-retention). Sections below.
(1) from-scratch spectral transfer learning
extract (read): per weight matrix W = U·Σ·Vᵀ; pool normalized singular-value
spectra per weight group {attention, ffn_up, ffn_down, embedding};
fit each pooled spectrum with 8 DCT coefficients via least squares
in inverse-softplus space (~128 bytes total); keep U,V per matrix.
apply (write): 1) spectral imprint — reshape student singular values to the
reconstructed teacher spectrum at init;
2) eigentransfer — blend student singular vectors 75% toward
teacher's, re-orthogonalize;
3) mod wheel — after every optimizer step,
W ← (1−s)·W + s·W_target, s=0.01 decaying ×0.9999/step.
not transferred: weights, activations, data, learned content.
measured (nanoGPT Shakespeare-char, 10.65M params, 3 seeds, committed artifacts):
speed: 11.8× median steps-to-baseline-best (resolved); 7.0× with schedule
matched (only spectral flags differ) — attribution control.
overfitting: recipe stable through 5,000 steps at both LRs tested; baseline
peaks ~step 1,350 then decays 1.78 → ~2.31 on all seeds.
structure: early advantage (~23% lower loss @ step 100) flat across
teacher/student data overlap 1.0 → 0.0 (difficulty-controlled).
cross-domain: advantage persists (Δloss 0.591 → 0.627) when student trains AND
is evaluated on a different corpus (Sherlock Holmes, token-JS 0.027).
teacher lever: advantage monotonic in teacher training, saturating at ~2,000
steps (where the teacher itself converges; 4k/8k flat at Δ≈+0.46);
100-step teacher is WORSE than random init (Δ −0.069).
lever evals: plain 75% full-direction blend beats grassmann pairing (which
eliminates the head start) and top-k=128 (uniformly ~0.05 worse)
at Sherlock-distance.
(2) finetune-retention (the other direction; docs/FINETUNE-RETENTION.md)
apply (write): keep the mod wheel ON during a finetune (init_from=resume),
self-anchored to the model's own pre-finetune weights, constant
pull (prism_anchor_mode=raw, decay=1.0). Entry point:
prism_finetune.py; benchmark: prism_finetune_eval.py.
measured (Shakespeare base → Sherlock finetune, 1000 steps, 3 seeds):
retention: up to ~10× less catastrophic forgetting — plain forgets +0.428
nats of the old domain, raw anchor(0.02) only +0.043; still beats
the from-scratch new-domain ceiling (learns, not frozen).
attribution: it is NOT the spectrum — anchoring spectrum-only (free directions)
forgets like plain (1.07×); a wrong-spectrum placebo HARMS (0.39×);
the raw whole-weight anchor Pareto-dominates the low-LR frontier
(~2× more retention at equal adaptation). Protection is a raw
DIRECTIONAL/weight anchor (L2-to-init / EWC-lite), not spectral.
the split: spectrum = transferable STRUCTURE (regime 1); directions = retained
CONTENT (regime 2). Structure transfers; content is retained.
(3) the arc — regimes 1 and 2 COMPOUND (docs/UNIFIED-ARC.md)
measured (3 seeds, matched Shakespeare val ~1.79, prism-vs-plain base, same anchor):
a PRISM-PRETRAINED base finetunes with ~ZERO forgetting (vs +0.05 for a plain base)
and adapts ~8% better — and a schedule-matched plain control (plain at prism's LR,
no spectral) forgets MORE, so it is the SPECTRAL GEOMETRY, not the learning rate. A
spectrally-healthy base is a better anchor; PRISM pretraining + finetuning reinforce.
(4) prior-fusion — geometry COMPOUNDS with statistics (docs/PRIOR-FUSED-PRISM.md)
fuse a fixed shared n-gram prior into the logits (product of experts: final =
model_logits + lambda*log p_ngram(next|last C chars)); model learns only the residual.
measured (1 seed, steps-to-baseline-best): PRISM alone 15x, n-gram prior alone 3.8x,
HYBRID 30x — double PRISM — reaching the best loss of all (below baseline, prism-alone,
and the n-gram floor; PRISM's geometry breaks the residual below the prior). A context-3
char n-gram already predicts Shakespeare val at ~2.57 bits/char = the neural baseline.
(literal 1000x reach-at-init is a knife-edge; needs a context-4+ prior below baseline.)
bounds: transfer results are 100-step probes; far corpus is same-language prose;
no generic-regularization control yet; 10.65M params only. Finetune-retention
shows old-domain RETENTION (not new-domain overfit prevention), far=Sherlock.
requires: a TRAINED model of the same architecture (dimension-specific directions).
evidence: github.com/timepointai/nanogpt-prism-shakespeare → results/*.json
(loss curves, provenance, censoring flags; partial runs refuse to score).