live demo — runs on my GPU
Spectral Text Encoder
A sentence embedding you can listen to the shape of: a transformer reads your sentence and emits, for every token, a handful of sine waves — amplitude, frequency, phase. They form internal frequency channels that the best model adds into one scalar waveform before the contrastive loss. Type a sentence and watch the real model output take shape.
Up to 20 words. The encoder runs on the same GPU as the wave sandbox — they share one inference worker.
1 · tokenize — GPT-2 BPE splits the sentence into pieces
2 · encode — each token emits 6 sine waves (amplitude, frequency, phase)
Each card is auto-scaled — the peak value shows its true loudness. Click a card to unfold its channels one by one; hover it to see that token’s contribution inside the final signal.
3 · sum — add 24 channels / √24 into the model output
The white curve is the exact 2,048-sample model output. Click the plot to inspect the channels that are added to produce it.
+ raw sine parameters (A · f · φ)
technical report
Text embeddings as sums of sine waves
Full training code, checkpoints and experiment logs: github.com/jackwarren430/spectral-text-encoder
The idea
Ordinary sentence embeddings are opaque: a pooled 768-dimensional vector with no internal structure. This project forces the representation through a physically structured bottleneck instead. A bidirectional transformer reads the sentence, and every token slot emits d_sine triples (A, f, φ) — amplitude, frequency, phase. The current best model assigns those sines to 24 fixed frequency bands. First it sums across tokens inside each band, producing 24 internal channel waveforms. Then it performs a final, variance-preserving channel readout signal(t) = Σc channelc(t) / √24. That one 2,048-sample waveform is L2-normalized and sent directly into the contrastive loss. There is no hidden pooled vector and no per-channel shortcut to the loss.
Why bother? Two reasons. A waveform is interpretable — you can inspect every token and internal band, as this page does, while still training on a genuinely one-channel symbol. And waves suggest a goal no pooled vector has: superposition. If embeddings are signals, adding two sentences’ waves should approximate the wave of both sentences together.
Does it work?
The served checkpoint is the 43,000-step model from the six-source run: AllNLI, Quora and Stack Exchange duplicates, AltLex, COCO captions, and sentence compression. It was trained CLIP-style with symmetric InfoNCE at batch 512. Crucially, synthesis does not stop at the 24 convenient internal bands: they are added into the scalar waveform before L2 normalization, similarity logits, and loss. Per-channel InfoNCE is disabled, so the optimizer never receives privileged access to the separated bands.
At step 43k, STS-B validation is Spearman 74.23 / Pearson 73.80. The best STS-B validation point was step 20k at 74.93 / 74.66; retrieval continues to improve, reaching 72.38% at the served checkpoint and 72.43% at the final logged validation. That retrieval number should not be compared directly with earlier, easier three-dataset validation runs. Separately, the served checkpoint scores as follows on the shared held-out semantic-textual-similarity test suite (correlations ×100):
| metric | STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R | avg |
|---|---|---|---|---|---|---|---|---|
| Spearman ρ | 49.90 | 48.78 | 52.85 | 68.96 | 68.72 | 66.18 | 65.20 | 60.08 |
| Pearson r | 52.46 | 48.76 | 56.63 | 68.69 | 67.98 | 67.47 | 76.70 | 62.67 |
On the separate shared held-out test suite, STS-B Spearman is 66.18 and the seven-task mean is 60.08, versus 62.3 and 50.2 for the former six-channel website model. The important result is not merely that 24 internal channels work; it is that their single summed waveform retains the semantic improvement after every channel boundary has been removed from the representation seen by the loss.
What changed from the earlier model
The former site checkpoint concatenated six separately visible waveforms into a 12,288-value embedding. The new model quadruples the internal frequency bands but collapses them to one 2,048-value signal before training sees the representation. The rows below provide context; retrieval validation differs by data mix and batch size, while the held-out STS columns are directly comparable.
| head | emb. dims | steps | val acc | STS-B val ρ / r | STS-B test ρ | STS avg test ρ |
|---|---|---|---|---|---|---|
| max-pool baseline | 768 | 6k | 89.0% @1024 | — | 70.4 | 63.1 |
| former best: 6 concatenated channels | 12,288 | 38k | 80.3% @512 | — | 62.3 | 50.2 |
| served model: 24 → 1 summed channel | 2,048 | 43k | 72.38% @512 | 74.23 / 73.80 | 66.18 | 60.08 |
A conventional pooled baseline still wins absolute STS quality, so the structured bottleneck has a measurable cost. But the current scalar spectral model narrows that gap while reducing the observable waveform from six channels to one. The 24 separated bands shown in the demo are an inspection of how that scalar is synthesized, not twelve dimensions secretly supplied to the objective.
The three models above
Best model
24 sines/token · STS-B val ρ74.23 / r73.80
The 43k-step, six-dataset checkpoint. Twenty-four anchored frequency channels are summed (and divided by sqrt(24)) into the actual one-channel model output before loss.
Extreme bottleneck
2 sines per token · 125M params · STS-B 56.7
Legacy BERT-base-sized trunk forced through only two sine channels. Its model output remains multichannel rather than using the new scalar channel sum.
Nearly additive
8 sines per token · 64M params · most compositional
Legacy model trained with a much heavier per-channel loss and stopped early: semantically weaker, but its channel waveforms are the most compositional of the three.
How this demo runs
All three encoders live on the GPU of the DGX Spark that serves this website, stripped of their training-time decoders (a synthesize click is one ~5 ms encoder forward). Requests share a single inference worker with the wave-model sandbox, so only one model ever touches the GPU at a time, and both demos draw on the same GPU duty-cycle budget — heavy use slows the sandbox’s frame rate rather than piling up work. The server returns full-precision sine parameters plus the best model’s exact 2,048 fp32 scalar samples. Your browser uses the samples for the completed high-density curve and the parameters for the animated buildup and 24-channel exploded view. That keeps every internal band inspectable without substituting a browser approximation for the observable model output. One serving subtlety: the models run in full fp32, because half precision perturbs each sine’s frequency by a fraction of a hertz — enough to drift the phases apart over the 1-second window and visibly change the interference pattern that is the embedding.