A picture
in the memory.

Can a fixed-size language memory produce an image?

The repository begins on September 24 with RWKV chat and a live architecture visualiser. We built a differentiable recurrent implementation, checked it against a reference, and connected its state to diffusion. The first image task was CIFAR-100: small images in 100 categories, paired with simple label sentences. A frozen 0.1B RWKV model processed the words; a reader and diffusion transformer learned to draw from its recurrent state.

The first answer was yes, within a narrow task. But changing the wording exposed a weakness: accuracy fell to 41.0% on held-out templates, 29.7% on synonyms, and 10.5% on descriptions. Learning a route from one phrasing to an image is easier than learning a reusable visual concept.

We then moved to precisely scored synthetic shapes. Each caption specifies one to six objects, their colour, shape, and cell in a 3×3 layout. The frozen 0.1B system reached 84.2% per-object accuracy, with performance falling as object count grew.

Next question: Is the memory losing the scene, or is the image system struggling to read it?

Evidence: results/summary.md · NOTES.md, Stages 1–2

Image supervision can teach the memory—and damage the language model.

We allowed image loss to update RWKV itself. Full tuning of the 0.1B model made synthetic scene drawing almost perfect, but its language perplexity exploded: WikiText rose from 31.16 to approximately 1.94 million. Perplexity measures how well a language model predicts text; lower is better.

LoRA adds small trainable low-rank updates to a frozen model. Combined with a KL-divergence anchor, it lets image training adjust the state while discouraging large changes to the original model’s next-token distribution.

Without the anchor, LoRA achieved 95.7% object accuracy but perplexity rose to 143.51. With too strong an anchor, image performance suffered. This established a useful tradeoff, not a guarantee of preserved conversational ability.

Next question: Will a larger language model automatically give us better scene memory?

Evidence: results/summary.md · NOTES.md, Stage 3

A bigger model was harder to read, not obviously worse at remembering.

Scaling the frozen backbone did not produce a clean improvement. The initial 7.2B diffusion-only shapes run achieved just 33.1% object accuracy, versus 84.2% for one 0.1B run. Single seeds varied substantially, and the readers differed in how they grouped the state, so this is not a controlled model-size ranking.

A compositional holdout also exposed a limit: when purple triangles were excluded from training, their accuracy was just 22.8% frozen and 55.0% with unanchored LoRA. Strong scores on familiar combinations did not solve unseen composition.

Direct scene probes recovered roughly 99% of tested facts from the frozen states. A reader trained directly toward scene targets reached 99.1% for 7.2B. That pointed toward a readout and optimisation problem: image failure alone did not demonstrate missing memory.

The 16×16 thumbnail objective supplied a short path for visual gradients to train the reader. It improved frozen 7.2B drawing dramatically. The Mac branch also tested early and late layer readers: auxiliary visual supervision rescued both, while simply using different state slices was unreliable.

Lesson: Distinguish memory capacity from the difficulty of extracting what the state already contains.

Evidence: NOTES.md, model-size ladder · sibling reader/mac results

From exact shapes to 256-pixel natural images.

COCO captions introduced open-ended scenes. Early 64×64 image experiments compared frozen states, anchored LoRA and a matched frozen continuation. The latent branch then used a pretrained VAE to represent 256×256 images compactly, allowing a larger diffusion model and finer image detail.

The final September run trained the frozen 7.2B system for 20,000 updates, then continued for 10,000 updates with anchored LoRA. Its mixture was 45% COCO, 40% text-to-image-2M examples (mainly generated images with prompts), and 15% synthetic shapes. The text anchor sampled half captions, a quarter chat, and a quarter WikiText.

Natural-image retrieval rose from 51.6% to 57.8% and project-specific FID fell from 39.70 to 34.95. WikiText perplexity increased by 5.23%, with smaller increases on caption and chat evaluation. Saved images show more coherent scenes but still contain distorted people, misplaced objects and caption failures.

There is no matched 256px frozen continuation. These are endpoint improvements across a longer continuation with a changed recipe; the entire gain cannot be assigned to LoRA.

Evidence: NOTES.md, final audit · runs/full/*/metrics.json

Can observations and extra processing improve a streaming memory?

The October branch switched to VisualRWKV 1.5B and a causal streaming protocol. PROCESS gives the recurrence another step without advancing scene time; ADVANCE advances by 0.25 seconds; SHOW renders the current state before any generated image is read back; DONE ends the sequence.

Static alignment and repair did not establish natural-image readiness. An 800-update PROCESS pilot rendered 384 images without a clear overall gain; it did not establish learned timing. Conditioning K/V adapters also gave no categorical gain on their development panel.

A three-seed reader-gradient-noise screen compared six 200-update runs. Both arms recovered 53/56 shape objects and 14/16 exact scenes, with all 48 paired shape predictions identical. Small validation-loss changes did not become visible image gains.

Stable Diffusion 1.5 produced more photographically coherent samples while still failing captions. A joint full-weight RWKV/SD1.5 comparison was attempted, then stopped before its final planned evaluation. A replacement experiment froze the inherited, already adapted SD decoder while training RWKV, reader and bridge. It completed 2,000 updates and a terminal checkpoint audit; its 40 new display images still show missing or substituted subjects.

The formal 480-image comparison remains held, with no formal images rendered. The latest parallel work reconstructs training and retrospective validation curves from saved checkpoints. These reused development panels and assistant visual reviews are not held-out or human-blinded evaluations.

Lesson: Good-looking images, runnable token semantics, and lower losses each need a separate semantic evaluation.

Evidence: scene-feedback study/continuation ledgers · final experiment receipts

Teach a small, measurable world before asking for open-ended video.

The local October 6–7 work built exact synthetic trajectories: coloured shapes move on a black 256×256 canvas. Training sequences interleave text, typed events and past image observations. Position, velocity, colour, shape and time can be measured against known targets.

The power-video system learns LoRA rank-32 adapters and new token rows, a 192-token reader, diffusion transformer, observation encoder and state head. The native language-model base and VAE remain frozen. A learned CNN compresses each image observation into four input tokens.

Training initially struggled to stabilise the visual reader and maintain autonomous motion. Later recipes used a ground-truth position hint for the diffusion loss, while inference derives that hint from recurrent memory. This is an explicit training aid: the rollout metrics below come from generated images, not target frames fed into future state.

The local lvt-v2 template differs from the original scene protocol: prompt → PROCESS ×2 → render, then ADVANCE → render. It has no SHOW token. Past generated frames enter an observation block only in feedback mode, before the next ADVANCE.

Synthetic motion success does not establish arbitrary video generation. The image renderer and the recurrent dynamics must be evaluated separately, especially when the model reads back its own drawings.

Next question: Can the state respond to “stop” or “move” at the correct time, without drifting while it reads the instruction?

Evidence: local-video-token and power-video HANDOFF.md/status.json

The instruction is part of the dynamics.

Version 5 added supervision at instruction-token anchors, continuing v4 for 1,000 updates to update 6,950. The model saw commands to stop, move, or recolour an object. We measured what its state head read immediately after the words and after PROCESS steps, then rendered held-out trajectories.

On eight rendered change episodes, memory-only post-change error fell from 18.03 to 12.30 pixels. The state’s post-PROCESS position error also improved, from 32.5 to 22.9 pixels, but remains far from the roughly 5-pixel target. More PROCESS steps were not always better: two were best in the tested sweep, and a comparable gain at zero steps points to improved sentence reading.

Reading back generated images did not improve with v5: rendered own-feedback post-change error changed from 10.34 to 10.96 pixels. Recolouring also leaned worse. There is no same-length v5 control, so the overall gains combine the new supervision with additional training.

A v3 language check on held-out WikiText, chat and caption windows found about 2–5% perplexity drift, with no language loss in video training. V4/v5 retention checks are queued. This is teacher-forced scoring, not a broad conversation benchmark.

A MOVi-A observation bridge was a negative transfer result: about 58-pixel named-object error versus a 61-pixel mean predictor, while simple persistence was much stronger. It does not demonstrate real-video generation.

Evidence: power-video-20261009-v5 final eval · MOVi bridge report

Separate instruction learning from “more training.”

A new bounded local queue began after v5. It compares a control against stronger instruction-read supervision (weights ×4 and ×16), a variable PROCESS count of one to four, and the combination of variable counts with ×4 supervision. All start from the v5 parent and use memory-only changes.

At the October 9 evening evidence check, the control, ×4, variable-count, and combined ×4/variable-count arms had finished. The final ×16 arm was still in progress. These short screens are provisional; they are not a final multi-seed conclusion.

A following queue is already scheduled: v4/v5 language retention, 32-frame horizons, late changes, PROCESS versus equal-count ordinary-token controls, and two changes including stop → resume. It then tests a control replicate, replacing PROCESS with “Okay.”, and two-change training. Results were pending at this snapshot.

The immediate issue is temporal discipline. Reading an instruction can inadvertently advance the decoded scene position even when the protocol says time is held. The next tests ask whether anchor supervision and varied processing steps reduce this error without weakening plain motion or colour changes.

Goal: A scene memory that can observe, remember, change and render while preserving the causal boundary between past observations and future targets.

Evidence: power-video-varK-20261009 · latest saved records