A 3.5-billion-parameter model that scores like a much larger one — not by adding weights, but by running the weights it already has more times. That is the result behind Huginn, the model from Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. Trained on 800 billion tokens at 3.5B parameters, it improves on math and coding benchmarks as you let it iterate, up to a compute load equivalent to a 50-billion-parameter model. Nothing about the checkpoint changes between the cheap answer and the expensive one. You just turn a loop more times.
That is the whole idea, and it opens a scaling axis most deployed systems never touch: depth spent at inference, decoupled from both parameter count and training data.
Two ways to make a model think longer
There are two ways to spend more compute on a hard question.
The familiar one is chain-of-thought. The model reasons out loud, emitting intermediate tokens, and each token it writes becomes context it reads back. Every extra step lengthens the context window and grows the KV cache. You pay in sequence length.
Recurrent depth spends compute the other way. Instead of writing its reasoning down, the model iterates a shared block of layers in latent space, refining a hidden state without emitting anything. Thinking gets deeper rather than longer. Because nothing is verbalized, the context window stays short and the KV cache does not balloon — the recurrent-depth paper notes exactly this memory advantage over chain-of-thought, which needs very long contexts and specialized training to match. And there is no reasoning-trace dataset to curate: the loop is trained end-to-end on ordinary next-token prediction.
Anatomy of the loop
The architecture splits into three functional groups:
- Prelude (P): several transformer layers that embed the input into a latent space, producing an embedding
e.
- Core recurrent block (R): the shared unit that does the actual thinking. This is the part that loops.
- Coda (C): a few layers that un-embed the final state back to token space, plus the prediction head.
The state starts as noise — s₀ ∼ 𝒩(0, σ²I) — and each iteration updates it. The one detail that makes or breaks the design: the embedded input e is re-injected at every step, concatenated with the current state s_i and mapped back down by an adapter matrix A. Feed e only at the start and the iteration drifts and destabilizes; feed it every time and the loop stays anchored to the question it is supposed to be answering.
Training does not fix the number of iterations. It samples a count per batch from a log-normal Poisson distribution — the large model trains around a mean of 32 loops. Because depth is randomized during training, the model learns to produce a usable state at any iteration count, which is what lets you dial compute at test time. In the paper's evaluations that means running the same weights at r = 4, 8, 16, 32, or 64 and watching accuracy climb.
# One decode step of a recurrent-depth model (sketch)
e = prelude(tokens) # embed once
s = torch.randn_like(e) * sigma # s0 ~ N(0, sigma^2)
for i in range(max_iters):
s_prev = s
s = core(adapter(cat([s, e]))) # re-inject e every loop
if kl(logits(s), logits(s_prev)) < 5e-4:
break # converged: stop early
next_token = coda(s).argmax(-1)
The exit is a router in disguise
The break above is where this gets interesting. The model does not have to loop a fixed number of times. It compares the token distribution between two successive steps, and when the KL divergence drops below 5e-4, it stops, samples the token, and moves on. Nothing was trained to produce this — the adaptive exit falls out for free from training on variable depths.
The consequence is per-token adaptive compute. An easy token — a closing bracket, a predictable word — converges in a few iterations and exits. A token that sits on the answer to a hard arithmetic step keeps looping. Compute flows to where the uncertainty is, without anyone writing a rule for it.
Once you see the exit as a decision about how much depth this token deserves, it stops looking like a stopping criterion and starts looking like a router.
MoE routing: the same idea, one axis over
Mixture-of-experts routing answers "which sub-network should process this token?" Recurrent depth raises a sibling question: "how many times should this token be processed?" Both are learned, per-token allocation decisions — one over width, one over depth.
Mixture-of-Recursions makes the depth version explicit. It replaces the free-running KL exit with lightweight routers that assign each token a recursion depth directly, in the expert-choice style familiar from sparse models. The payoffs are the ones you would predict: quadratic attention only runs over tokens still active at a given depth, and a KV-sharing variant reuses the key/value pairs from the first recursion so deeper loops cost almost nothing in memory. Across 135M to 1.7B parameters it lands on a better compute-versus-quality frontier than a standard transformer at equal training FLOPs — lower perplexity, higher throughput.
Nothing stops you from combining both axes. Put a sparse MoE feed-forward inside the recurrent block, and each loop routes its tokens to a handful of experts while the loop itself decides how many times to run. Width routing and depth routing compose, and recent looped-MoE work is chasing exactly that product. The unifying insight is that "loop again" and "pick an expert" are the same primitive — a small router spending a compute budget where the input warrants it — applied to two different dimensions of the network.
What you give up
Latent reasoning is not free of tradeoffs. Because the thinking never becomes tokens, you cannot read it. A follow-up study, Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer, probes whether those internal iterations correspond to anything like the interpretable steps of a written chain-of-thought, and the honest answer is: not cleanly. You trade the auditability of a visible reasoning trace for the efficiency of an invisible one.
Adaptive exits also make latency variable. Different tokens loop different numbers of times, so a batch runs until its slowest, deepest member converges — and naive implementations waste the saved compute waiting. Extracting the throughput win takes real scheduling work.
The takeaway
If you only ever scale a model by making it bigger or feeding it more data, you are leaving a third dial untouched. Recurrent depth turns inference compute into a tunable knob on a fixed checkpoint: ship one set of weights, then choose per request — or per token — how hard to think. Before you reach for a larger model to squeeze out a few points on a reasoning benchmark, it is worth asking whether the model you have simply needs to loop a few more times, and whether a router could decide when.
Sources: Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach (arXiv:2502.05171), Mixture-of-Recursions (arXiv:2507.10524), Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer (arXiv:2507.02199)