The repetition trap in grammar-constrained decoding

I disqualified a 30B vision model for being too slow, and I was wrong about why. On my DGX Spark it timed out past 21 minutes on trivial documents, with no reading gain over its 8B sibling to show for the wait, so I binned it. Then I re-ran the same task on an RTX 5090 and successful runs finished in 22 to 140 seconds, so speed was never the real problem. The model still fails, because under grammar-constrained decoding it drops into a repetition loop that no sampler setting I tried gets it out of, and on the Spark that loop had just run slowly enough to look like a timeout.

The task is structured extraction from dense, text-heavy document images, part of a pipeline I work on: hand the model a rendered page and a Pydantic schema, get back a JSON list of items, each with a name, a couple of numeric fields and some flags. The work splits into two jobs - reading the fine print off the page, and driving the schema - and this post is mostly about the second. Locally, driving the schema means grammar-constrained decoding: llama.cpp compiles the JSON schema into a GBNF grammar and, at each step, blocks any token that would break it, so the model can only emit conforming output. That should guarantee valid JSON. What Qwen3-VL-30B-A3B actually produced looked like this:

{"items": [
  {"name": "REGION 4", "param_1": 2.7, "flagged": false},
  {"name": "REGION 4", "param_1": 2.7, "flagged": false},
  {"name": "REGION 4", "param_1": 2.7, "flagged": false},
  ...

One valid item, repeated until the context window or the token cap ran out - truncated invalid JSON after anywhere from 90 seconds to nearly 18 minutes.

The Qwen family was in the pipeline for its vision encoder rather than its parameter count. Most VLM encoders, the ones in much bigger models included, resize the page into a fixed budget of vision tokens, so fine print blurs away in the downsample before the language model ever sees it. Qwen3-VL’s encoder runs at native resolution and scales its token count with the image, and its DeepStack design feeds ViT features from several depths into the language model instead of only the last one, which keeps small labels in a dense diagram legible. That’s the property the reading job depends on, and it’s why I was keen for the 30B to work.

The sweep #

I swept the obvious levers on the 5090 (32 GB, llama.cpp via LM Studio), two trivial pages per config. The documents come from a working set I can’t share, so the numbers throughout are aggregates:

No configuration got both pages out clean, and the failing page changed between runs with the same settings. It isn’t the page, the quant or the context: the failure is stochastic. Two pages per config is a small sweep, but the 37-page runs later in the post show the same failure at scale.

The repeat-penalty row is the important one. Push the penalty high enough to break the loop and the output turns schema-valid with zero items in it. If you score runs by schema validity, that config reads 2/2 while extracting nothing. Score item counts rather than validity - a grammar makes valid output nearly free, so a valid result tells you nothing about whether the model actually extracted anything.

60k is well inside this model’s 262k native window, so long-context degradation isn’t the cause either. All the window size changed was fail speed: at 60k a loop self-terminates on context exhaustion in two to three minutes, at 262k it ran to the 32,000-token cap and took almost 18. Load a small context for extraction work, a big one just makes the failures slower.

Why no setting fixes it #

The schema demands repetition. Every item carries the same keys - a name, the same numeric fields, the same flags - so a correct answer for a 20-item page is twenty near-identical structures in a row. A token-level repetition penalty can’t tell “emitting the same item forever” from “listing the eighth distinct item”, because at the token level they look alike.

That brackets the model with no reliable middle. Penalty too low, and near-greedy decoding (temperature 0.1, so the model almost always takes its top token) locks onto one valid object inside the schema’s unbounded items array and repeats it. Penalty high enough to break that, and it also suppresses the structural repetition the schema requires, so the model stops emitting items at all. Take the grammar off entirely and you lose conformance instead - the grammar was the only thing keeping the JSON shaped.

                    grammar on?
                   /           \
                yes             no
                 |               |
         repeat penalty?    non-conformant JSON
            /       \       (or the schema definition
          low       high     echoed back)
           |          |
       loops on    schema-valid
       one item    but empty -
       forever     zero items

It isn’t one broken model #

I ran the dense 8B over a 37-page set and it looped on 15 of them under grammar decode - the same failure with no MoE involved, so this spans the model family rather than one architecture.

Part of it may be an implementation defect. llama.cpp #16960 reports this exact 30B in Q4_K_M GGUF producing degenerate output in ordinary generation while the 8B works, but it’s a single unconfirmed report on a CPU backend and has since been closed - a hint at a MoE inference problem, not a diagnosis. The structural part is better documented: the repetition thread on the text sibling Qwen3-30B-A3B reports loops even with the recommended sampler settings, and the structured-output literature documents infinite repetition under JSON-schema constrained decoding as a known failure that hits smaller models harder, and calls repetition penalties an ad-hoc fix, because the real problem is that constrained decoding distorts the model’s output distribution. The failure follows the mechanism rather than the library: the same paper notes outlines had its own infinite-whitespace variant of the loop, so switching serving stacks doesn’t dodge it.

One caveat: I never ran Qwen’s full recommended sampler stack together (temp 0.7, top_p 0.8, top_k 20, repeat 1.05). My extraction path runs near-greedy at temp 0.1, which is the regime most prone to loops, and top_k in particular trims the candidate pool in a way that can damp them. Given the reports above say the repetition survives the recommended settings, I doubt it rescues the model, but it’s the one setting I didn’t test.

Why gemma mostly doesn’t loop here #

gemma-4-31b-qat ran the same 37 pages at 95% valid with real content - 35 of 37, the best local result - while reading fine text worse: 0.74 on exact transcription of small printed text against the 8B’s 0.92. Two things explain the gap.

The first is vision tokens. Gemma’s encoder hands the language model far fewer of them - about 11k input tokens for a full page against Qwen’s roughly 44k - so it reads the fine print worse, and it produces shorter constrained output that rarely locks into the loop.

The second is my schema, because gemma isn’t immune. My two failures were real loops, and other people report the same failure: an ollama issue documents gemma-4-31b looping during grammar-constrained JSON generation on long free-text string fields, with repeat penalty at 1.0, 1.15 and 1.5 failing identically, and a deepmind issue reports the same collapse in both the 31B dense and 26B MoE variants, most reliably under constrained JSON. My extraction schema keeps string fields short - labels, not paragraphs - which plausibly explains why gemma survived here where their long free-text fields failed.

Those external reports also rule out a vendor quirk: the trap spans gemma as well as the Qwen family, and their repeat-penalty sweep matches mine. Across every local model I tested, reliability and reading detail traded off, and none delivered both.

What I’d do instead #

Three moves fall out of this, in order of effort.

Use a hosted model for the schema step. On the same 37 pages Claude Haiku went 36/37 valid with real item counts and Sonnet 34/37, with no grammar needed, and it works today.

Keep local models on the reading job. The 8B beat every hosted tier I tested at reading small printed text (0.92 exact transcription vs Haiku’s 0.62, and the gap widens on rotated text). Reading and driving a schema are different jobs, and local currently wins one of them.

If you want extraction local, fine-tune rather than tune samplers. Fine-tune the 8B on pages paired with gold JSON - distilled from the hosted model’s outputs - so it emits conformant JSON in free generation and the grammar comes off entirely, which sidesteps the loop instead of fighting it. A 32 GB card can QLoRA an 8B, and Unsloth supports Qwen3-VL. It only pays at volume, so it’s my roadmap item rather than my next step.

The grammar guarantees every token is valid and says nothing about whether the model ever stops. When your schema mandates repeated structure, that gap is the failure mode, and no amount of hardware closes it. The 5090 didn’t fix anything the Spark got wrong, it just surfaced the same failure in seconds instead of letting it hide behind a 21-minute timeout.