A missing special token didn't crash the model.It made a perfectly valid base LLM generate garbage , and the bug only became obvious when we compared the exact input IDs.
My model was loaded correctly. Prompt was correctThe seed was fixed.And Gemma 4 E2B base still repeated the same phrase forever:
This book is largely concerned with Hobbits, and from its pages,“its pages, its pages, its pages, its page”
Then I made one change:Add a <bos> token to the beginning of the input.Same model.Same prompt.Same seed.The output became coherent.
The model itself wasn't broken.The problem was in the input-tokenization path.Gemma 4 base models could reach inference without the BOS behavior expected for raw completion. The -it models didn't show the same problem because their chat templates already emit BOS.
That made this easy to misdiagnose:It looked like a model-generation problem.It was actually an input-tokenization problem.
The first fix was straightforward: enable BOS for the affected Gemma 4 base models.But review uncovered more edge cases.
A simple model-name check wasn't enough:→ Local paths don't necessarily match the Hub name→ Quantized models can have additional suffixes→ Chat templates may already emit BOS→ Adding another one creates duplicate BOS tokens
So the final implementation detects Gemma 4 from the loaded tokenizer/model configuration, rather than relying only on the model name.For raw completion, it repairs the missing BOS behavior.For chat, it checks whether the rendered prompt already contains BOS before adding another.
The final verification was more interesting than simply checking whether the test passed.For unsloth/gemma-4-E2B, with seed 3407:Before the fix:49–59 token longest repetition runsWith the fix:1–3 token longest repetition runs
The generated token IDs also matched the known-good hardcoded-BOS control exactly.We then tested 28 real tokenizers and verified that existing models — Llama, Qwen, Mistral, Phi, Gemma, etc. — remained byte-identical before and after the change.
What I learned:
1. LLM bugs don't always look like model bugs.If generation becomes repetitive or nonsensical, inspect the input IDs before changing sampling parameters or blaming the checkpoint
2. Special tokens are part of the model interfaceBOS/EOS tokens, tokenizer configuration, and chat templates aren't just preprocessing. They directly affect the sequence the model sees.
3. Fixing a tokenizer can introduce another bugAdding BOS globally might fix raw completion but create duplicate BOS when a chat template already emits it.
The right question isn't:“Should I add BOS?”It's:“Does the final sequence already contain BOS?”
4. Test token IDs, not just decoded textDecoded output can make a tokenizer bug look like a model-quality problem. Comparing the actual token sequence gives a cleaner signal.
One special token can completely change the behavior of an otherwise healthy LLM.
https://lnkd.in/gHbyK5Zthttps://github.com/unslothai/unsloth/pull/10312