Chapter 4: Verbatim collapse and the flexibility frontier
What the second epoch learns
C/H3's outputs are correct (86–89%) and 83% of them reproduce the reference at >99% similarity. Median similarity: 1.000. The model maps any question paraphrase to the canonical answer text. The mapping did generalize across phrasings (recall: 0/2,661 eval questions appear in training), but the answer side never rephrases.
Why: the canonical answer appears ~7× per cluster (once per question paraphrase), each synthetic variant 1×. Majority surface wins; the second epoch hardens it.
Mapping the frontier (similarity-banded eval)
We added flexibility metrics: candidate/reference similarity distribution, and accuracy per similarity band. The frontier, all on the same judge:
| Model | Acc | Median sim | %verbatim | Mechanism |
|---|---|---|---|---|
| H3 (2ep, comp-only) | 89.2% | 1.000 | 84% | recall |
| C + softening prompt | 81.9% | 1.000 | 78% | prompt can't unlock it |
| D (1ep, comp-only, K5) | 38.8% | 0.372 | 29% | half-remembered rephrase |
| G2 (D-data + balanced answers) | 32.1% | 0.241 | 19% | balanced mixture |
| G1 (canonical ×2) | 31.8% | 0.262 | 19% | half-balanced |
| F1a/F1b (fully balanced) | 16.6–17.5% | ~0.15 | 5% | full diversity |
The frontier is monotone and steep: every step toward diversity costs accuracy under a strict judge, because a rephrased answer must retain every fact to score TRUE.
What didn't work
- Inference-time softening (system prompt "answer in your own words"): changed the wording, walked back to canonical anyway. It cost 4pp and bought no flexibility. The verbatim lock survives prompting.
- Balancing variant frequency (each answer surface seen equally): succeeded wildly at diversity (median sim 0.15) and collapsed accuracy to 17%. The judge fails nearly every free rephrase.
- NEFTune from epoch 2 (noise on embeddings in the second epoch only): our first wave ran with the noise silently inactive. We caught it by missing activation markers and re-ran with verified markers. With activation confirmed, the dose-response is a cliff, not a dial:
| α | acc | median sim | %verbatim |
|---|---|---|---|
| 0 (control H3) | 89.2% | 1.000 | 84% |
| 5 | 88.3% | 1.000 | 82% |
| 10 | 78.7% | 1.000 | 74% |
| 20 | 27.7% | 0.333 | 14% |
The mechanism works: α=20 achieves D-level flexibility from a 2-epoch base. But no dose lands mid-frontier, and α=20's accuracy falls below the 1-epoch flexible point (27.7 vs 38.8%). Embedding noise cannot buy the middle; the frontier there belongs to data/schedule choices (Chapter 3-4), or to rephrasing after retrieval.
The structural insight
Strict binary judging and free rephrasing are nearly incompatible by construction: the reference is the fact list, and paraphrase-model outputs usually drop something the judge checks for. If your product needs flexible phrasing, either the judge must become a fact-coverage rubric (extract the reference's facts, check each in the candidate), or you rephrase after retrieval with a bigger model. We chose to ship accuracy and guard the failure mode (Chapter 5).
Artifacts: results/exp003/{F1*,G*,H*,C_softprompt}/