Teaching a 2B model to answer AWS questions in Malay
TL;DR: We finetuned Qwen3.5-2B with unsloth LoRA on 18k paraphrase-linked Malay AWS QA pairs, evaluated with a strict completely-correct LLM judge on 2,661 unseen questions, and took it from 4.1% → 91.2%, while mapping exactly where flexibility dies, where accuracy comes from, and which failures remain dangerous.
| Chapter | |
|---|---|
| 1. Setup & the judge | task, data, strict TRUE/FALSE evaluation, why trust the numbers |
| 2. The sweep | where the knowledge lives (modules, layers, rank, LR) |
| 3. Data vs hyperparams | paraphrase diversity, epochs, completion-only |
| 4. The flexibility frontier | verbatim collapse, balancing acts, NEFTune |
| 5. Failures & the guardrail | hallucination quantified, grounding check |
| 6. What we shipped | final recipe, notebook, honest limits |
| Model responses by service | same question, four models, real generations |
Headline numbers
| Model | Strict acc (full unseen set) | Median similarity to reference |
|---|---|---|
| Base Qwen3.5-2B | 4.1% | — |
| D: 1 epoch, completion-only, +synthetic K5 | 35.7% | 0.29 (rephrases) |
| C: 2 epochs, full-sequence loss | 86.8% | 1.00 (verbatim) |
| H4: H3 + hard-service data | 91.2 / 91.0 (two seeds) | 1.00 (verbatim) |
| H3: 2 epochs, completion-only | 87.3% | 1.00 (verbatim) |
Every number on this site comes from artifacts in this repository
(results/), generated by the experiments logged in order below.