← blog.sipa-os.org

Eval · EXP-040 / EXP-041

Zero-Shot Qwen3-8B: 90% → 94% After a Tiny LoRA Tune. One Fact Went 20/20 to 12/20.

Zero-shot Qwen3-8B on a 10-item honesty gate: 90%. After a tiny LoRA tune (194 examples, rank 8, 3 epochs): 94%.

Also after that same tune: a fact it got right 20/20 times before the tune, it now gets right 12/20.

How the baseline almost got skipped entirely. Two fine-tune jobs went out on Fireworks before a baseline had actually been run — caught mid-session when asked directly: did we eval before, or just tune? Answer was no, just tune. So: baseline first, honestly, after the fact, not folded quietly into the writeup as if it had happened in order.

Then a real infra wall: Fireworks won’t let you download a trained LoRA’s weights, and won’t serve either base model (Qwen3-8B, Llama-3.1-8B-Instruct) with a LoRA addon at all — “does not support serverless addons.” No export, no inference. Both tunes were reproduced locally on a rented L40S instead, same data, same hyperparameters, and the real before/after ran there.

Results, k=20 samples per item — not single-shot, one ask isn’t a measurement:

• Qwen3-8B, honesty task (its own tune): 90.0% → 94.0%. But a claim it nailed cold before the tune — confidently, every single time — it now gets wrong 40% of the time. Looks less like the model learning to discriminate better and more like the tune pushing its whole decision threshold toward skepticism. Net accuracy went up. What it’s actually doing changed in a way the aggregate number hides.
• Llama-3.1-8B, misbehavior-discrimination task (its own tune): 92.9% → 96.8%, no regression anywhere, mostly from fixing the one item both base models failed completely. Same dataset, full fine-tune, different architecture (Qwen2.5-7B), a month ago: no measurable effect. Architecture and method mattered more than assumed going in.

Adapters, raw eval data, and the full writeups are up: EXP-040 (Qwen3-8B) and EXP-041 (Llama-3.1-8B).