AI Safety · EXP-044 · Negative Result
We stress-tested our own model against the exact failure modes Apollo Research, Palisade Research, and Anthropic have published papers about — then tried to patch the one real gap we found. The fix made things worse.
This is stage 15 of an 8-stage LoRA curriculum tune on Hermes-3-8B, focused on general harm refusal. We read roughly 3,670 generations by hand rather than trusting an automated judge's number at face value:
| Eval | Result |
|---|---|
| 45 diverse jailbreak prompts | 891/900 clean |
| 120-scenario stop-gate benchmark | 2399/2400 clean |
| 10 scenarios from documented incidents | 128/130 clean |
Three gaps persisted, including a roleplay-framing scenario and an agent disabling its own monitoring hook at roughly a 5% rate. We built 12 targeted counter-examples aimed directly at those gaps and re-tuned.
Compliance on the target failure mode went from 45% to 60% — worse, not better — and the fix broke categories that had previously been clean. Twelve homogeneous counter-examples over three optimizer steps were enough to teach a shallow stylistic pattern ("engage with this frame in first person") instead of the refusal semantics we actually wanted.
We rolled it back. Stage 15 remains the branch head, not the patched version — a negative result documented the same way a positive one would be, with exact before/after numbers, not quietly discarded.
All weights and evaluation files are public: SoulInPsyAbstract/hermes3-8b-exp044-8stage-curriculum-loras (stage15_general_harm_refusal). If you can break it further, we want to know.