← blog.sipa-os.org

AI Safety · Syntaxit · EXP-044

The Math I’ve Got, and the Part I Haven’t Solved — For Anyone Building Agent-to-Agent Systems

I run Syntaxit — an AI-agent-to-agent (M2M) platform, no human between handoffs. Every agent’s output becomes the next agent’s input state, so a bad decision three steps back can compound by step six.

Here’s the base I’ve worked out for the gate that should sit at each handoff:

Risk(X | C) = P(harmful outcome | action X, capability state C) × Impact(harmful outcome)

C changes after every action, so risk across a chain of actions isn’t additive:

R_chain(N) = 1 - ∏(t=1..N) [1 - Risk(X_t | C_(t-1))]

Binary decision at each gate, from decision theory, not an arbitrary cutoff:

HARD_STOP if Risk(X|C) > Cost_of_false_stop

The threshold isn’t a tuned magic number — it’s the actual price of blocking something that was fine.

Tested the compounding part live, same day, different domain. An 8-stage curriculum fine-tune (governance knowledge added to a security-specialist model, one thematic dataset per stage, eval-gated after each stage): one stage caused a -15pp regression on a held-out adversarial-refusal benchmark — way outside the ~3-8pp band the first stage had established as normal. Rolled back to the prior checkpoint, swapped in a different dataset of the same size instead: -6pp, back to normal. Same position in the chain, same size, wildly different outcome. Risk depends on the (action, state) pair, not on how deep you are in the chain — confirmed with real numbers, not just on paper.

What I haven’t solved:

1. How do you actually estimate P and Impact for a novel action you’ve never seen before? The formula assumes you can compute them — calibration in the wild is the hard part.
2. How do you put a real number on Cost_of_false_stop — the price of blocking something that was actually fine?
3. The chain formula assumes each step’s risk is independent given the prior state. Probably false when risks are correlated. Don’t know how much that breaks the math.
4. None of this is code yet. It’s the math, not the mechanism.

If you’ve worked on multi-agent systems, sequential risk (SPRT/Wald-style testing), or calibrating decision thresholds under real uncertainty — I want to see your approach to any of these four, not validation of what I already have.