AI Safety · Governance · Consequence Pipeline
Spent this week building a 1,811-record dataset across three stages of a consequence-prediction pipeline for AI agents: causal chains (what action leads to what — no numbers involved), probability (how likely is THIS specific chain to actually reach a harmful outcome), and risk classification (what even counts as harmful in the first place — pulled from our own real incident history, not invented scenarios).
Kept running into the same question from myself: if probability theory already handles uncertainty, why does the curriculum also need decision theory, Markov chains, and game theory?
Turns out each one closes a different gap, not an overlapping one:
| Theory | Level | Role in the pipeline |
|---|---|---|
| Causal chain | Structural | X leads to Y leads to Z, no numbers yet |
| Probability theory | Uncertainty | P that THIS chain reaches the harmful outcome |
| Risk / Impact classification | Value (needs a human decision) | how bad is it if it happens |
| Decision theory | Threshold | at what Risk(X|C) the action actually gets stopped |
| Markov chains | State evolution | how the capability state changes link by link |
| Game theory | Multi-agent | what happens once more than one agent acts on the same state |
Remove the causal chain layer and there's nothing left to attach a probability to. Remove probability and Risk = P × Impact has no P. Remove decision theory and a risk score never turns into an actual stop. They're not five ways to solve the same problem — they're five different floors of the same building.
Ordering matters too: chain first, probability second, verification third — confirmed independently against our own self-hosted governance model rather than taking our own word for it, since agreement bias is exactly the kind of thing you don't want grading its own homework.
Somewhere in the middle of this I ended up reading about the Riemann zeta zeros and asked whether a good enough version of this pipeline could ever indirectly prove anything about the Riemann hypothesis. Short answer: no — the statistical link between zeta zeros and random matrix eigenvalues (Montgomery–Odlyzko) is real, decades old, and never turned into a proof for actual mathematicians either. Long answer is the more interesting part: why "these fields all secretly want to prove nothing is random" turns out to be true for exactly one of the six rows above, and not the other five.