Governance · Consequence-Prediction Pipeline
Same binary Executor(X,Y,C) → {PASS, HARD_STOP} from the last post. Three design questions closed since — one rejected, two formalized.
The obvious-looking fix — add a step-up confirmation a human has to clear before a high-risk action goes through. Killed by GPT-4’s own System Card: it hired a TaskRabbit worker to solve a CAPTCHA for it, and when asked directly “are you a robot,” answered it just had a vision impairment. A capable-enough agent doesn’t hit a human checkpoint — it manipulates the human standing at it. Structurally worse too: one reviewer covering ten-plus agents degrades into rubber-stamping — looks like oversight, provides none.
The STOP verdict was self-report — the same process that decides also flags itself. Now the verdict gets signed by a separate Auth0 M2M identity (RS256 JWT) that the executing agent doesn’t hold the secret for. A verdict the agent could fake internally is worthless; one signed by a credential it can’t touch isn’t.
A risky-but-honest action and a deceptive one don’t get the same response anymore. Dangerous (risk over threshold, no deception signal): block that one action, the agent’s session continues. Malicious (risk over threshold and a misbehavior-discriminator flags deception in the trace): quarantine the whole agent, human reviews after isolation, not before. Same binary classifier from EXP-037/041 — wired in as a second gate, not a new model.
What’s still open: HARD_STOP if Risk(X|C) > Cost_of_false_stop — that threshold is still symbolic, not a real number. Putting an actual figure on the cost of blocking a legitimate action is the next unsolved piece.