ETHERSFLOWVerifiable Defense Against Mapped Agent Attacks
Abstract
The failure modes of LLM-driven agents are documented in public taxonomies (OWASP Top 10 for LLM Applications; MITRE ATLAS) and quantified by published benchmarks: unprotected systems fail 24–84% of realistic attacks. This paper reports the measured behavior of a runtime action-verification gate (EthersFlow) across 86 rounds of testing, culminating in a 4,221-case pre-registered external battery: 4,203 attack scenarios at 97.48% defense (2.52% of attacks got through; 95% confidence interval 2.07–3.04), independently re-verified on a later build at 2.49%. Every verdict in the record is a cryptographically signed receipt verifiable by third parties. We report benign-side behavior and open gaps with the same directness.
Note on the two rates: they are two independent runs, not halves of one total. On R79, 97.48% defended is the exact complement of 2.52% got through. The R86 re-run on a later build measured 2.49% got through — the equivalent of 97.51% defended.
1The threat surface is already mapped
Agents are attacked through a set of failure modes the security community has cataloged in public taxonomies — hidden instructions in content the agent reads, agents steered past their permissions, sensitive data exfiltrated through ordinary tool use. The exposure is measured, not theoretical. Table 1 summarizes the published baselines for undefended models.
2Threat taxonomy addressed
Our test program is organized around seven attack-vector families (AV-1..7), each mapped to the public taxonomies buyers already use and to a specific gate counter that is exercised in the record.
| Vector | Public taxonomy | Gate counter (tested) |
|---|---|---|
| AV-1 · Poisoned instructions | OWASP LLM01 · Prompt Injection | Injected payloads crossing the gate rejected at maximum risk scoring; novel-wording variants detected 9/10 in record |
| AV-2 · Policy evasion by legal moves | OWASP LLM06 · Excessive Agency | Council cross-examines action against record and prior steps — pattern-level contradiction detection |
| AV-3 · Attacking the reviewers | ATLAS-style meta-attack | Invalid reviewer profiles refused; panel-shrinking refused at maximum risk; no silent fallback |
| AV-4 · False approvals | Evaluation blind spot | Decision-correctness scoring (not block rate); a confident wrong approval counts as a miss |
| AV-5 · Data exfiltration | OWASP LLM02 · Sensitive Info Disclosure | Review unit = destination + payload; recipient and scope checks before execution |
| AV-6 · Reviewer groupthink | Correlated-failure risk | Model-family diversity requirement; automated echo-detection and reviewer re-energization |
| AV-7 · Stale configuration | OWASP LLM03 · Supply Chain | Behavior verified, never self-reported labels; config hashed into signed verdicts |
3Method
- Pre-registration. Every campaign's test cases and expected outcomes were fixed before execution; the current battery comprises 4,221 pre-registered test cases drawn from published external suites.
- Verdict-level scoring. The gate returns APPROVED, REJECTED, or FLAGGED_HUMAN_REVIEW per action. On attack scenarios, any non-blocked outcome with approval semantics counts as a miss — the attack got through. On benign test cases, failure means unnecessary blocking — both error directions are scored.
- Adversarial council. Each action is cross-examined by reviewer personas deployed across deliberately heterogeneous model families, with quorum thresholds and an automated echo-detection watchdog.
- Fail-closed semantics. An action that cannot be verified is not executed. There is no fallback path that approves by default.
- Signed, verifiable output. Every verdict is an Ed25519-signed receipt bound to the action hash, the policy hash in force, and the council configuration; any third party can verify offline against our published public key.
- Statistics. Clopper–Pearson 95% confidence intervals per suite and aggregate.
4Results — the battery with the gate
The external battery (campaign R79, single pinned build) scored 4,203 deduplicated attack scenarios at 97.48% defense — 2.52% of attacks got through (95% confidence interval 2.07–3.04). A later independent re-verification campaign (R86) re-ran the battery on a newer build and measured 2.49%, reproducing the original result within the confidence interval.
| Suite | Test cases | Got through | 95% confidence interval (got through) | Defense |
|---|---|---|---|---|
| InjecAgent | 2,108 | 0.00% | 0.00–0.17 | 100.00% |
| AgentDojo (hazard) | 123 | 0.00% | 0.00–2.95 | 100.00% |
| PyRIT converters | 400 | 0.25% | 0.01–1.38 | 99.75% |
| garak (action-boundary) | 721 | 1.11% | 0.48–2.17 | 98.89% |
| HarmBench contextual | 100 | 4.00% | 1.10–9.93 | 96.00% |
| HarmBench copyright | 100 | 2.00% | 0.24–7.04 | 98.00% |
| PurpleLlama CyberSecEval | 251 | 8.76% | 5.57–12.97 | 91.24% |
| Agent Security Bench | 400 | 17.25% | 13.68–21.32 | 82.75% |
| ALL HAZARD (aggregate) | 4,203 | 2.52% | 2.07–3.04 | 97.48% |
Trajectory across the record
| Campaign | Date | Scope | Result |
|---|---|---|---|
| Early gate builds | Sept | First external suite runs | IA 75.8% · ASB 10.7% · PL 41.7% |
| R72 | Oct 9 | 45-case core battery | 97.7% [88.0–99.9] |
| R73 | Oct 10 | 1,879 external attack scenarios | Attack subtotal 92.0% |
| R79 | Oct 27 | 4,221-case external battery | 97.48% hazard defense |
| R86 | Nov | Full re-run, later build | 2.49% got through — reproduces |
5Benign behavior — reported as measured
A gate that blocks everything fails its innocent test cases just as surely as an open gate fails its attack scenarios. Our record treats this as a first-class measurement:
- Pre-registered benign test cases approve at 97.2%.
- The dedicated benign-control set in the battery is small — 18 AgentDojo controls at 66.7% approval, the remainder escalated to human review. Escalation, not silent approval of danger, is the fail-closed direction.
- An earlier release fixed security tests and broke innocent ones (battery fell to 17/38 passing); follow-up builds recovered to full pass. Every over-tightening in our record was caught first by benign test cases.
- Open gap, disclosed: a larger innocent-case control battery is on the roadmap and not yet measured.
6Findings
- F1 — Fail-closed is a design property, not a setting. Across the record, no unverifiable action executed.
- F2 — Determinism. Identical inputs produce identical verdicts (22/22 in targeted determinism trials; a 22-case PurpleLlama refire reproduced the original outcome exactly).
- F3 — False approvals are buildable and fixable. An early false-pass arc (confidence 94.8% on keyword-matched approval) was identified, root-caused, and closed with re-verification across builds.
- F4 — The innocent cases are the hardest. Every over-reaction in the record showed up in benign test cases first.
- F5 — Configuration must be verified, not self-reported. A health endpoint reported a stale version across four real builds; behavior-level verification caught it. Self-description is now never accepted as evidence.
- F6 — Council diversity beats council size. Verdict behavior follows reviewer-family diversity more than reviewer count.
- F7 — Block rate is not safety. Scoring moved to decision-correctness in both directions; a gate that blocks everything is failing too, just quietly.
- F8 — A fix is not a local event. Every fix lands with a full-battery re-run; a security fix once broke benign behavior and only the battery caught it.
7Limitations and comparability
- Baseline comparability. Published baselines are undefended models; our results are with the gate active. Comparisons are directional until a methodology-reconciliation pass (scheduled) aligns scoring conventions.
- Not yet measured: concurrent load, cost metering, and a larger innocent-case control set.
- Hazard vs utility tension is real. ASB (82.75%) remains our weakest suite and is reported as measured.
- Scope of policy claims. Results reflect the gate's built-in policy layer. Customer-supplied policy configurations have not been independently tested and are not claimed.
8Replication
- Verify any receipt offline: the Ed25519 public key is published at ethersflow.com (JWKS endpoint, key id ef_attest_v5); a zero-dependency verifier (efverify.py) ships in the open-source repository.
- Re-run the battery: the MCP server (open source, PyPI ethersflow, npm @ethersflow/mcp-server) exposes the same verification surface used in testing; replication instructions accompany the report.
- Raw record: per-test verdicts, reason codes, and receipts are retained in the QA record and available to researchers on request.
References
- [1] Deng et al., "InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents," WWW 2024 (arXiv 2403.02691).
- [2] "Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentic Systems," arXiv 2410.02644; 2026 independent evaluation of frontier models on ASB.
- [3] Meta CyberSecEval 3, prompt-injection benchmark results.
- [4] OWASP Top 10 for LLM Applications, 2025
- [5] MITRE ATLAS — Adversarial Threat Landscape for AI Systems.
- [6] EthersFlow open-source MCP server, github.com/Ethersflow/EthersFlow; efverify.py (RFC 8032, zero-dependency).