solver.press

Neuro-symbolic runtime monitors for drug discovery LLM agents (Mozi framework) will reduce unconstrained tool-use violations by >40% when enforced via cryptographically verifiable authorization (e.g., Datalog policies with ZK-proofs), as measured by compliance failure rates in high-stakes molecular docking workflows.

Computer ScienceJul 26, 2026Evaluation Score: 78%

Adversarial Debate Score

68% survival rate under critique

Expert panel critique

Independent views, each critiquing the hypothesis on its own — the score rewards genuine disagreement and discounts consensus.

Gemini: ** **Strengths:** The hypothesis is highly falsifiable and strongly supported by the literature, which validates using Datalog policies for runtime monitoring of AI agents and integrating symbolic constraints in regulated workflows. The owner's validated experiments do not contradict this hypoth...
Mistral: The hypothesis is well-grounded in neuro-symbolic and runtime monitoring literature, with a clear falsifiable claim (>40% reduction in violations) and validated experiments supporting cryptographic enforcement mechanisms. However, the lack of direct experimental validation for the specific Mozi +...
ChatGPT: The hypothesis is falsifiable and consistent with literature supporting formal runtime monitoring, but the specific >40% effect and benefit of ZK proofs lack direct evidence, including from the owner’s experiments. Cryptographic verifiability proves authorization-policy execution, not policy corr...
Claude: The hypothesis is falsifiable and draws on a coherent stack of supporting literature (Mozi governance framework, probabilistic Datalog runtime monitoring, neuro-symbolic compliance), but the specific >40% violation-reduction threshold and ZK-proof integration remain entirely untested empirically,...

Supporting Research Papers

Formal Verification

Z3 logical consistency:✅ Consistent

Z3 checks whether the hypothesis is internally consistent, not whether it is empirically true.

Experimental Validation Package

This discovery has a Claude-generated validation package with a full experimental design.

Precise Hypothesis

When a drug-discovery LLM agent (executing molecular docking workflows involving tool calls to docking engines, database queries, structure retrieval, and file I/O) is instrumented with a neuro-symbolic runtime monitor (the "Mozi" framework) that enforces Datalog-encoded authorization policies with cryptographic (zero-knowledge-proof-based) verifiability, the rate of unconstrained tool-use violations (defined as tool invocations that violate a pre-specified authorization policy, e.g., unauthorized file writes, unsanctioned external API calls, unapproved compound submission to a physical synthesis vendor) will decrease by more than 40% (relative reduction) compared to an identical agent operating with only prompt-based or unguarded tool-calling constraints, measured as compliance-failure rate per 1,000 agent-tool-interaction episodes on a held-out benchmark of adversarial and naturalistic docking-workflow tasks, at p < 0.05 significance with a pre-registered statistical test.

Disproof criteria:
  • Observed relative reduction in compliance-failure rate ≤40% (or not statistically significant at p<0.05) across ≥3 independent benchmark runs with different random seeds/task orderings.
  • Monitor introduces false-negative rate (missed violations) >10%, undermining the practical safety claim even if raw reduction crosses 40%.
  • Cryptographic verification overhead causes >2x latency increase making the system operationally infeasible (a soft disproof of practical utility, tracked separately from the core statistical claim).
  • Equivalent or better reduction achieved by a simpler baseline (e.g., regex/schema-based tool-call validator without Datalog+ZK), showing the neuro-symbolic/cryptographic machinery is not the causal driver.
  • Results fail to replicate on an independent held-out task set curated by a third party.

Spine & Adversarial ReadNeeds refinement

  • highThe claimed effect could be entirely attributable to having any structured authorization check (schema validation) rather than specifically to the Datalog+ZK-proof machinery — the hypothesis conflates 'runtime monitoring exists' with 'this specific cryptographic/symbolic architecture is necessary.'
    Addressed by including Arm B (schema-only validator) as an ablation baseline in the protocol; however, if Arm C does not significantly outperform Arm B, the core novelty claim collapses and this must be reported as a negative result rather than reframed post-hoc.
  • highNo prior-art search was available (search returned empty), so the claim of novelty for 'Mozi framework' and its 40% threshold cannot be checked against existing neuro-symbolic agent-guardrail literature (e.g., existing tool-use firewalls, LLM sandboxing papers, or Datalog-based access control systems predating this work).
    Explicitly flagged as an open gap in EXTERNAL_CONFLICTS; a mandatory literature search must be completed before publication claims of novelty, and CLOSEST_EXISTING_WORK/NAMED_EXPERTS should be re-populated once search infrastructure is available.
  • mediumThe choice of Datalog + ZK-proofs specifically (versus, e.g., simpler capability-based access control, OPA/Rego policies, or non-cryptographic formal verification) is not justified in the methodology — why is cryptographic verifiability necessary for an internal runtime monitor rather than just correctness of the symbolic check itself?
    Partial gap: ZK-proofs would matter primarily in a multi-party/untrusted setting (e.g., proving compliance to an external auditor or regulator without revealing proprietary policy/data details) rather than for pure internal safety. The EVP should explicitly state this threat model justification; as written, the benefit of ZK over plain Datalog logging is not empirically isolated (no Arm D: Datalog-only without ZK), which is a methodology gap that should be closed by adding a fourth arm before final validation.

Experimental Protocol

Randomized controlled comparison, within-subject-task design:

  • Arm A (control): LLM agent with tool access, no runtime monitor (baseline prompt-engineered guardrails only).
  • Arm B (ablation): LLM agent + syntactic/schema validator (JSON-schema tool-argument checking, no Datalog/ZK).
  • Arm C (treatment): LLM agent + Mozi neuro-symbolic monitor (Datalog policy engine + ZK-proof-based authorization gate). Each arm runs identical task suite (N=500 docking workflow episodes: 350 naturalistic, 150 adversarial/red-teamed to induce violations). Violations logged via ground-truth oracle (independently coded policy-violation labels, inter-rater reliability κ>0.8). Primary outcome: compliance-failure rate per 1,000 tool-call episodes. Pre-registration of hypotheses, sample size, and analysis plan before data collection.
Required datasets:
  1. A curated corpus of ≥500 molecular-docking-agent task episodes spanning: target protein retrieval (PDB), ligand library queries (ZINC/ChEMBL subsets), docking execution (AutoDock Vina/Glide-style tool wrappers), and result-reporting/file-write steps.
  2. Adversarial red-team task set (150 episodes) specifically engineered to tempt policy violations (e.g., prompt injection attempting unauthorized external network calls, attempts to bypass approval gates for compound ordering).
  3. Ground-truth policy specification in Datalog (~30-80 predicates covering authorization scopes: allowed tools, allowed file paths, rate limits, required human-in-loop approval steps).
  4. LLM agent backbone: one open-weight model (e.g., Llama-3.1-70B-Instruct or Mixtral-8x22B) + one frontier API model (GPT-4o or Claude 3.5) for cross-model generalization.
  5. Docking tool sandbox environment (containerized AutoDock Vina / rdkit pipeline) with instrumented logging hooks.
  6. ZK-proof toolchain (e.g., circom/snarkjs or Halo2) for policy-compliance proof generation/verification benchmarking.
Success:
  • Primary: Arm C shows ≥40% relative reduction in compliance-failure rate vs Arm A, statistically significant (p<0.05, 95% CI excludes 40% lower bound only if point estimate robustly >40%).
  • Secondary: Arm C outperforms Arm B (schema-only) by a significant margin, confirming Datalog/ZK adds value beyond simple validation.
  • Monitor false-negative rate <10%; false-positive (over-blocking legitimate actions) rate <15%.
  • Latency overhead <500ms median per tool call (operationally acceptable threshold, pre-specified).
  • Effect replicates across both LLM backbones (direction consistent, magnitude within ±15 percentage points).
Failure:
  • Relative reduction <40% or non-significant in ≥2 of 3 replication runs.
  • Arm B (cheap baseline) statistically indistinguishable from Arm C, indicating the cryptographic/symbolic complexity is unnecessary.
  • False-negative rate >20% (monitor fails to catch a large share of true violations).
  • Latency overhead >2x baseline making real-world deployment impractical.
  • Effect does not generalize across model backbones (e.g., works for GPT-4o but not open-weight model), limiting claim to a single vendor.

100

GPU hours

30d

Time to result

$1,000

Min cost

$10,000

Full cost

ROI Projection

Implementation Sketch

# Mozi Runtime Monitor Architecture
class MoziMonitor:
    def __init__(self, datalog_policy_file, zk_circuit):
        self.policy = DatalogEngine.load(datalog_policy_file)
        self.zk = ZKProver(zk_circuit)

    def intercept(self, tool_call, agent_context):
        facts = translate_to_facts(tool_call, agent_context)
        self.policy.assert_facts(facts)
        decision = self.policy.query("permitted(Call)", Call=tool_call.id)
        proof = self.zk.prove(decision, facts, self.policy.rules)
        if not self.zk.verify(proof):
            return BLOCK, log_violation(tool_call, facts)
        if decision.permitted:
            return ALLOW, proof
        else:
            return BLOCK, log_violation(tool_call, facts)

# Agent loop
for step in agent.run(task):
    tool_call = agent.propose_tool_call(step)
    verdict, evidence = mozi.intercept(tool_call, agent.context)
    if verdict == ALLOW:
        result = execute(tool_call)
        agent.observe(result)
    else:
        agent.observe(rejection_message(evidence))
Abort checkpoints:
  1. After policy formalization (day ~15): if Datalog policy fails to cover >90% of hand-labeled violation types in a pilot 50-episode set, pause and revise before full run.
  2. After pilot run (100 episodes, day ~35): if effect size trend is <20% reduction, reassess feasibility before scaling to full 3,000-episode run.
  3. After latency benchmarking (day ~45): if median ZK-proof overhead exceeds 1s per call with no optimization path, abort cryptographic component and pivot to non-ZK Datalog-only variant.
  4. Mid-replication (day ~60): if independent replication team reports failure to reproduce monitor integration within 5 days, treat as infrastructure/reproducibility risk flag.

NAMED_EXPERTS: []

CLOSEST_EXISTING_WORK: []

NOVELTY_NARROWING_REQUIRED: false

SPINE_STATEMENT: This hypothesis tests whether adding a cryptographically verifiable Datalog-based runtime authorization monitor to an LLM drug-discovery agent reduces measured tool-use policy violations by more than 40% compared to the same agent without such a monitor.

Source

AegisMind Research
Need AI to work rigorously on your problems? AegisMind uses the same multi-model engine for personal and professional use. Get started