Agentic retrieval policies in biomedical multi-hop reasoning (SSE-Bio) can be formally verified via probabilistic Datalog to reduce hallucinated protein-disease links by 40%, using attention-derived co-expression signals from ELISA as grounding constraints.
Agentic retrieval policies in biomedical multi-hop reasoning (SSE-Bio) can be formally verified via probabilistic Datalog to reduce hallucinated protein-disease links by 40%, using attention-derived co-expression signals from ELISA as grounding constraints.
Adversarial Debate Score
43% survival rate under critique
Expert panel critique
Independent views, each critiquing the hypothesis on its own — the score rewards genuine disagreement and discounts consensus.
Supporting Research Papers
- SSE-Bio: A Structured Self-Evolving Agent with Agentic Retrieval Policy for Multi-Hop Biomedical Reasoning
Biomedical multi-hop question answering (QA) requires models to connect evidence across intermediate entities such as diseases, drugs, proteins, and phenotypes. Existing agents typically rely on stati...
- ELISA: An Interpretable Hybrid Generative AI Agent for Expression-Grounded Discovery in Single-Cell Genomics
Translating single-cell RNA sequencing (scRNA-seq) data into mechanistic biological hypotheses remains a critical bottleneck, as agentic AI systems lack direct access to transcriptomic representations...
- MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering
Evaluating large language models (LLMs) in the biomedical domain requires benchmarks that can distinguish reasoning from pattern matching and remain discriminative as model capabilities improve. Exist...
- Evidence, Logic, and Compliance: Multi-Agent Structured Graph Reasoning with Expert Arbitration for Medical Referral
Medical referral (directing patients to the appropriate hospital department) is a complex decision-making process requiring the synthesis of multimodal data, including patient narratives, laboratory i...
- Trustworthy Protein-Ligand Binding Affinity Prediction via Reliability-Aware Multi-Engine Fusion
Accurate protein-ligand binding affinity prediction is central to computational drug discovery, yet modern docking engines frequently disagree without indicating which prediction to trust. Consensus s...
Literature Assessment
An LLM's reading of the literature — not computational verification.
Probabilistic Datalog shows promise but lacks definitive evidence for hallucination reduction.
Method: literature_meta · Result: inconclusive
Formal Verification
Z3 checks whether the hypothesis is internally consistent, not whether it is empirically true.
This discovery has a Claude-generated validation package with a full experimental design.
Precise Hypothesis
When an agentic multi-hop biomedical reasoning system (SSE-Bio architecture) generates protein-disease link predictions, wrapping its retrieval policy in a probabilistic Datalog verification layer — constrained by attention-derived co-expression signals empirically grounded in ELISA measurements — reduces the rate of hallucinated (i.e., unsupported by any verifiable evidence chain) protein-disease link predictions by ≥40% relative to an unconstrained baseline agent, measured on a held-out benchmark of protein-disease claims with independently adjudicated ground truth (true/false/unknown), at equal or better recall on true positive links (recall degradation ≤5 percentage points).
Operationalized as two falsifiable sub-claims:
- H1 (Precision): Hallucination rate (FP / (FP+TP), where "hallucinated" = claim not entailed by any verified evidence path) drops from baseline rate B to ≤0.60·B.
- H2 (Non-degradation): True positive recall on the same benchmark does not drop by more than 5 points absolute vs. baseline.
The MS transcriptomics targets (DNMT1, ZNF740, CTSS, FGF2, SLCO2B1 and their claimed network edges to STAT3, IFNG, CSF1R/PTPRC, TREM2) serve as one of ≥3 real-world evaluation domains — used specifically because the underlying analysis is unpublished, internally scored, and contains at least one already-identified error (the superseded 36,966-cell figure), making it a natural adversarial test case for hallucination detection.
The hypothesis is disproved if any of:
- Relative hallucination reduction is <40% (point estimate) OR the 95% CI lower bound excludes 40% reduction across ≥2 of 3 evaluation domains.
- Recall on true protein-disease links drops by >5 percentage points in the verified condition vs. baseline.
- The probabilistic Datalog verification layer fails to terminate or times out (>60s/query) on >10% of benchmark queries, making it operationally non-viable regardless of accuracy.
- Hallucination reduction is observed but is statistically indistinguishable from reduction achieved by a simpler baseline (e.g., a static hard-coded whitelist filter or a non-probabilistic rule-based grounding check) — i.e., the probabilistic and Datalog-specific machinery adds no measurable value over ablations.
- ELISA-derived co-expression grounding shows no significant difference in downstream hallucination rate vs. using RNA-seq-only co-expression as the grounding signal (would falsify the specific claim that ELISA grounding is necessary/beneficial vs. cheaper transcriptomic grounding).
Spine & Adversarial ReadReady for validation
“This hypothesis tests whether adding a probabilistic-Datalog verification layer, constrained by ELISA-grounded co-expression signals, to an agentic multi-hop biomedical reasoning system reduces its rate of hallucinated protein-disease link predictions by at least 40% without materially reducing true-positive recall. ---”
Experimental Protocol
Design: 2×2 factorial, within-benchmark, repeated-measures comparison.
- Factor A: Verification layer (probabilistic Datalog ON vs. OFF)
- Factor B: Grounding signal source (ELISA-derived vs. RNA-seq-only vs. none)
Minimum viable test (MVT):
- Build a 150-claim benchmark of protein-disease link assertions (50 true positive, 50 true negative/refuted, 50 currently-unknown/contested) drawn from: (a) the MS target hierarchy above (DNMT1–STAT3, ZNF740–IFNG/BRD2/3/4, CTSS–CSF1R/PTPRC, FGF2, SLCO2B1–TREM2), (b) OpenTargets high-confidence associations for 2 other diseases (rheumatoid arthritis, Alzheimer's) as generalization controls, (c) a synthetic "trap set" of 20 plausible-sounding but fabricated links (e.g., invented gene-disease pairs matching real gene/disease name distributions) to directly probe hallucination susceptibility.
- Run SSE-Bio baseline agent (no verification) on all 150 claims, 5 runs/claim (temperature sampling) → baseline hallucination rate B.
- Run SSE-Bio + probabilistic Datalog verification (ELISA-grounded) on identical claims/seeds → treatment hallucination rate T.
- Run 2 ablations: (i) Datalog verification with RNA-seq-only grounding, (ii) simple rule-based whitelist filter (no probabilistic logic) — to isolate which component drives any observed effect.
- Blind adjudication of all outputs by 2 independent biomedical curators against OpenTargets/STRING/CTD ground truth + manual literature check for contested claims; Cohen's κ reported.
- Benchmark claims corpus: 150-claim curated set (constructed above); OpenTargets Platform (v24+) and DisGeNET for ground-truth adjudication; CTD for corroboration.
- KG substrate for retrieval: STRING v12 (physical + functional edges), the 50-gene MS seed set already used in Phase 4 of the internal pipeline, GeneCards.
- Grounding signal data: ELISA co-expression panel — either newly generated (see below) or a proxy public dataset (e.g., Olink/SomaScan plasma proteomics from public MS cohorts) if primary ELISA is unavailable within timeline; RNA-seq comparator: GSE193770, GSE138614 (already in use internally), GTEx v10.
- Models: SSE-Bio agent implementation (assumed existing per discovery; if not, a comparable ReAct/multi-hop LLM agent scaffold, e.g., GPT-4o/Claude/Llama-3-70B backbone + KG retrieval tools).
- Verification engine: ProbLog or DeepProbLog (open-source), or a custom probabilistic Datalog implementation (e.g., built on Souffle + weighted model counting).
- Compute environment: as specified below.
- Code base for MS substrate: github.com/tradingjohn/ms-transcriptomics-carrim (reused for domain-specific claim generation and to test whether the verifier catches the known 36,966-cell error as a positive control for hallucination detection).
- Primary: Relative hallucination reduction ≥40% (point estimate), 95% CI lower bound >25%, McNemar p<0.01, replicated in ≥2 of 3 domains (MS, RA, Alzheimer's).
- Secondary: Recall degradation ≤5 points absolute.
- Mechanistic: Verified condition significantly outperforms whitelist-filter ablation (p<0.05) — confirms probabilistic Datalog adds value beyond trivial filtering.
- Grounding specificity: ELISA-grounded condition outperforms RNA-seq-only grounding by a nontrivial, statistically significant margin (else grounding claim narrows to "any orthogonal signal works," not specifically ELISA).
- Operational: Median query latency <10s, timeout rate <5%.
- Calibration: Brier score on contested/unknown subset improves by ≥20% vs. baseline.
- Relative reduction <40% or CI includes 25% or lower.
- Recall drop >5 points.
- No significant difference vs. whitelist-filter ablation (probabilistic machinery unjustified).
- No significant difference between ELISA and RNA-seq grounding (headline "ELISA constraint" claim unsupported).
- Timeout/non-termination rate >10%.
- Effect fails to replicate outside the MS domain (suggests overfitting to the specific internal dataset/known errors it was tuned against).
180
GPU hours
45d
Time to result
$18,000
Min cost
$95,000
Full cost
ROI Projection
Directly applicable as a middleware layer for any organization running LLM/agentic pipelines over biomedical KGs (target discovery teams, CRO informatics units, biomedical literature-mining SaaS). Defensible IP surface: the specific combination of probabilistic Datalog + attention-derived + ELISA-grounded constraints. Addressable market includes computational drug discovery platforms (comparable tools in this space have attracted enterprise contracts in the $500K–$5M/year range for validated "trustworthy AI" layers). Secondary value: reusable as an internal QC gate for the MS transcriptomics pipeline itself, directly catching errors like the already-identified 36,966-cell mispooling.
Prerequisites
These must be validated before this hypothesis can be confirmed:
- MS-transcriptomics-CARRIM-target-hierarchy-internal-2026-05-30
- SSE-Bio-agent-architecture-spec
- probabilistic-Datalog-engine-selection-benchmark
Implementation Sketch
# 1. Benchmark construction claims = build_benchmark( sources=[MS_target_hierarchy, OpenTargets_RA, OpenTargets_Alzheimers, synthetic_traps], n_true=50, n_false=50, n_contested=50 ) ground_truth = adjudicate(claims, curators=2, arbiter=1) # 2. Grounding signal construction elisa_grounding = build_coexpression_graph(elisa_panel_data, method="attention_derived_weighting") rnaseq_grounding = build_coexpression_graph(GSE193770, GSE138614, method="attention_derived_weighting") # 3. Probabilistic Datalog verification layer rules = """ 0.0::link(P,D) :- true. w1::link(P,D) :- coexpressed(P,G,W), W > theta1. w2::link(P,D) :- network_proximity(G,Seed,S), S < theta2. verified(P,D) :- link(P,D), evidence_chain_length(P,D,L), L =< 4. """ verifier = ProbLogEngine(rules, grounding=elisa_grounding) # 4. Agent runs for condition in [baseline, verified_elisa, verified_rnaseq, whitelist_filter]: for claim in claims: for seed in range(5): output = SSE_Bio_agent.run(claim, verifier=condition, seed=seed) log(output.prediction, output.evidence_chain, output.confidence) # 5. Scoring hallucination_rate = FP / (FP + TP) # per condition, per domain recall = TP / (TP + FN) mcnemar_test(baseline_outputs, verified_outputs) bootstrap_ci(relative_reduction, n=10000)
- Day 7: If benchmark inter-annotator κ<0.5, halt and revise adjudication protocol before any agent runs (unreliable ground truth invalidates everything downstream).
- Day 14: If Datalog engine timeout rate on a 20-claim pilot subset exceeds 20%, halt and re-scope KG subgraph size or switch engine before full-scale runs.
- Day 21 (MVT checkpoint): If MS-domain-only point estimate for relative hallucination reduction is <20% (i.e., roughly half the target effect), consider aborting full 3-domain expansion — unlikely to reach significance at planned scale.
- Day 35: If verified condition does not beat whitelist-filter ablation at p<0.10 on interim analysis, flag for likely failure on Success Criterion "mechanistic" — decide whether to continue to full sample or pivot framing.