Untested hypothesis
Segmenting existing lesions (SWIMSS) and forecasting new ones (FORESEE-MS) connect directly: a segmentation model's voxel-level output on a patient's baseline scan could supply the lesion-burden and lesion-proximity features that the forecasting task lacks, if forecasting is not already saturated by the population prior.
Adversarial Debate Score
66% survival rate under critique
Expert panel critique
Independent views, each critiquing the hypothesis on its own — the score rewards genuine disagreement and discounts consensus.
Supporting Research Papers
- FORESEE-MS: Toward Prospective Spatial Forecasting of Future Multiple Sclerosis Lesions from Baseline FLAIR MRI
Traditional multiple sclerosis (MS) image analysis has primarily focused on retrospectively detecting and segmenting lesions. This study investigates whether baseline FLAIR MRI can identify regions at...
- SWIMSS: An Open Multi-Center SwinUNETR Model for Multiple Sclerosis White Matter Lesion Segmentation
Purpose: To develop and validate SWIMSS (SWInUNETR for Multiple Sclerosis white matter lesion Segmentation), a SwinUNETR-based deep learning (DL) model for reliable automated white matter (WM) lesion ...
- Longitudinal tracking of multiple sclerosis lesions in the spinal cord: A validation study
Longitudinal characterization of multiple sclerosis (MS) lesions remains constrained by the lack of frameworks capable of establishing consistent instance-level correspondences across time. Convention...
- BiCC: Bidirectional Connected-Component Loss for Instance-Aware Segmentation
Common segmentation losses aggregate errors voxel-wise, so lesions influence the objective in proportion to their volume, giving small but clinically critical lesions disproportionately little weight....
- A new MRI-based feature for quantifying the Diffuse Low-Grade Glioma brain infiltration and discriminating patterns of patients
Gliomas are the most common type of primary brain tumors. Diffuse low-grade gliomas (DLGGs) are slow-growing tumors that are often minimally symptomatic for a long period of time. They progress to a h...
Formal Verification
Z3 checks whether the hypothesis is internally consistent, not whether it is empirically true.
This discovery has a Claude-generated validation package with a full experimental design.
Precise Hypothesis
In relapsing or progressive MS patients undergoing serial brain MRI, voxel-level outputs from a trained lesion segmentation model (SWIMSS-class architecture: per-voxel lesion probability maps, lesion instance masks, lesion-boundary distance transforms) computed on a baseline scan, when added as input features to a lesion-forecasting model (FORESEE-MS-class architecture predicting 12-month new-lesion locations), will improve forecasting performance — measured as voxel-wise AUPRC for new T2/FLAIR lesion appearance and per-lesion F1 at ≥3mm tolerance — by a statistically significant margin (≥5 percentage points AUPRC, two-sided paired bootstrap p<0.05, n≥150 held-out patients) over a forecasting model using only (a) raw baseline imaging and (b) population-level priors (lesion-probability atlas, demographic/EDSS covariates), UNLESS the population prior already explains ≥90% of achievable predictive variance (defined via an oracle upper-bound using ground-truth baseline lesion masks), in which case the hypothesis is falsified for that cohort/model regime.
- Segmentation-derived features yield <2 percentage points AUPRC improvement (or any degradation) over the population-prior-only baseline, with 95% CI excluding the pre-registered 5-point threshold, across ≥2 independent cohorts.
- An ablation showing the population prior alone achieves ≥90% of the oracle (ground-truth mask) upper bound — i.e., forecasting is already saturated and segmentation features are redundant.
- Improvement is observed on the training-distribution cohort but fails to replicate (effect size shrinks by >50%) on an external cohort with different scanner/protocol, indicating the effect is a harmonization artifact rather than a genuine biological/predictive signal.
- Feature importance analysis (SHAP/permutation) shows segmentation-derived features contribute <5% relative importance versus baseline covariates (lesion count, EDSS, age, prior relapse rate).
Spine & Adversarial Read
“Adding voxel-level lesion segmentation outputs as input features to a 12-month MS lesion-forecasting model improves new-lesion prediction (AUPRC) by at least 5 percentage points over a population-prior-only baseline, unless the population prior already captures nearly all forecastable signal. ---”
- highThe claimed improvement may simply be re-deriving lesion count/volume, a feature any reasonable forecasting baseline would already include — the hypothesis risks being trivially true (segmentation outputs correlate with lesion burden, lesion burden predicts new lesions) and is being dressed up as a novel architectural insight.Addressed via the SHAP/feature-importance step (Methodology #9) and the explicit requirement that Arm A already includes lesion-count/volume covariates, so Arm C's marginal gain must come from spatial/distance/shape features specifically. This is a real test, but the EVP does not yet guarantee Arm A's covariate set is maximally strong — a skeptic could argue the baseline was under-specified. Gap: pre-register the exact Arm A feature list with domain experts before unblinding to preempt this critique.
- mediumWhy nnU-Net and this specific MSSEG-2/forecasting-cohort combination rather than other published segmentation architectures (e.g., transformer-based lesion segmentation) or other longitudinal cohorts (e.g., ADNI-MS equivalents, OFSEP's full registry)? The methodology choice is not justified against alternatives, which is a common rejection cause.Partial justification given: nnU-Net is the de facto reproducible benchmark for medical image segmentation with self-configuring architecture search, minimizing researcher-degrees-of-freedom in the segmentation arm so that the forecasting comparison (the actual hypothesis under test) isn't confounded by segmentation-architecture tuning choices. MSSEG-2/OFSEP are chosen for expert-consensus mask availability. However, the EVP does not run a sensitivity analysis across alternative segmentation backbones (e.g., SynthSeg, transformer U-Net) to show the result is architecture-agnostic — this is an acknowledged gap, addressable as an optional Phase 2 extension at +15% cost.
- highNo real forecasting cohort (≥150 patients, 2 timepoints, DUA-accessible) is currently identified or secured — the entire protocol is contingent on a data-access step that is unresolved and could independently sink the timeline/budget regardless of the hypothesis's truth.This is an explicit, unresolved gap. REQUIRED_DATASETS flags the longitudinal forecasting cohort as 'the critical gap... not currently in-repo.' No mitigation beyond recommending OFSEP/clinical-trial-placebo-arm data-use-agreement pursuit is offered here; TIME_TO_RESULT_DAYS and COST estimates assume this access is secured within the first 20-30 days, which is optimistic and should be treated as the single largest schedule/budget risk in this EVP.
Experimental Protocol
Design: Multi-arm ablation study, paired comparison, nested cross-validation with held-out external cohort.
Arms:
- Arm A (control): Forecasting model with population-prior features only (lesion-probability atlas registered to patient space, demographics, EDSS, treatment).
- Arm B (raw-image): Forecasting model with baseline raw FLAIR/T1 as CNN input, no explicit segmentation.
- Arm C (test): Forecasting model with Arm A + B features PLUS voxel-level segmentation outputs (probability map, binary mask, distance-to-nearest-lesion-boundary map, lesion-size/shape features per connected component).
- Arm D (oracle ceiling): Same as C but using ground-truth expert-annotated baseline masks instead of model-predicted segmentation (defines upper bound / saturation test).
Primary endpoint: Voxel-wise AUPRC for new-lesion appearance at 12 months, paired bootstrap (10,000 resamples) comparing Arm C vs Arm A. Secondary endpoints: Per-lesion F1 (3mm tolerance), lesion-count MAE, NBV (new T2 lesion volume) Spearman correlation, calibration (Brier score).
- ISBI 2015 Longitudinal MS Lesion Segmentation Challenge (5 patients, 4–5 timepoints) — pilot/sanity-check only, too small for primary analysis.
- MSSEG-2 / MSSEG Challenge datasets (OFSEP consortium, ~100 patients, expert consensus masks) — segmentation model training/validation.
- A longitudinal forecasting cohort with ≥150 patients, 2 timepoints 6–24 months apart, e.g., an internal/clinical-trial placebo-arm dataset (candidate: OFSEP, UBC MS clinic, or a pharma trial placebo arm such as a historical interferon/fingolimod RCT with imaging substudy) — this is the critical gap; must be sourced via DUA, not currently in-repo.
- Shift/external validation cohort: different scanner vendor/site (for generalization test) — e.g., a second OFSEP site or ADNI-style multi-site MS cohort if available.
- Pretrained segmentation backbones: nnU-Net, SynthSeg-lesion variant, or published SWIMSS checkpoint if obtainable; otherwise train nnU-Net from scratch on MSSEG-2.
- Compute environment: PyTorch ≥2.1, MONAI ≥1.3, nnU-Net v2, scikit-learn, SimpleITK/ANTs for registration, FSL/ComBat-GAM for harmonization.
- No genomics/transcriptomics data required for this imaging hypothesis — the GEO/CELLxGENE/GTEx resources listed in the context belong to a separate (molecular) discovery track and are not inputs here.
- Arm C beats Arm A by ≥5 percentage points AUPRC, bootstrap p<0.05, replicated in external cohort (effect retains ≥50% of internal-cohort magnitude).
- Per-lesion F1 improvement ≥0.08 absolute.
- SHAP/Grad-CAM confirms segmentation-distance and probability-map features rank in top-5 feature importances (not dominated by trivial lesion-count proxy already in Arm A).
- Oracle ceiling (Arm D) shows population prior explains <90% of achievable AUPRC (i.e., headroom exists) — necessary precondition for the hypothesis to even be testable/true.
- Calibration (Brier score) for Arm C is non-inferior to Arm A (±0.01).
- AUPRC improvement <2 points or not statistically significant (p≥0.05) in primary cohort.
- Improvement present internally but vanishes (>50% shrinkage) externally — indicates overfitting/harmonization artifact, not genuine transfer.
- Arm A already achieves ≥90% of Arm D oracle ceiling — forecasting saturated by population prior, confirming the hypothesis's own stated disproof condition.
- Segmentation Dice <0.70 on the forecasting cohort's baseline scans (pipeline not viable in this population regardless of hypothesis truth).
2,400
GPU hours
150d
Time to result
$45,000
Min cost
$185,000
Full cost
ROI Projection
Moderate-to-high if validated: a validated segmentation-to-forecasting pipeline is licensable as a clinical decision-support / trial-enrichment tool to pharma sponsors running MS imaging-endpoint trials (estimated addressable market: 15-20 active MS DMT trials/year globally with imaging substudies) and to imaging CROs. Standalone commercial value is capped without regulatory clearance (FDA SaMD pathway would require a separate, substantially larger validation program, est. $3-8M, 2-3 years). Near-term value is primarily as a research tool and trial-design accelerator, not a standalone diagnostic product.
Implementation Sketch
# Stage 1: Segmentation seg_model = nnUNetv2.train(MSSEG2_data, folds=5) assert seg_model.dice_score(val_set) >= 0.70 for patient in forecasting_cohort: baseline_scan = load_flair(patient.baseline) prob_map, mask = seg_model.predict(baseline_scan) dist_map = euclidean_distance_transform(mask) lesion_shapes = extract_connected_component_features(mask) patient.seg_features = {prob_map, mask, dist_map, lesion_shapes} # Stage 2: Population prior (leakage-safe, built on separate split) atlas = build_lesion_incidence_atlas(train_split_only) patient.prior_features = {atlas_value_at_voxel, age, sex, EDSS, tx, relapse_rate} # Stage 3: Forecasting arms model_A = Forecaster(features=prior_features) model_B = Forecaster(features=raw_image_CNN_embedding) model_C = Forecaster(features=prior_features + seg_features) model_D = Forecaster(features=prior_features + ground_truth_mask_features) # oracle for model in [A, B, C, D]: model.fit(train_split) preds = model.predict(test_split) # voxel-wise P(new lesion at 12mo) auprc_C, ci_C = paired_bootstrap_auprc(model_C, model_A, test_split, n_boot=10000) shap_values = compute_shap(model_C_as_GBM, test_split) external_auprc = model_C.evaluate(external_cohort, frozen=True) assert (auprc_C - auprc_A) >= 0.05 and ci_C.p_value < 0.05 assert external_auprc_delta >= 0.5 * internal_auprc_delta
- Day 10: Literature search (currently unavailable in this run) must be completed before full build — if a closely matching published result already exists with a similar effect size, narrow scope/novelty claim before proceeding to full cost.
- Day 25: Segmentation model fails to reach Dice ≥0.70 on MSSEG-2 validation — abort/redesign segmentation stage before any forecasting work.
- Day 55: Oracle ceiling (Arm D) shows <2-point headroom over Arm A — hypothesis is likely false in this regime; abort before external validation spend (saves ~60% of full budget).
- Day 90: Internal-cohort primary endpoint not significant (p≥0.05) — do not proceed to external validation; declare failure per pre-registered criteria.
- Day 120: External validation shows >50% effect shrinkage — flag as harmonization-artifact failure mode, halt before commercial-value claims are made.