solver.press

Adaptive sampling algorithms from structural optimization can improve the efficiency of LLM-driven zeroth-order optimization.

PhysicsMar 10, 2026Evaluation Score: 62%

Adversarial Debate Score

57% survival rate under critique

Expert panel critique

Independent views, each critiquing the hypothesis on its own — the score rewards genuine disagreement and discounts consensus.

Gemini: The hypothesis is plausible and falsifiable, with some support from the papers on adaptive sampling and LLM-driven optimization. However, the connection between structural optimization algorithms and LLM optimization efficiency needs further clarification and empirical validation.
ChatGPT: It’s falsifiable (compare query/sample efficiency of LLM-driven zeroth-order optimizers with vs. without structural-optimization-style adaptive sampling), and AdaEvolve supports the general idea that adaptive schedules help, but the cited structural optimization work targets ROM surrogate accurac...
Claude: The hypothesis connects two real research threads (AdaEvolve's adaptive LLM-driven zeroth-order optimization and the structural optimization adaptive sampling paper), but the papers provided show no direct integration or empirical bridge between these domains, and the conceptual transfer faces si...

Supporting Research Papers

Formal Verification

Z3 logical consistency:✅ Consistent

Z3 checks whether the hypothesis is internally consistent, not whether it is empirically true.

Experimental Validation Package

This discovery has a Claude-generated validation package with a full experimental design.

Precise Hypothesis

Adaptive sampling strategies borrowed from structural/materials optimization (specifically quasi-random low-discrepancy sequences, trust-region methods, and Bayesian surrogate-assisted sampling) can reduce the number of LLM inference calls required to find a near-optimal solution in zeroth-order optimization tasks by ≥30% compared to uniform random sampling baselines, while achieving equivalent or superior final objective quality (within 5% of best-found optimum), across at least three distinct benchmark problem classes (prompt optimization, hyperparameter search, and combinatorial text planning).

Disproof criteria:
  1. PRIMARY DISPROOF: Adaptive sampling achieves <15% reduction in LLM calls to reach 95% of the best-found objective across all three benchmark classes, with p > 0.05 (Wilcoxon signed-rank test, n ≥ 30 independent runs per condition).
  2. QUALITY DISPROOF: Final solution quality under adaptive sampling is >5% worse (absolute) than uniform random sampling at matched call budgets in ≥2 of 3 benchmark classes.
  3. SURROGATE FAILURE: GP surrogate model achieves Spearman rank correlation <0.3 between predicted and actual LLM objective scores on held-out validation points (n=50), indicating the landscape is not learnable.
  4. OVERHEAD DISPROOF: Wall-clock time including surrogate fitting exceeds uniform sampling wall-clock time by >50% at budgets ≤100 LLM calls, eliminating practical utility.
  5. GENERALIZATION FAILURE: Gains observed on one benchmark class (e.g., prompt optimization) do not transfer to a second class (e.g., hyperparameter search) — i.e., effect size Cohen's d < 0.2 on the second class.
  6. REPLICATION FAILURE: Results are not reproducible across two distinct LLM backends (e.g., GPT-4o and Llama-3-70B) with the same algorithm configuration.

Experimental Protocol

PHASE A — Baseline Establishment (Days 1–7): Implement uniform random sampling (URS) as the control optimizer. Run URS on three benchmark tasks: (A1) APE/PromptBench prompt optimization (maximize accuracy on GSM8K subset, n=200 problems), (A2) ML hyperparameter optimization (maximize validation F1 on a fixed tabular dataset using LLM-suggested configs), (A3) combinatorial text planning (maximize ROUGE-L on CNN/DailyMail summarization with LLM-selected sentence orderings). Record objective value vs. LLM call number for 30 independent seeds per task, budgets of 25/50/100/200/500 calls.

PHASE B — Adaptive Sampler Implementation (Days 8–21): Implement four adaptive strategies: (B1) Sobol sequence initialization + GP-EI active learning, (B2) Latin Hypercube Sampling warm-start + TuRBO local optimization, (B3) CMA-ES adapted for discrete/mixed LLM search spaces, (B4) Thompson Sampling with GP surrogate. Each strategy is applied to all three benchmark tasks under identical budget constraints and seeds.

PHASE C — Ablation and Sensitivity Analysis (Days 22–35): Ablate: surrogate model type (GP vs. random forest vs. neural network), acquisition function (EI vs. UCB vs. PI), initialization size (5/10/20% of budget), and LLM temperature (0.0/0.5/1.0). Measure sensitivity of efficiency gains to each factor.

PHASE D — Cross-LLM Replication (Days 36–42): Replicate best-performing adaptive strategy on GPT-4o-mini, Llama-3-70B (via Together AI), and Mistral-7B to test generalization across model families.

Required datasets:
  1. GSM8K (math reasoning benchmark) — 8,500 problems; used for prompt optimization objective; publicly available via HuggingFace datasets.
  2. CNN/DailyMail summarization dataset — 300,000 article-summary pairs; use 1,000-item held-out test split; HuggingFace datasets.
  3. OpenML-CC18 tabular benchmark suite — 72 classification datasets; select 5 with n=1,000–10,000 samples for hyperparameter optimization task; openml.org.
  4. PromptBench / APE evaluation harness — GitHub microsoft/promptbench; used as evaluation wrapper.
  5. BayesOpt benchmark library (BoTorch v0.10+) — for GP surrogate and acquisition function implementations; pip install botorch.
  6. BBOB (Black-Box Optimization Benchmark) functions — for synthetic landscape validation of surrogate quality; pip install ioh.
  7. LLM API access: OpenAI GPT-4o-mini (primary, $0.15/1M input tokens), Together AI Llama-3-70B ($0.90/1M tokens), Mistral-7B-Instruct (local or API).
  8. Optuna / Ax framework — for TuRBO and CMA-ES baselines; pip install optuna ax-platform.
Success:
  1. PRIMARY: ≥1 adaptive strategy achieves ≥30% reduction in C95 vs. URS on ≥2 of 3 benchmark tasks, with Wilcoxon p < 0.0042 (Bonferroni-corrected) and Cohen's d ≥ 0.5 (medium effect).
  2. QUALITY: Final objective quality of best adaptive strategy is within 2% (absolute) of URS at matched full budget on all 3 tasks.
  3. SURROGATE: GP surrogate achieves mean Spearman ρ ≥ 0.5 on held-out validation points across ≥80% of experimental runs.
  4. REPLICATION: Relative efficiency gain ≥20% is observed on ≥2 of 3 LLM backends (GPT-4o-mini, Llama-3-70B, Mistral-7B).
  5. OVERHEAD: Surrogate fitting + acquisition optimization adds <15% to total wall-clock time at budget=100 LLM calls.
  6. ABLATION: GP surrogate outperforms random forest surrogate by ≥10% in C95 reduction, confirming that principled uncertainty quantification is the active ingredient.
Failure:
  1. Relative efficiency gain <15% on all 3 tasks for all 4 adaptive strategies (null result).
  2. Final solution quality degrades >5% vs. URS at any budget level on ≥2 tasks.
  3. Surrogate Spearman ρ < 0.3 in >40% of runs (landscape not learnable by GP).
  4. Surrogate overhead exceeds 50% of total wall-clock time at budget=100.
  5. Cross-LLM replication fails: efficiency gain <10% on Llama-3-70B and Mistral-7B simultaneously.
  6. CMA-ES and Sobol+GP both fail (p > 0.05) — indicating the failure is not algorithm-specific but structural.
  7. Ablation shows no significant difference between GP and uniform random surrogate (ρ_ablation p > 0.1), suggesting observed gains are due to initialization bias rather than adaptive sampling.

12

GPU hours

42d

Time to result

$420

Min cost

$2,800

Full cost

ROI Projection

Commercial:
  1. AUTOML INTEGRATION: Adaptive LLM-driven hyperparameter optimization could be integrated into AutoML platforms (Google Vertex AI, AWS SageMaker, Azure ML) as a premium feature. Market size for AutoML: $1.1B (2024) growing to $6.4B (2030); efficiency improvements are a key differentiator.
  2. PROMPT ENGINEERING TOOLS: Companies like PromptLayer, Weights & Biases (Prompts), and LangSmith could integrate adaptive sampling to reduce customer API costs — a direct monetizable feature.
  3. LLM API PROVIDERS: OpenAI, Anthropic, and Google could offer "efficient optimization mode" using adaptive sampling as a value-added service, reducing customer churn due to cost concerns.
  4. ENTERPRISE AI OPS: For enterprises running LLM-based optimization pipelines (e.g., ad copy generation, drug discovery prompt optimization, code generation tuning), 30% cost reduction on a $500K/year LLM budget = $150K direct savings.
  5. PATENT POTENTIAL: The specific combination of low-discrepancy sequences + GP surrogate + LLM zeroth-order optimization is likely patentable as a novel method; estimated licensing value $500K–$2M over 10 years.
  6. OPEN-SOURCE LIBRARY: A well-packaged Python library (e.g., "AdaptLLM-Opt") could achieve 5,000–20,000 GitHub stars within 2 years, driving consulting and enterprise support revenue.

🔓 If proven, this unlocks

Proving this hypothesis is a prerequisite for the following downstream discoveries and applications:

  • 1adaptive-llm-hyperparameter-optimization-production-2026
  • 2surrogate-assisted-llm-agent-planning-2026
  • 3low-discrepancy-sampling-for-rlhf-reward-optimization-2026
  • 4structural-optimization-transfer-to-nlp-search-2026

Prerequisites

These must be validated before this hypothesis can be confirmed:

  • zeroth-order-llm-optimization-baseline-benchmark-2024
  • botorch-discrete-space-extension-2023
  • llm-prompt-optimization-ape-2023

Implementation Sketch

# ============================================================
# AdaptiveLLMOptimizer — Core Architecture Sketch
# ============================================================

import numpy as np
from botorch.models import SingleTaskGP
from botorch.acquisition import ExpectedImprovement
from botorch.optim import optimize_acqf
from botorch.utils.transforms import normalize, unnormalize
from scipy.stats import qmc  # Sobol, LHS
import torch
from abc import ABC, abstractmethod
from typing import List, Tuple, Dict, Any, Callable

# ---- 1. Abstract Optimizer Interface ----
class ZerothOrderOptimizer(ABC):
    def __init__(self, search_space: Dict[str, Tuple], budget: int):
        self.search_space = search_space  # {param: (low, high, type)}
        self.budget = budget
        self.dim = len(search_space)
        self.X_observed = []  # List of normalized configs
        self.Y_observed = []  # List of scalar objectives
    
    @abstractmethod
    def suggest(self, n_points: int = 1) -> List[Dict]:
        """Return n_points configs to evaluate next."""
        pass
    
    def observe(self, config: Dict, score: float) -> None:
        x_norm = self._normalize(config)
        self.X_observed.append(x_norm)
        self.Y_observed.append(score)
    
    def best(self) -> Tuple[Dict, float]:
        idx = np.argmax(self.Y_observed)
        return self._unnormalize(self.X_observed[idx]), self.Y_observed[idx]
    
    def _normalize(self, config: Dict) -> np.ndarray:
        return np.array([(config[k] - v[0]) / (v[1] - v[0]) 
                         for k, v in self.search_space.items()])
    
    def _unnormalize(self, x_norm: np.ndarray) -> Dict:
        return {k: x_norm[i] * (v[1] - v[0]) + v[0] 
                for i, (k, v) in enumerate(self.search_space.items())}

# ---- 2. Uniform Random Sampling Baseline ----
class UniformRandomSampler(ZerothOrderOptimizer):
    def suggest(self, n_points: int = 1) -> List[Dict]:
        configs = []
        for _ in range(n_points):
            x = np.random.uniform(0, 1, self.dim)
            configs.append(self._unnormalize(x))
        return configs

# ---- 3. Sobol + GP-EI Adaptive Sampler ----
class SobolGPEISampler(ZerothOrderOptimizer):
    def __init__(self, search_space, budget, init_fraction=0.1):
        super().__init__(search_space, budget)
        self.n_init = max(5, int(budget * init_fraction))
        self.sobol = qmc.Sobol(d=self.dim, scramble=True)
        self.initialized = False
        self.model = None
    
    def suggest(self, n_points: int = 1) -> List[Dict]:
        n_obs = len(self.X_observed)
        
        # Phase 1: Sobol initialization
        if n_obs < self.n_init:
            x_sobol = self.sobol.random(n_points)  # shape (n_points, dim)
            return [self._unnormalize(x) for x in x_sobol]
        
        # Phase 2: GP-EI active learning
        self._fit_gp()
        candidates = self._optimize_acquisition(n_points)
        return [self._unnormalize(c.numpy()) for c in candidates]
    
    def _fit_gp(self):
        X = torch.tensor(np.array(self.X_observed), dtype=torch.double)
        Y = torch.tensor(np.array(self.Y_observed), dtype=torch.double).unsqueeze(-1)
        Y_norm = (Y - Y.mean()) / (Y.std() + 1e-8)
        self.model = SingleTaskGP(X, Y_norm)
        # MLE hyperparameter fitting
        from botorch.fit import fit_gpytorch_mll
        from gpytorch.mlls import ExactMarginalLogLikelihood
        mll = ExactMarginalLogLikelihood(self.model.likelihood, self.model)
        fit_gpytorch_mll(mll)
        self.best_f = Y_norm.max()
    
    def _optimize_acquisition(self, n_points: int):
        EI = ExpectedImprovement(model=self.model, best_f=self.best_f)
        bounds = torch.stack([torch.zeros(self.dim), torch.ones(self.dim)])
        candidates, _ = optimize_acqf(
            acq_function=EI,
            bounds=bounds.double(),
            q=n_points,
            num_restarts=20,
            raw_samples=512,
        )
        return candidates.detach()
    
    def surrogate_quality(self) -> float:
        """Leave-one-out Spearman rho for surrogate validation."""
        from scipy.stats import spearmanr
        if len(self.X_observed) < 10:
            return np.nan
        preds = []
        for i in range(len(self.X_observed)):
            X_loo = [x for j, x in enumerate(self.X_observed) if j != i]
            Y_loo = [y for j, y in enumerate(self.Y_observed) if j != i]
            X_t = torch.tensor(np.array(X_loo), dtype=torch.double)
            Y_t = torch.tensor(np.array(Y_loo), dtype=torch.double).unsqueeze(-1)
            m = SingleTaskGP(X_t, Y_t)
            x_test = torch.tensor(self.X_observed[i], dtype=torch.double).unsqueeze(0)
            with torch.no_grad():
                pred = m.posterior(x_test).mean.item()
            preds.append(pred)
        rho, _ = spearmanr(preds, self.Y_observed)
        return rho

# ---- 4. LLM Oracle Wrapper ----
class LLMOracle:
    def __init__(self, task: str, model: str = "gpt-4o-mini", 
                 temperature: float = 0.0):
        self.task = task
        self.model = model
        self.temperature = temperature
        self.call_count = 0
        self.call_log = []
    
    def evaluate(self, config: Dict) -> float:
        """Convert config dict to prompt, call LLM, return objective score."""
        prompt = self._config_to_prompt(config)
        response = self._call_llm(prompt)
        score = self._score_response(response)
        self.call_count += 1
        self.call_log.append({
            'call_idx': self.call_count,
            'config': config,
            'score': score,
            'model': self.model
        })
        return score
    
    def _config_to_prompt(self, config: Dict) -> str:
        # Task-specific prompt construction
        if self.task == "gsm8k_prompt_opt":
            return self._build_math_prompt(config)
        elif self.task == "hyperparam_opt":
            return self._build_hyperparam_prompt(config)
        elif self.task == "text_planning":
            return self._build_planning_prompt(config)
    
    def _call_llm(self, prompt: str) -> str:
        import openai
        client = openai.OpenAI()
        response = client.chat.completions.create(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
            temperature=self.temperature,
            max_tokens=512
        )
        return response.choices[0].message.content
    
    def _score_response(self, response: str) -> float:
        # Task-specific scoring (BLEU, accuracy, F1, etc.)
        raise NotImplementedError

# ---- 5. Experiment Runner ----
class ExperimentRunner:
    def __init__(self, optimizer: ZerothOrderOptimizer, 
                 oracle: LLMOracle, budget: int, seed: int):
        self.optimizer = optimizer
        self.oracle = oracle
        self.budget = budget
        self.seed = seed
        np.random.seed(seed)
        torch.manual_seed(seed)
    
    def run(self) -> Dict:
        trajectory = []  # (call_idx, best_score_so_far)
        best_score = -np.inf
        
        for call_idx in range(self.budget):
            # Suggest next config
            configs = self.optimizer.suggest(n_points=1)
            config = configs[0]
            
            # Evaluate via LLM
            score = self.oracle.evaluate(config)
            
            # Update optimizer
            self.optimizer.observe(config, score)
            
            # Track best
            best_score = max(best_score, score)
            trajectory.append({
                'call_idx': call_idx + 1,
                'score': score,
                'best_score': best_score,
                'surrogate_rho': (self.optimizer.surrogate_quality() 
                                  if hasattr(self.optimizer, 'surrogate_quality') 
                                  else np.nan)
            })
            
            # Abort checkpoint: surrogate quality check at call 20
            if call_idx == 19 and hasattr(self.optimizer, 'surrogate_quality'):
                rho = self.optimizer.surrogate_quality()
                if not np.isnan(rho) and rho < 0.1:
                    print(f"ABORT: Surrogate rho={rho:.3f} < 0.1 at call 20. "
                          f"Landscape not learnable. Seed={self.seed}")
                    break
        
        return {
            'seed': self.seed,
            'optimizer': type(self.optimizer).__name__,
            'task': self.oracle.task,
            'budget': self.budget,
            'trajectory': trajectory,
            'best_config': self.optimizer.best()[0],
            'best_score': self.optimizer.best()[1],
            'total_llm_calls': self.oracle.call_count
        }

# ---- 6. Efficiency Metric Computation ----
def compute_c95(trajectory: List[Dict], global_best: float) -> int:
    """Calls-to-95%-optimum metric."""
    threshold = 0.95 * global_best
    for point in trajectory:
        if point['best_score'] >= threshold:
            return point['call_idx']
    return len(trajectory)  # Never reached threshold

def compute_relative_efficiency(c95_baseline: float, c95_adaptive: float) -> float:
    return (c95_baseline - c95_adaptive) / c95_baseline * 100.0

# ---- 7. Main Experiment Grid ----
def run_full_experiment():
    import itertools, pandas as pd
    
    TASKS = ["gsm8k_prompt_opt", "hyperparam_opt", "text_planning"]
    BUDGETS = [25, 50, 100, 200, 500]
    SEEDS = list(range(30))
    OPTIMIZERS = {
        'URS': UniformRandomSampler,
        'SobolGPEI': SobolGPEISampler,
        # 'TuRBO': TuRBOSampler,  # implement analogously
        # 'CMAES': CMAESSampler,
        # 'Thompson': ThompsonSampler,
    }
    
    results = []
    for task, budget, seed, (opt_name, OptClass) in itertools.product(
            TASKS, BUDGETS, SEEDS, OPTIMIZERS.items()):
        
        search_space = get_search_space(task)  # task-specific
        oracle = LLMOracle(task=task, model="gpt-4o-mini", temperature=0.0)
        optimizer = OptClass(search_space=search_space, budget=budget)
        runner = ExperimentRunner(optimizer, oracle, budget, seed)
        
        result = runner.run()
        results.append(result)
        
        # Save incrementally
        pd.DataFrame(results).to_parquet("results/experiment_log.parquet")
    
    return results

# ---- 8. Statistical Analysis ----
def analyze_results(results_df):
    from scipy.stats import wilcoxon
    from scipy.stats import spearmanr
    import statsmodels.stats.multitest as mt
    
    tasks = results_df['task'].unique()
    optimizers = [o for o in results_df['optimizer'].unique() if o != 'URS']
    
    p_values = {}
    effect_sizes = {}
    
    for task in tasks:
        for opt in optimizers:
            urs_c95 = results_df[(results_df['task']==task) & 
                                  (results_df['optimizer']=='URS')]['c95'].values
            opt_c95 = results_df[(results_df['task']==task) & 
                                  (results_df['optimizer']==opt)]['c95'].values
            
            stat, p = wilcoxon(urs_c95, opt_c95, alternative='greater')
            d = (np.mean(urs_c95) - np.mean(opt_c95)) / np.std(urs_c95 - opt_c95)
            
            p_values[(task, opt)] = p
            effect_sizes[(task, opt)] = d
    
    # Bonferroni correction
    keys = list(p_values.keys())
    pvals = [p_values[k] for k in keys]
    reject, pvals_corrected, _, _ = mt.multipletests(pvals, method='bonferroni')
    
    return {k: {'p_raw': p_values[k], 'p_corrected': pvals_corrected[i], 
                'reject': reject[i], 'cohens_d': effect_sizes[k]}
            for i, k in enumerate(keys)}
Abort checkpoints:

CHECKPOINT 1 — Day 3 (After URS Pilot, n=5 seeds, budget=50, 1 task): Abort condition: Mean C95 for URS is <10 calls (task too easy) or =budget (task too hard, no method reaches 95% optimum). Action: Replace task with harder/easier variant before full experiment.

CHECKPOINT 2 — Day 10 (After 50 Sobol+GP runs, budget=100, task=gsm8k): Abort condition: Surrogate Spearman ρ < 0.2 across >60% of runs at call 30. Interpretation: GP cannot model the LLM objective landscape. Action: Switch to random forest surrogate or abandon GP-based methods; pivot to CMA-ES only.

CHECKPOINT 3 — Day 15 (After 150 adaptive runs, all tasks, budget=50): Abort condition: Relative efficiency gain <10% for all 4 adaptive strategies on all 3 tasks (no signal). Action: Conduct emergency ablation — test whether the search space encoding is the problem (try alternative encodings). If still <10%, declare null result and pivot to characterizing why the hypothesis fails.

CHECKPOINT 4 — Day 21 (After full Phase B, before ablations): Abort condition: Best adaptive strategy achieves <20% efficiency gain AND final solution quality is >3% worse than URS. Interpretation: Adaptive sampling is actively harmful (over-exploitation). Action: Abort ablation phase; focus remaining resources on failure mode analysis and write-up of negative result (publishable as "LLM objective landscapes resist surrogate modeling").

CHECKPOINT 5 — Day 35 (After ablations, before cross-LLM replication): Abort condition: Ablation shows that the efficiency gain is entirely attributable to initialization (Sobol init alone, without GP, achieves same C95 as full Sobol+GP). Interpretation: Low-discrepancy initialization is the active ingredient, not adaptive sampling per se. Action: Reframe hypothesis — publish as "Quasi-random initialization improves LLM zeroth-order optimization" (narrower but valid claim); skip cross-LLM replication of GP component.

CHECKPOINT 6 — Day 38 (Cross-LLM replication, after Llama-3-70B runs): Abort condition: Efficiency gain on Llama-3-70B is <5% (vs. ≥30% on GPT-4o-mini). Interpretation: Effect is model-specific, not general. Action: Restrict claims to GPT-4o-class models; add boundary condition; do not run Mistral-7B to save cost ($180 saved).

Source

AegisMind Research
Need AI to work rigorously on your problems? AegisMind uses the same multi-model engine for personal and professional use. Get started