Adaptive gradient sampling inspired by uncertainty-aware reduced-order models can reduce the number of expensive function evaluations needed in zeroth-order LLM optimization.

This discovery has a Claude-generated validation package with a full experimental design.

Precise Hypothesis

Spine & Adversarial ReadReady for validation

“This hypothesis tests whether replacing isotropic random-direction sampling with an uncertainty-adaptive, low-rank subspace sampling scheme in zeroth-order LLM optimization reduces the number of forward-pass function evaluations needed to reach a fixed accuracy target compared to a matched-budget isotropic ZO baseline. ---”

• highThe claimed benefit may be entirely attributable to low-rank/subspace restriction of the perturbation directions (a well-known variance-reduction trick in ES/CMA-ES literature), with the 'uncertainty-aware ROM' framing adding no measurable additional gain — making the core novelty claim unsupported.
  Addressed via mandatory ablation (Step 6, low-rank-only vs. uncertainty-weighted) with an explicit success criterion (≥5% additional gain, p<0.05) required to validate the uncertainty-awareness claim specifically; if this ablation fails, the EVP's failure criteria correctly reclassify the finding as 'low-rank ZO,' not novel ROM-inspired uncertainty sampling.
• mediumWhy LoRA-restricted fine-tuning (low intrinsic dimensionality) rather than full-parameter ZO — the methodology choice may be selecting a regime specifically favorable to low-rank/adaptive methods, limiting generalizability of any positive result to the broader claim about 'zeroth-order LLM optimization' generally.
  Partially addressed: protocol includes full-parameter MeZO as an ablation baseline and explicitly restricts the falsifiable hypothesis's strongest form to LoRA/adapter-based fine-tuning; however, the EVP does not yet include a dedicated full-parameter-scale test with adequate power, so claims beyond adapter-tuned regimes remain unresolved and should be stated as an explicit scope limitation in any publication.
• mediumThe 'uncertainty-aware reduced-order model' terminology borrowed from computational physics/UQ may not map cleanly onto a stochastic gradient estimation problem — the analogy could be post-hoc branding of an existing active-subspace/natural-ES idea rather than a substantively new mechanism, and no named prior-art comparison against active-subspace methods (Constantine et al.) or guided evolution strategies is included in the protocol.
  Not resolved in current EVP — no live search results were available to confirm specific competing papers; a required addition before publication is a direct empirical comparison against at least one published active-subspace or covariance-adaptive ES baseline (e.g., CMA-ES, guided ES) to establish genuine incremental novelty beyond terminology.

Experimental Protocol

ROI Projection

Implementation Sketch

# Baseline: MeZO-style SPSA
for step in range(T):
    z = sample_isotropic_gaussian(dim=d)
    loss_plus  = forward(theta + eps*z)
    loss_minus = forward(theta - eps*z)
    grad_est = (loss_plus - loss_minus) / (2*eps) * z
    theta -= lr * grad_est

# Proposed: Adaptive uncertainty-aware low-rank ZO
U = init_random_basis(d, r)          # low-rank basis
Sigma = init_uncertainty(r)          # per-direction uncertainty (ROM-style)
for step in range(T):
    probs = softmax(Sigma / temperature)      # sample high-uncertainty dirs more
    idx = sample_direction(probs, r)
    z = U[:, idx]
    loss_plus  = forward(theta + eps*z)
    loss_minus = forward(theta - eps*z)
    grad_est = (loss_plus - loss_minus) / (2*eps) * z
    theta -= lr * grad_est

    # ROM-style uncertainty/basis update (streaming PCA / Kalman-like)
    residual = compute_residual(grad_est, U, Sigma)
    Sigma = update_uncertainty(Sigma, residual)      # shrink for well-estimated dirs
    if step % K == 0:
        U, Sigma = refresh_subspace(U, Sigma, recent_grad_estimates)  # ROM basis adaptation