Low-rank approximation of Adam optimizer momentum matrices, analogous to matrix-interpolatory reduced-order models, will reduce memory overhead without degrading convergence on transformer architectures.

Replacing the full-rank first- and second-moment estimate matrices (m_t, v_t) in Adam with rank-r approximations (r << min(d_in, d_out) per weight matrix), constructed via an interpolatory/projection-based low-rank update scheme analogous to matrix-interpolatory reduced-order modeling (e.g., randomized range-finding or Krylov-subspace projection applied to the momentum update recursion), will (a) reduce optimizer state memory by a factor proportional to (d_in + d_out)/r per parameter tensor of shape (d_in, d_out), and (b) yield final validation loss/perplexity within 1% relative degradation and convergence-step-count within 5% of full-rank Adam, when training transformer models in the 50M–1.3B parameter range on standard language-modeling benchmarks.

Minimum viable test (MVT): Train a 125M-parameter GPT-2-style transformer on a 2–5B token subset of a standard corpus (e.g., OpenWebText or C4 slice) for 20K–50K steps, comparing (1) full-rank AdamW, (2) proposed low-rank-momentum optimizer at 2–3 rank settings, (3) Adafactor baseline, (4) GaLore baseline (if code available) as an existing low-rank-optimizer control. All runs matched on token budget, batch size, LR schedule, seed set (n=3 seeds). Primary endpoints: validation loss/perplexity trajectory, optimizer-state memory footprint (measured, not just theoretical), wall-clock time per step, gradient-cosine-similarity diagnostics between low-rank and full-rank momentum estimates.

class LowRankMomentumAdam(Optimizer):
    def __init__(self, params, lr, betas, rank_r, refresh_every=200):
        # for each 2D param tensor W (d_in x d_out):
        #   maintain U_m (d_in x r), V_m (r x d_out) factors for m_t
        #   maintain U_v (d_in x r), V_v (r x d_out) factors for v_t (or diag-approx)
        ...

    def step(self):
        for W, grad in params_with_grads:
            if W.ndim == 2 and min(W.shape) > rank_threshold:
                # project gradient onto current subspace
                g_proj = U_m.T @ grad  # (r x d_out)
                # interpolatory update (Krylov-style): extend subspace periodically
                if step_count % refresh_every == 0:
                    U_m, V_m = randomized_svd_update(U_m, V_m, grad, rank_r)
                m_factors = beta1 * m_factors + (1 - beta1) * g_proj
                v_factors = beta2 * v_factors + (1 - beta2) * g_proj**2
                update = U_m @ (m_factors / (sqrt(U_v @ v_factors) + eps))
                W -= lr * update
            else:
                # fall back to full-rank Adam for 1D params (bias, layernorm)
                standard_adam_step(W, grad)

Key design choices requiring empirical tuning: (a) subspace refresh schedule (every step = accurate but expensive; periodic = interpolatory ROM-style efficiency), (b) whether v_t is tracked low-rank or via Adafactor-style row/col factorization (hybrid may outperform pure low-rank), (c) initialization of subspace (random vs warm-start from first K full-rank steps).