Statistical Guarantees for Flexible Machine Learning
Nuisance-Robust Learning and Inference
Learned models increasingly serve as measurements, objectives, and inputs to further decisions—not just as predictors. For example, a model trained on human judgments can become an evaluator or a reward for training language models; a predictor trained on laboratory measurements can guide which experiments are performed next. These uses help overcome a practical bottleneck: measurements and experiments are expensive, whereas predictions can be produced at scale. The statistical challenge is therefore to use limited observations to support evaluations and decisions at scale while controlling the errors introduced by learned models.
Good prediction does not guarantee good decisions or valid inference—and good decisions and valid inference do not always require highly accurate predictions. What matters is how model errors affect the final task. An optimizer can favor precisely the candidates a model overestimates, while plugging learned functions into an estimator can introduce bias that overwhelms its sampling uncertainty, even when average prediction errors are small. Conversely, some errors have little effect on the final target or can be corrected, relaxing the accuracy required of individual components. The relevant accuracy requirements must therefore be determined by how estimation errors affect the final task. This reasoning becomes even more important under adaptivity, where decisions also shape the data from which later conclusions are drawn. Early estimation errors and random outcomes can influence what is observed next, changing the sampling process itself. Consequently, correcting model error alone need not restore valid confidence intervals: one must also establish which features of the evolving sampling process matter for the final estimates and decisions, and how to control them.
My research develops the statistical foundations of machine learning for reliable inference and effective decision-making. I characterize how estimation errors propagate through inference and optimization, establish when adaptive sampling preserves inferential efficiency, and design corrections when familiar procedures fail. An important possibility is that the final procedure can perform well under weaker requirements than its individual components might suggest: a bias-corrected estimator can converge faster than its learned nuisance functions, and efficient inference can survive without the entire adaptive design stabilizing. My work identifies and exploits these distinctions in new estimation and learning problems. The broader aim is to extract more benefit from flexible models and available data without demanding more accuracy, stability, or fresh measurements than the task requires.
Statistical Foundations of Inference after Adaptive Experiments
Adaptive experiments use past outcomes to choose future assignments, then reuse the resulting trajectory for inference. For estimators whose leading term is a martingale sum, a Gaussian limit typically requires two ingredients: no individual increment is asymptotically influential, and the normalized quadratic variation—or, under mild conditions, the accumulated conditional variance—stabilizes. Adaptive allocation can violate the second requirement because the realized assignment path may preserve non-negligible information about earlier outcome fluctuations; in fixed-horizon examples, an inverse-propensity estimator with known assignment probabilities then converges to a non-Gaussian mixture of normals. It can also violate the first requirement when some assignment probabilities are of order $1/T$, allowing a few inverse-propensity-weighted observations to retain non-vanishing influence. Consequently, naive Wald intervals based on the standard martingale Gaussian approximation can be miscalibrated.
I develop procedures that restore validity while preserving the information accumulated by the adaptive design. For structured outcomes, I construct a learn-then-test procedure that estimates an informative witness direction, projects a doubly robust RKHS score onto it, and applies predictable variance normalization, yielding calibrated distributional tests under adaptive assignment [AI1]. In my recent work, I introduce directional stability, a target-specific condition requiring only the design information relevant to the parameter of interest—formalized through its Riesz representer—to stabilize, rather than the entire information matrix. To characterize efficiency in this setting, I develop local asymptotic normality and convolution theory for sequences of experiments whose adaptive sampling laws change with the horizon. Within this theory, I show that directional stability is sufficient for classical one-step estimators to remain asymptotically normal and attain the semiparametric efficiency bound, without generic adaptive reweighting [AI2].
Questions:
How can distributional tests remain calibrated when adaptive assignment changes the covariance geometry of the outcome representation? [AI1]
What target-specific stability is sufficient for classical one-step inference to remain valid and efficient under adaptive sampling? [AI2]
How should semiparametric efficiency be defined when the sampling law itself changes with the experiment? [AI2]
Distributional Causal Inference for Structured Outcomes
Most causal inference reduces intervention effects to means, quantiles, or other scalar summaries. This can miss changes in dispersion, modes, tails, or rare configurations, while the field still lacks a general statistical theory and broadly applicable machine-learning methods for causal inference with images, sequences, graphs, and other structured outcomes. I address this gap by developing kernel-based procedures that represent entire interventional outcome distributions while remaining robust to estimation errors in propensity and conditional outcome models.
I first introduced a global, doubly robust representation and test for determining whether two interventions induce the same outcome distribution [SO1]. An omnibus rejection, however, establishes that the distributions differ without revealing how. I therefore developed a second procedure that learns informative outcome prototypes and evaluates the discrepancy at those locations, yielding an interpretable and semiparametrically efficient test that localizes the effect while remaining valid after data-driven prototype selection [SO2] (NeurIPS 2026).
How can entire interventional outcome distributions be represented, estimated, and compared for structured outcomes? [SO1]
How can a global distributional difference be localized and interpreted without sacrificing statistical efficiency or validity after data-driven selection? [SO2]
Nuisance-Robust Learning and Inference for Nested and Inverse Problems
I develop nuisance-robust methods for finite-dimensional functionals of solutions to nested and inverse problems. In mediation and bilevel optimization, these functionals are built from iterated regressions or a lower-level population optimizer and its derivative; in proxy and instrumental-variable analysis, they are built from bridge or least-squares solutions to conditional moment equations. Plug-in estimation is fragile: errors propagate through nested layers, while inverse operators discard information, making exact solutions unstable, non-unique, or potentially nonexistent. I therefore target the final functional directly using orthogonal scores, Riesz corrections, complementary bridge equations, and cross-fitting.
For mediation, I developed a kernel-localized orthogonal estimator for continuous-treatment mediated response curves [N1]. A companion comparative mediation study benchmarks classical, multiply robust, and double-machine-learning estimators across binary, continuous, and multidimensional mediators, including under model misspecification, weak overlap, and in a UK Biobank application [N2]. For latent confounding, in joint work we developed density-ratio-free doubly robust bridge estimators based on kernel mean embeddings [N3]. In more recent joint work on instrumental variables, we introduced a projected parameter defined through primal and dual population least-squares problems. It remains well defined when the exact moment equations have no solution, is invariant to non-unique least-squares minimizers, and coincides with the usual structural estimand under exact specification [N4]. We subsequently extended the bridge approach to neural representations and continuous or structured treatments [N5]. I also developed the first semiparametric efficiency theory for the unregularized population bilevel gradient, together with an orthogonal estimator controlled uniformly over the outer parameter so that it can serve as a statistical gradient oracle [N6].
Recent questions
What should nonparametric IV analysis estimate when its moment equations are only approximate? Can a projected least-squares parameter recover the usual estimand under exact specification and still support debiased inference otherwise? [N4]
How can the unregularized population hypergradient be estimated without the first-order bias of differentiating fitted lower-level models, and controlled uniformly enough to guide optimization? [N6]
Selected Papers
[N4] Nonparametric Instrumental Variable Analysis Without Structural Equations: Debiased Inference on Functionals of Inverse Problems with No Solutions
Zikai Shen, Nathan Kallus, Dimitri Meunier, Houssam Zenati, Arthur Gretton, Aurélien Bibaut. Preprint, 2026. [Paper]
[N6] Semiparametric Efficient Bilevel Gradient Estimation
Fares El Khoury*, Houssam Zenati*, Nathan Kallus, Michael Arbel, Aurelien Bibaut.
* Marks authors who contributed equally to the paper. Preprint, 2026. [Paper]
Statistical and Algorithmic Foundations of Policy Learning
I develop statistical and algorithmic foundations for policy learning across fully online, batched-online, and offline contextual-bandit settings. From fixed logged data, I first studied direct policy optimization with continuous actions, using smooth importance-weighted objectives and proximal methods to stabilize a nonconvex learning problem [PL1]. In online contextual bandits, I then made nonparametric value-based learning computationally scalable through incremental Nyström approximations [PL2], and studied how hierarchical similarities between actions can reduce the effective complexity of exploration [PL3].
On the policy-based side, I introduced in Sequential CRM a Hölderian error-bound principle linking policy suboptimality to importance-weight variance. Combined with variance-sensitive updates and limited redeployment, this yields fast excess-risk and regret rates for parametric policy classes [PL4]. I also studied how covariance information revealed by semi-bandit observations can improve online exploration [PL5]. Returning to offline continuous-action learning, I completed the pipeline with joint context–action kernel models and principled offline model selection and evaluation [PL6].
Building on these foundations, my recent work extends the fast-rate principle to fully online, agnostic learning over rich nonparametric policy classes: O2PL establishes the first fast best-in-class regret rates in this setting [PL7]. I also introduce a semiparametric natural-gradient construction that debiases the learned policy itself, rather than merely estimating or optimizing a debiased policy value [PL8]. This recent work asks:
What structural conditions allow fully online, agnostic contextual bandits to attain fast regret against the best policy in a rich nonparametric class, without reward-model realizability? [PL7]
Can policy learning itself, not merely the evaluation of a fixed policy, be semiparametrically targeted through a functional natural-gradient flow, so that rich policy classes attain root-N regret while policy approximation and environment-estimation errors interact only through second-order product remainders? [PL8]
[PL7] Fast Best-in-Class Regret for Contextual Bandits
Samuel Girard, Aurélien Bibaut, Jill-Jênn Vie, Arthur Gretton, Nathan Kallus, and Houssam Zenati. UAI, 2026. [Proceedings]
Generative Representation Learning and Adversarial Optimization
My earlier work in adversarial learning first focused on generative representation learning for anomaly detection and semi-supervised prediction. I initiated and led a line of work that made GAN-based anomaly detection practical at test time. I introduced an amortized detector that replaced the costly per-example latent optimization used by earlier GAN methods with direct encoder-based inference, achieving state-of-the-art results while reducing test-time computation by several hundred-fold [AR1]. I also contributed to work showing how GANs can encode data-manifold geometry through a Monte Carlo approximation of Laplacian regularization for semi-supervised learning [AR2], [AR3].
I then developed ALAD, strengthening bidirectional adversarial learning with data- and latent-space cycle consistency, spectral normalization, and discriminator-based representations to improve anomaly detection across image and tabular benchmarks [AR4]. In retinal imaging, I applied semi-supervised representation learning to reduce reliance on expert annotations for abnormality classification [AR5]. This line of work became a widely cited reference point in generative anomaly detection, and ALAD was subsequently used beyond the original benchmarks.
A complementary line of work examined optimization in adversarial min–max problems. It characterized the last-iterate dynamics of mirror descent under coherence, showed why standard updates can cycle even in bilinear games, and established convergence of optimistic corrections across the broader class of coherent problems [AR6]. I also studied practical unsupervised anomaly detection for retinal images, where clinically relevant abnormalities may be sparse or localized [AR7].
Questions:
How can adversarial generative models make anomaly detection computationally practical, avoiding per-example latent optimization at inference time? [AR1]
How can generators encode data-manifold geometry and exploit unlabeled observations for semi-supervised learning? [AR2], [AR3]
How can bidirectional adversarial representations improve anomaly detection across image and tabular data? [AR4]
How can semi-supervised representation learning reduce expert-annotation requirements in medical imaging? [AR5]
Under what structural conditions do optimistic first-order methods achieve last-iterate convergence in adversarial min–max problems? [AR6]
How can unsupervised representation learning support practical detection and localization of abnormalities in retinal images? [AR7]
