Statistical Efficiency and Inference of Quantile Distributional Reinforcement Learning

This paper analyzes the statistical efficiency of quantile-based distributional reinforcement learning, specifically focusing on distributional policy evaluation to characterize the return distribution.

• The authors construct an estimator based on an empirical Markov decision process assuming access to a generative model.
• They establish a non-asymptotic error bound under the supremum W_infinity metric, showing estimation error scales as O(sqrt(m/n)).
• The study derives the asymptotic distribution of quantile parameters and characterizes the semiparametric efficiency bound attained by the estimator.
• In the diverging quantile regime, the limit covariance structure matches the semiparametric efficiency bound of the nonparametric model.
• A Berry-Esseen theorem is established for smooth functionals to provide a foundation for statistically valid inference.

The work demonstrates that quantile-based estimators achieve optimal parametric convergence rates and remain asymptotically efficient in infinite-dimensional limits.