Rates of Convergence in a Class of Native Spaces for Reinforcement Learning and Control

A. Bouland, S. Niu, S.T. Paruchuri, A.J. Kurdila, J.A. Burns, E. Schuster

American Control Conference (ACC)

Toronto, Canada, July 8-12, 2024

*Published in IEEE Control Systems Letters

Abstract

This letter studies convergence rates for some value function approximations that arise in a collection of reproducing kernel Hilbert spaces (RKHS) H. By casting an optimal control problem in a specific class of native spaces, strong rates of convergence are derived for the operator equation that enables offline approximations that appear in policy iteration. Explicit upper bounds on error in value function and control law approximations are derived in terms of power function P_{H,N} for the space of finite dimensional approximants H_N in the native space H. These bounds exhibit a distinctive geometric nature, refine and build upon some well-known, now classical results concerning the convergence of approximations of value functions.