Skip to Main content Skip to Navigation
Conference papers

Value learning from trajectory optimization and Sobolev descent: A step toward reinforcement learning with superlinear convergence properties

Abstract : The recent successes in deep reinforcement learning largely rely on the capabilities of generating masses of data, which in turn implies the use of a simulator. In particular, current progress in multi body dynamic simulators are underpinning the implementation of reinforcement learning for endto-end control of robotic systems. Yet simulators are mostly considered as black boxes while we have the knowledge to make them produce a richer information. In this paper, we are proposing to use the derivatives of the simulator to help with the convergence of the learning. For that, we combine model-based trajectory optimization to produce informative trials using 1st-and 2nd-order simulation derivatives. These locally-optimal runs give fair estimates of the value function and its derivatives, that we use to accelerate the convergence of the critics using Sobolev learning. We empirically demonstrate that the algorithm leads to a faster and more accurate estimation of the value function. The resulting value estimate is used in model-predictive controller as a proxy for shortening the preview horizon. We believe that it is also a first step toward superlinear reinforcement learning algorithm using simulation derivatives, that we need for end-to-end legged locomotion.
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03356261
Contributor : Amit Parag Connect in order to contact the contributor
Submitted on : Monday, September 27, 2021 - 6:38:16 PM
Last modification on : Tuesday, October 19, 2021 - 11:18:03 PM

File

icra_2021.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03356261, version 1

Citation

Amit Parag, Sébastien Kleff, Léo Saci, Nicolas Mansard, Olivier Stasse. Value learning from trajectory optimization and Sobolev descent: A step toward reinforcement learning with superlinear convergence properties. International Conference on Robotics and Automation (ICRA 2022), May 2022, Philadelphia, United States. ⟨hal-03356261⟩

Share

Metrics

Record views

51

Files downloads

105