PufferLib 2.0: Reinforcement Learning at 1M steps/s

Joseph Suarez

Evaluation, Benchmarks Wednesday, August 6 Poster #41 Accepted — RLC 2025

Abstract

PufferLib is an open-source reinforcement learning project built around efficient and

broadly compatible simulation. Our first-party suite of 12 environments each run at 1M

steps/second. For existing environments, PufferLib provides one-line wrappers that eliminate

common compatibility problems and fast vectorization to accelerate training. With PufferLib,

you can use familiar libraries like CleanRL and SB3 to scale from classic benchmarks like

Atari and Procgen to complex simulators like NetHack and Neural MMO 3.