All projects

MSc thesis · Reinforcement learning · 2023

E3B on MiniGrid

Adaptation of Meta AI's E3B exploration method (Exploration via Elliptical Episodic Bonuses) to MiniGrid, integrated into PPO and A2C to solve sparse-reward, partially observable tasks on modest hardware.

  • 5MiniGrid environments
  • 5agents compared: A2C, PPO, RIDE, NovelD and E3B
  • No GPUtrained on a home computer

Highlights

  • Implemented PPO and A2C baselines and integrated E3B's elliptical episodic bonus into both.
  • Used an autoencoder to reduce the dimensionality of observations and make E3B practical on low-compute systems.
  • Benchmarked on MiniGrid MultiRoom, DoorKey and ObstructedMaze: E3B improved convergence speed and mean reward over the baselines, approaching more recent state-of-the-art methods.

The problem

Reinforcement-learning agents struggle when rewards are rare and the world is only partially observable: they keep repeating what they already know instead of exploring. E3B rewards the agent for visiting states that are new within the current episode, measured with an elliptical bonus over learned state embeddings.

What I did

Reviewed the literature on exploration and intrinsic rewards, implemented A2C and PPO, integrated E3B into both, and simplified the method (with an autoencoder for dimensionality reduction) so it could be trained without large compute budgets.

Results

Adding E3B improved both baselines in convergence speed and mean reward on the harder environments, and the simplified version performed close to newer state-of-the-art algorithms. Key result: in the three hardest environments (MultiRoom-N4, MultiRoom-N6 and ObstructedMaze), where A2C and PPO never learned a viable policy — zero reward after millions of steps — E3B did, needing about 2.5× the training steps of NovelD, a newer state-of-the-art method, on a home computer without a GPU.

Technologies

  • Reinforcement learning
  • PPO
  • A2C
  • E3B
  • Autoencoders
  • MiniGrid
  • Python

Let's talk

Hiring for data or AI? I'd love to hear about the role.