Description
The goal of the assignment is to explore OpenAI Gym environments and implement value function approximation algorithms. In the first part of the project we will implement deep Q learning (DQN), following DeepMind’s paper that explains how reinforcement learning algorithms can learn to play Atari from raw pixels. The purpose of this project is to understand the effectiveness of deep neural networks as well as some of the techniques used in practice to stabilize training and achieve better performance. We will train our networks on two OpenAI gym or other complex environments. In the second part of the project we will implement an improvement to the DQN algorithm, focusing on Double Deep Q-learning (DDQN) or Prioritized Experience Replay (PER).
Part 1 – Implementing and applying DQN
1.1 Implement DQN
Implement DQN from scratch following DeepMind’s paper ([mnih2015human] and [mnih-atari-2013]). You may use Keras/Tensorflow/Pytorch.
1.2 Apply DQN to Complex Environments
Test your DQN algorithm on any TWO OpenAI Gym environments or Google Research Football. You may also use your custom made multi-agent environment or any other complex environemnt, that you will use for your Final Project (this has to be confirmed with the course staff). Compare the results. Describe the environments that you used (e.g. possible actions, states, agent, goal, rewards, etc). Provide reward dynamics (average reward in t-steps). Important notes:
- One of the environments has to be either Google Research Football or has to use CCN (Convolution Neural Network) for the state preprocessing (e.g. OpenAI Atari).
- The environment with multiple versions considers as one environment.
Suggested environments:
- OpenAI CartPole
- OpenAI LunarLander
- OpenAI Atari Breakout
- OpenAI MountainCar
- OpenAI Space Invadors
- Google Research Football
Part 2 – Improving DQN
DQN had a lot of success in applications in various domains. Some improvements have also been done to the vanilla (DQN) algorithm. In this part we will implement one of improved algorithms that is based on DQN. Modify your DQN code from Part 1.1 to one of the improved version and apply it to two environments, that were used in Part 1.2.
Algorithm, that built on top of vanilla DQN:
- Double DQN
- Dueling DQN
- Prioritized Experience Replay (PER)
