This project is no longer accepting applications. Subscribe to our newsletter to be notified of new projects!
Develop a reinforcement learning controller for a lunar lander and test it in simulation!
From industrial automation with robotic arms to optimizing machine movement, in recent years many entry level mechanical engineering jobs in the US started requiring knowledge of machine learning based controlling techniques. In this Build Project, you will wear the hat of Mechanical Engineer and you will develop an interactive Google Colab with reinforcement learning based controller that will control a simulated lunar lander. Under the supervision of an experienced industry expert you’ll learn the basic principles of reinforcement learning controllers and you will implement one from scratch using Python in a simulation environment and test it. You will become familiar with how mechanical engineers approach designing controllers for their mechanical systems and learn how to use common industry tools like Python. All this will happen in an environment that simulates the operations of a real mechanical and control engineering team.
We will start by exploring real life examples of reinforcement learning (RL) like walking “robotic dogs”! Next, we will cover foundational RL topics like state, action, environment as well as exploration-exploitation trade-off. Lastly in small groups we will identify how real-life physical systems can be formulated as RL problems.
Modern RL controllers are built based on neural networks (NN). In the second workshop, we will look at the basics of neural networks. We will cover forward pass, backpropagation, SGD optimizer and different model structures. We will also code and train one from scratch using Pytorch!
After implementing a simple training loop, this week we will experiment with different hyperparameters like learning rate, layer sizes or optimizers. We will experiment to see how much we can improve our NN. At the end of the class the best students will briefly present their solutions and talk about their reasoning behind it!
With high level grounding in RL and NN next we will learn about Deep Q-Networks. We will review the famous “Playing Atari with Deep Reinforcement Leaning” paper from DeepMind and review the Deep Q-Networks (DQNs) algorithms. Then we will try to implement our own in Pytorch.
In week 5 we will try to apply our knowledge directly with the OpenAI Gym’s Lunar Lander environment to implement and test simple RL algorithms. We will learn how to interact with the simulator, send commands and receive actions. By the end of the session, you should see your simulated lander flying!
Once we have a simple lunar lander working with DQN in a simulated environment we will learn about and the code in Pytorch two common techniques used to improve the performance and stability of our algorithm – experience replay and targe networks.
In this session we will finish up all our DQN with experience replay and target networks and we will focus on tunning parameters of our lunar landers, fixing any possible bugs and experimenting with different neural network architectures for our RL controllers!
In the final week everyone will present recorded GIF videos of their lunar lander. We will run a small contest with prizes for winner! After that we will review a case study of real life. We will review RL from Human Feedback (RLHF) i.e. mechanism used to tune OpenAI ChatGPT and architecture behind it – Proximal Policy Optimization (PPO).
Get access to all of our Build projects, including this one, by creating your Build account!
Get started by submitting your application.
We'll notify you when projects reopen. In the meantime, you can explore our resources and learn more about our Fellows.
Igor Sadalski is a Mechanical Engineering Build Fellow at Open Avenues, where he works with students leading projects in mechanical engineering, robotics and AI.
Igor is an AI Data Scientist at Somite.ai, where he focuses on both theoretical and engineering aspects of building transformer based foundational models for biology.
Igor has over 4 years of experience in the Engineering and Sciences field. Already in highschool, with his team, he won a silver medal for Poland in the international Young Physicist Tournament. During his undergraduate studies he twice received research scholarships from Caltech, where he worked on robotics and AI. Recently he was a visiting researcher at Harvard University working on efficient transformer based architectures.
He holds a Bachelor of Mechanical Engineering from Imperial College London.
A fun fact about Igor is that the family on his mother's side has a small honey business in Poland!