Your application

Please complete the following fields to be considered for this project.

Please fill in this required field.
Please fill in this required field.
Please fill in this required field.
Please fill in this required field.
Please fill in this required field.
How much commitment will you have to this project?
Please select an option.
Are you available to dedicate 1-2 hours per week to the Build Project?
Please select an option.
Your application has been 
successfully submitted!
Explore more projects
Close
You already submitted an application for this project.
Explore more projects
Close
There was an error submitting your form. Please try again later or contact us.
Oops! Something went wrong while submitting the form.

This project is no longer accepting applications. Subscribe to our newsletter to be notified of new projects!

Get updates
Build and Train Transformer Language Model
Kacper Raczy
Kacper Raczy
Get updates
Register today
Apply now

Build and Train Transformer Language Model

Utilizing the current state of the art techniques in natural language processing, you’ll develop a text generation language model featuring the attention mechanism (using GPT and transformer architecture).

Register today
Apply now
Tuesdays
 at
6:00
P.M.
 ET /
3:00
P.M.
PT
8 weeks, 2-3 hours per week
Expert
No experience required
No experience required
Some experience required
Degree and experience required

Description

With transformers dominating the machine learning world by setting new benchmarks in various NLP (natural language processing) tasks and Vision tasks, this Build Project will help you learn about transformers, specifically GPT models (which are powering the AI chat assistance like ChatGPT).

We will delve into the intricacies of transformer architecture using PyTorch. You will learn to build, train, and deploy GPT models from scratch, understanding their superior capabilities compared to prior NLP baselines. This knowledge is valuable in today’s job market where the demand for AI expertise is rapidly growing.

Session timeline

  • Applications open
    December 1, 2024
  • Application deadline
    January 15, 2025
  • Project start date
    Week of July 8, 2024
    Week of
    February 3, 2025
  • Project end date
    Week of

What you will learn

  • Acquire familiarity with PyTorch machine learning library, visualization tools like matplotlib and plotly and python programming environment
  • Learn how to collect, preprocess, and tokenize text data for training a language model.
  • Implement and compare baseline models or traditional NLP methods to highlight the improvements offered by transformer-based approaches.
  • Gain a deep understanding of the components and functioning of transformer models, including self-attention, multi-head attention, and positional encoding.
  • Develop and train your own character level generative pretrained transformer (GPT) model for specific task like creating Shakespear inspired stories, generating code snippets or composing music lyrics

Project workshops

1
Introduction
2
Data Collection, Preprocessing, and Tokenization
3
Baseline Models for Language processing
4
Introduction to Transformer Models
5
Training the Transformer Model
6
Evaluating Model Outputs and Practical Deployment Strategies
7
Fine-Tuning and Hyperparameter Tuning
8
Presentations

Prerequisites

  • Prior programming experience in Python, and experience working with Python tools and environments
  • Some first exposure to additional libraries like numpy, pandas and matplotlib
  • Familiarity with machine learning concepts like feedforward networks, activation functions, loss functions, and backpropagation.
  • Familiarity with statistical concepts such as probability distributions and basic understanding of linear algebra, enabling students to understand and interpret the inner workings of the model effectively.

Sign up today

Get access to all of our Build projects, including this one, by creating your Build account!

Register today
Log in

Apply to

Kacper

's project today!

Get started by submitting your application.

Apply now

Stay updated!

Subscribe to our newsletter to be notified when projects reopen!

Please fill in this required field.
By clicking “Subscribe” you agree to our Terms of Services and Privacy Policy.

Thanks for subscribing!

We'll notify you when projects reopen. In the meantime, you can explore our resources and learn more about our Fellows.

Discover our articles
There was an error submitting your form. Please try again later or contact us.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
About the expert

Kacper is a Data Science Build Fellow at Open Avenues where he works with students leading projects in Data Science.

Kacper is a Research Engineer at comma.ai where he focuses on driving evaluation infrastructure and driving-related metrics.

Kacper has over 5 years of experience in the Data Science and Software Engineering fields. Started in mobile app space as iOS engineer, then pivoted to machine learning to make driving chill.

He holds an M.S. in Computer Science.

A fun fact about Kacper, in his free time, he enjoys piloting aircrafts and surfing.

Visit
Kacper
's Linkedin
More like this Project