CS 150: Generative Models

Generative models have achieved remarkable success across various applications, including Large Language Models (LLMs) and multimodal generative AI. This course introduces the probabilistic foundations of deep generative models and provides hands-on experience in coding these models.

Course goal

This course aims to introduce the foundamental of generative models and provide the initial experience of working with generative models.
By the end of the course, a student should have a theoretical understanding of generative models. A student should also be able to code and train a generative model for a small-scale generation problem.

Prerequisites

1. Probability theory: students should have knowledge of probability theory, which includes concepts of probability (density) functions, conditional/marginal probabilities, log-likelihood, and simple distributions (uniform distribution, Gaussian distribution). A prior course is strongly recommended.

2. Linear algebra and calculus: students should know basic operations over matrices and vectors, such as matrix-matrix multiplication, matrix-vector multiplication, the determinant of a square matrix (eigenvalue decomposition is not needed). Students should know the concepts of gradients, partial derivatives, and chain rule. A prior course is strongly recommended.

3. Programming: students should be comfortable with programming to multidimensional arrays with numpy, torch, or other pytchon packages. For example, let temp be an array with shape (T, H, W) representing the temperature over T time steps and an H x W area. Can you read out the temerature values at location (i, j) over the T steps? Can you get the mean temperature at a time step t? Can you find the maximum temperature over T steps for each of the H x W locations? Note that you need to use numpy or torch operations WITHOUT a for-loop in your code.

4. Machine learning and deep learning: the course will require several machine learning concepts including: training/testing split, minimization of training loss, gradient-based optimization. While not required, it is HIGHLY recommended that students have familiarity with elements of training deep neural networks (batch training, opimizer settings) and common neural network layers (fully-connected, convolutional, non-linear layers).

I hope you understand that generative models represent some of the most recent advancements in deep learning, and it’s natural that they build upon years of accumulated knowledge. While the list above demands a solid foundation, we’ll explore core concepts and techniques of generative learning throughout the course.

The prerequisites above have different emphrasis.

• For 1 and 2, I will briefly go through the materials to help you to recall your knowlege from prior courses. If you have not taken relevant courses, it is almost impossible for you to learn the materials while taking the course.
• For 3, you may learn the programming skills through plenty of practices. Please do so and get prepared before the course starts in the fall. In the class, you are assumed to have this skill to work on assignments.
• For 4, if you haven’t taken a machine learning course, you may need to use online materials to learn the concepts listed above. If you don’t have experiences with deep learning and torch programming, please check Chapter 3 of the d2l book. There you will see an example of learning a linear regression model (I hope you know linear regression already) with torch. For machine learning (not deep learning), I will briefly cover the concepts listed above. For deep learning, I will spend no more than two classes on torch programming and neural networks. Note that deep learning will not be the main focus of the course.

Modules

• Theoretical foundation: we cast the generation problme as a sampling problem: the learning model needs to 1) be abel to fit the data, and 2) have a relatively inexpensive sampling procedure.

  To develop this notion, the instructions will start with simple distributions and emphasize three aspects of a model: 1) how to parameterize a learning model; 2) with the parameterization, how to generate a new sample; and 3) with the parameterization, how to train the learning model.

  Course work: this module will have two written assignments.

• Generative models for language. We will introduce the transformer achitecture and sequential modeling. We will introduce the techniques for training a transformer model for language generation.

  Course work: this module has one coding assignment that requires the implementation of a language generative model.

• Generative models for images. We will cover three types of generative models for images: diffusion models, flow matching, and token-based image generation.

  Course work: this module has one coding assignment that requires the implementation of an image generative model.

• Generation with control. In many learning tasks, we need to control the behavior of the generative model. We will introduce two types of controlled generation: conditioning and RL-based control.

  Course work: in this part of work, students need to read at least two research papers and take quizzes about these research papers.

All coding work are REQUIRED to use LLMs. It means that you have the ability to understand the code from LLMs and make necessary corrections.

Grading

Your final grade will be caculated based your performance in coursework and participation.

• written assignment (30 %)
• coding assignment 1 (25 %)
• coding assignment 2 (25 %)
• quizzes from paper reading (10%)
• participation (10 %)

A separate note: the goal of the grading is to assign grades that reflect the understanding of the topic. The instructor and TAs will check students understanding based on the evidence in course work and class activies. Therefore, the students are expected to take effort to show such evidence, which includes (but is not limited to) asking questions in the class, writing clear assignments, making clear comments to the code, and actively participating class discussions.

Academic Integrity Policy

This course will strictly follow the Academic Integrity Policy of Tufts University. Students are expected to finish course work independently when instructed, and to acknowledge all collaborators appropriately when group work is allowed. Submitted work should truthfully represent the time and effort applied.

Please refer to the Academic Integrity Policy at the following URL: https://students.tufts.edu/community-standards/academic-integrity/academic-integrity-overview

Accessibility

Tufts and the instructor of CS 135 strive to create a learning environment that is welcoming students of all backgrounds and abilities. Respect is demanded at all times throughout the course. It is expected that everyone in the course is treated with dignity and respect. We realize everyone comes from a different background with different experiences and abilities. Our knowledge will always be used to better everyone in the class.

If you have a disability that requires reasonable accommodations, please contact the Student Accessibility Services office at Accessibility@tufts.edu or 617-627-4539 to make an appointment with an SAS representative to determine appropriate accommodations. Please be aware that accommodations cannot be enacted retroactively, making timeliness a critical aspect for their provision.

Please see the detailed accessibility policy at the following URL: https://students.tufts.edu/student-accessibility-services

If you feel uncomfortable or unwelcome for any reason, please talk to your instructor so we can work to make things better. If you feel uncomfortable talking to members of the teaching staff, consider reaching out to your academic advisor, the department chair, or your dean.