World Models Course

A self-study adaptation with videos papers Colab work and a research project

Prepared for Soham • Revised 13 September 2026

Purpose and scope

This revision follows a 23-topic sequence and supplies a self-study route for Soham. It assumes Python and basic machine learning experience but no previous world-model research. The main change is breadth: generative modeling, spatial prediction, physical simulation, robotics, and language agents now receive dedicated study units.

Course structure

The course has 23 lectures, three assignments, project milestones, a presentation, and a final assessment.

Study materials

The videos, paper selections, exercises, estimated workload, and assessment prompts in this pack are our supplemental self-study choices.

Pace and prerequisites

Use 16 study weeks with about 8 to 10 hours per week. The first 12 weeks cover the lectures; the project starts in week 6 and continues through week 16. The four generative-model lectures are dense: allow 12 hours in weeks 4 and 5 or stretch them. This follows lecture order at a self-study pace.

Optional preparation before lecture 1

Check conditional probability, matrix multiplication, gradients, train and validation splits, and training a small neural network. If needed, use MIT Introduction to Deep Learning and Deep Sequence Modeling, plus Berkeley CS285 Fall 2023 lecture 3 PyTorch Tutorial. Spend one or two extra weeks here instead of adding prerequisite material to the numbered lecture sequence.

A repeatable audiovisual study routine

Watch 45 to 90 minutes of the selected material, pausing every 15 minutes to draw the inputs, state, prediction, and objective. Read only the assigned paper portions on the first pass. Record a two-minute answer to the oral prompt and a three-minute explanation of one figure. Then complete the short practice task or the relevant assignment section.

How to use recordings

Select the named topic within the linked playlist and use its stated year. Study allocations are not verified video runtimes. Mila links are whole-day recordings: locate the named speaker using the program; event clock times are not recording timestamps. A demonstration is labeled as such.

Visual classroom

World_Models_Visual_Classroom.html gives every lecture an illustrated introduction, a slider experiment, plain-language maths and links to its guided lab. Use it before the linked recordings and papers. It is an original study aid, intended for self-study.

Colab and notebook order

Upload a notebook through Colab File then Upload notebook. Filenames retain their old numbers so existing work remains usable; follow the assignment mapping, not filename order. Notebook 05 begins filtering early, while notebook 01 planning is finished later. CPU sections need NumPy, scikit learn, and Matplotlib. Notebook 04 adds PushT dependencies and an optional DINOv2 GPU path.

What completing the course establishes

You should be able to identify a model’s state and action interface, distinguish reconstruction from prediction, implement small baselines, evaluate rollouts, and write a controlled research proposal. Reading frontier methods gives conceptual coverage; reproducing their large training runs is outside the teaching budget.

Study schedule

Week	Lectures or session	Work
1	1 and 2	Vocabulary and probabilistic foundations
2	3 and 4	Trajectory audit and filtering
3	5 and 6	Representations; release HW1
4	7 and 8	VAE GAN and autoregressive study
5	9 and 10	HW1 due after 9; release HW2 after 10
6	11 and 12	Capstone proposal after 11; begin C2; HW2 due after 12
7	13 and 14	Release HW3; begin project baseline
8	15 and 16	Geometry and temporal consistency
9	17 and 18	Physics and robot transfer
10	19 and 20	Action and language interfaces
11	21 and 22	Checkpoint before 21; HW3 due after 21
12	23 and synthesis	Evaluation protocol and failure analysis
13 and 14	Project work	Controlled comparisons and repeat runs
15	Sessions 24 and 25	Project presentation and final self-assessment
16	Final submission	Report code and next research proposal

The order below follows the 23 course topics.

Lecture 1 World model overview

Learn. Trace observation, internal state, action, predicted future, and decision through one agent. Distinguish a predictive model from the controller that uses it.

Watch and listen. Watch David Ha’s World Models author talk. Budget 45 to 75 minutes.

Resources: David Ha World Models author talk.

Required paper. World Models — Ha and Schmidhuber 2018. Read the abstract, architecture figure, and Vision Memory Controller explanation. Save optimization details for lecture 11.

Explain aloud. What question can a world model answer that a policy alone does not answer?

Practice. Sketch a robot, a video generator, and a text-game agent using the same five labels. No training yet.

Lecture 2 Probabilistic foundations and hidden state

Learn. Write a trajectory as conditional probabilities and explain why latent state or history may be necessary. Compare a one-step fit with a recursively predicted sequence.

Watch and listen. Use David Silver lectures 1 and 2 for the Markov property and partial observability. Budget 60 to 90 minutes.

Resources: David Silver RL lectures.

Required paper. Reinforcement Learning A Survey — Kaelbling et al 1996. Read the problem formulation and hidden-state discussion; this survey supplies the vocabulary, not a modern methods catalogue.

Explain aloud. Why can two identical observations have different futures under the same action?

Practice. Draw the factorization for three transitions and mark observed versus latent variables.

Lecture 3 Environment APIs and trajectory collection

Learn. Treat reset, step, action meaning, episode boundaries, and observation timing as part of the experiment. Learn why termination and time limits need different handling.

Watch and listen. Watch the agent–environment portion of MIT Deep Reinforcement Learning, then inspect the Gymnasium example. Budget 45 to 60 minutes.

Resources: MIT 6 S191 2026 Deep Reinforcement Learning; Gymnasium current environment API tutorial.

Required paper. OpenAI Gym — Brockman et al 2016. Read the benchmark-interface design and reproducibility discussion. Use current Gymnasium documentation for code; the 2016 paper predates its present API.

Explain aloud. Does each stored image describe the state before or after its paired action?

Practice. In notebook 01 inspect trajectory generation and the episode split. Stop before the planning sections. Prepare the HW1 data card.

Lecture 4 Filtering and state estimation

Learn. Separate transition and observation models. Compare a causal filtered estimate with a smoother that also uses later measurements; assess uncertainty during missing observations.

Watch and listen. Watch the linear Kalman portion of Cyrill Stachniss’s Kalman Filter and EKF lecture. Defer EKF derivations. Budget 60 to 90 minutes.

Resources: Cyrill Stachniss Kalman Filter and EKF lecture.

Required paper. Deep Kalman Filters — Krishnan Shalit and Sontag 2015. Read the introduction and state-space model formulation; identify transition and emission distributions. Return to variational inference after lecture 7.

Explain aloud. Why must a future-aware smoother not be reported as an online estimator?

Practice. Run notebook 05 Part A. Mask observations, plot filtered uncertainty, and compare filtering with smoothing. This is HW1 Part A.

Lecture 5 Learning visual representations

Learn. Compare what reconstruction, masking, contrastive training, and autoregressive objectives encourage an encoder to preserve. Use probes without treating their failure as proof of missing information.

Watch and listen. Watch the UvA SimCLR tutorial recordings. Reuse only its encoder and contrastive-loss explanations on the first pass. Budget 60 to 90 minutes.

Resources: UvA SimCLR tutorial with recordings and Colab.

Required paper. A Simple Framework for Contrastive Learning of Visual Representations — Chen et al 2020. Read the architecture, augmentation pipeline, and contrastive objective. Identify which transformations are encouraged to become irrelevant.

Companions: DINOv2 Learning Robust Visual Features without Supervision — Oquab et al 2023.

Explain aloud. When could an invariance that helps classification harm a robot’s prediction?

Practice. Run notebook 02 through its visual ambiguity and PCA sections. Compare frame and history features for HW1 Part B.

Lecture 6 Predictive embeddings and JEPA

Learn. Trace context encoder, target encoder, predictor, masking, and target updates. Separate passive video pretraining from learning how actions change a future.

Watch and listen. Use Mido Assran’s V-JEPA 2 session in Mila day 1 for an advanced explanation. Budget 60 to 90 minutes including pauses.

Resources: Mila workshop day 1 recording; Mila World Modeling Workshop 2026 program and recordings.

Required paper. Self Supervised Learning from Images with a Joint Embedding Predictive Architecture — Assran et al 2023. Read the model figure, masking design, and target construction. Explain the collapse problem before following training details.

Companions: V JEPA 2 Self Supervised Video Models Enable Understanding Prediction and Planning — Assran et al 2025; DINO WM — Zhou et al 2024 preprint and ICML 2025.

Explain aloud. What additional evidence is needed to turn a video predictor into an action-conditioned model?

Practice. Submit HW1 after lecture 9; its representation comparison can be completed now. Sketch a JEPA training path and a DINO-WM action interface.

Lecture 7 Latent generators and adversarial learning

Learn. Interpret the ELBO as a training objective and separate posterior collapse from GAN mode collapse. Contrast tractable bounds with adversarial training signals.

Watch and listen. Use MIT Deep Generative Modeling for a first pass; choose the VAE and GAN topics in Stanford CS236 for depth. Budget 75 to 100 minutes.

Resources: MIT 6 S191 2026 Deep Generative Modeling; Stanford CS236 2023 public lecture playlist; Stanford CS236 2023 topic index and slides.

Required paper. Auto Encoding Variational Bayes — Kingma and Welling 2013. Read the reparameterization method and ELBO. Also read the GAN algorithm and minimax objective in the companion paper; skip its proof on the first pass.

Companions: Generative Adversarial Networks — Goodfellow et al 2014.

Explain aloud. What does each model optimize, and which probability can you actually evaluate?

Practice. HW2 warm-up: derive a scalar Gaussian KL and compare a reconstruction-only model with a VAE design.

Lecture 8 Autoregressive generation

Learn. Factorize a joint distribution, trace causal masking, and distinguish parallel teacher-forced training from sequential sampling. Diagnose conditioning on imperfect generated history.

Watch and listen. Watch Stanford CS236 Autoregressive Models; use MIT Deep Sequence Modeling if causal sequence prediction is unfamiliar. Budget 60 to 90 minutes.

Resources: Stanford CS236 2023 public lecture playlist; Stanford CS236 2023 topic index and slides; MIT 6 S191 2026 Deep Sequence Modeling.

Required paper. Pixel Recurrent Neural Networks — van den Oord et al 2016. Read the probability factorization and architecture overview. Translate pixel ordering into an ordering of video tokens or state tokens.

Companions: Attention Is All You Need — Vaswani et al 2017.

Explain aloud. Why can low teacher-forced loss coexist with poor long generated sequences?

Practice. HW2 Part A: implement a small conditional token table on quantized toy trajectories and compare teacher-forced and sampled rollouts.

Lecture 9 Denoising and flow matching

Learn. Relate a noisy sample, its denoising target, a probability path, and a learned velocity. Distinguish time along a generative path from time in the physical world.

Watch and listen. Watch selected sections of MIT 6 S184 2025 lectures 1 and 3. Use lecture 2 only for a probability-path derivation that remains unclear. Budget 90 minutes.

Resources: MIT 6 S184 2025 recordings and Colab labs; MIT 6 S184 lecture 1 Flow and Diffusion Models; MIT 6 S184 lecture 3 Training Flow and Diffusion Models.

Required paper. Denoising Diffusion Probabilistic Models — Ho et al 2020. Read the forward noise process and training and sampling algorithms. In Flow Matching, read the conditional objective and probability-path figure.

Companions: Flow Matching for Generative Modeling — Lipman et al 2022 preprint.

Explain aloud. Is diffusion time the same as the time step predicted by a world model?

Practice. Finish HW1. For HW2 Part B use MIT 2025 Lab 2 on a 2D toy distribution; hold initial noise fixed while changing solver steps.

Lecture 10 Invertible and autoregressive flows

Learn. Use change of variables, invertibility, and triangular Jacobians to explain exact density training. Compare discrete flow layers with velocity-field sampling.

Watch and listen. Watch Stanford CS236 Normalizing Flows. Then inspect the TARFlow and STARFlow figures. Budget 75 to 100 minutes.

Resources: Stanford CS236 2023 public lecture playlist; Stanford CS236 2023 topic index and slides.

Required paper. Density estimation using Real NVP — Dinh et al 2016 preprint. Work through an affine coupling layer and log determinant. Read TARFlow’s architecture overview; STARFlow is the optional scaling follow-up.

Companions: Normalizing Flows are Capable Generative Models — Zhai et al ICML 2025; STARFlow — Gu et al 2025.

Explain aloud. Why can two models both be called flows while having different training and sampling costs?

Practice. HW2 Part C: implement and invert a 2D affine coupling layer, check the Jacobian numerically, and write a six-method comparison. HW2 is now formally released.

Lecture 11 Sequential latent generation

Learn. Combine deterministic memory and stochastic latent state. Identify prior and posterior inputs and explain why predicted trajectories must work without future images.

Watch and listen. Revisit the World Models memory explanation, then watch the PlaNet project’s author video. Budget 60 to 90 minutes.

Resources: David Ha World Models author talk; PlaNet author video and project.

Required paper. Learning Latent Dynamics for Planning from Pixels — Hafner et al 2018 preprint and ICML 2019. Read the recurrent state-space model, its training objective, and the rollout description. Trace one observed and one imagined step.

Companions: World Models — Ha and Schmidhuber 2018.

Explain aloud. Which path uses an observed image during training and is absent during imagination?

Practice. Write RSSM shape pseudocode and run the history comparison in notebook 02. Submit the capstone proposal between lectures 11 and 12.

Lecture 12 Planning and model based control

Learn. Compare MPC with CEM, trajectory optimization, and an actor trained on imagined futures. Diagnose model exploitation and the role of uncertainty.

Watch and listen. Select CS285 Fall 2023 lecture 10 planning and lecture 12 imagination/model-based policy learning; use Dreamer’s author material for the architecture. Budget 90 minutes.

Resources: Berkeley CS285 Fall 2023 playlist; Berkeley CS285 Fall 2023 lecture index; Dreamer author project and videos.

Required paper. Dream to Control — Hafner et al 2019 preprint and ICLR 2020. Read imagination and actor/value learning. In PETS, inspect candidate planning and ensemble dynamics; in MBPO, read the model-bias motivation.

Companions: PETS — Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models — Chua et al 2018; When to Trust Your Model — Janner et al 2019; Mastering Diverse Domains through World Models — Hafner et al 2023 preprint; Mastering Atari Go Chess and Shogi by Planning with a Learned Model — Schrittwieser et al 2019 preprint.

Explain aloud. Why might a larger search budget improve predicted return while reducing actual return?

Practice. Finish notebook 01 planning and notebook 03’s selected-error experiment as HW3 preparation. HW2 is due after this lecture.

Lecture 13 Video prediction architectures

Learn. Compare pixels, discrete tokens, and latent features as video prediction targets. Distinguish frame-level generation from chunks and causally conditioned blocks.

Watch and listen. Inspect DIAMOND demonstrations and Stephen Spencer’s Genie session in Mila day 2. These complement the generative foundations. Budget 60 to 90 minutes.

Resources: DIAMOND official project videos and code; Mila workshop day 2 recording; Mila World Modeling Workshop 2026 program and recordings.

Required paper. Diffusion for World Modeling Visual Details Matter in Atari — Alonso et al 2024. Read the world-model architecture and visual-detail experiments. Relate its denoising model to the action and frame history supplied at inference.

Companions: Genie Generative Interactive Environments — Bruce et al 2024; STARFlow V video modeling — Gu et al 2026.

Explain aloud. Which conditioning variables make a generated video a testable interactive prediction?

Practice. HW3 starts here. Make an architecture comparison for DIAMOND, Genie, and a latent predictor; label the prediction unit and action interface.

Lecture 14 Interactive video and long memory

Learn. Separate sequence length from persistent information. Study action alignment, drift, and the latency costs of producing a useful next decision.

Watch and listen. Use Sherry Yang’s Mila day 1 talk on world-model pretraining and physical-agent post-training; inspect the Dreamer 4 project videos. Budget 60 to 90 minutes.

Resources: Mila workshop day 1 recording; Mila World Modeling Workshop 2026 program and recordings; Dreamer 4 author project and videos.

Required paper. Training Agents Inside of Scalable World Models — Hafner et al 2025. Read the model and agent overview, temporal generation design, and efficiency evaluation. Treat large-scale training as a reading exercise.

Companions: V JEPA 2 Self Supervised Video Models Enable Understanding Prediction and Planning — Assran et al 2025.

Explain aloud. Which information must persist when an agent revisits an object after a long gap?

Practice. For HW3 compare short and longer rollouts at fixed context. Record error and runtime separately; formulate an occlusion failure test.

Lecture 15 Geometry and spatial representations

Learn. Project between camera and world coordinates. Compare depth, points, occupancy, radiance fields, and Gaussian primitives by the queries they support.

Watch and listen. Watch the NeRF ECCV technical talk. Budget 60 to 90 minutes.

Resources: NeRF ECCV technical talk; NeRF official project videos and code.

Required paper. NeRF — Mildenhall et al 2020. Read the scene parameterization, ray sampling, and rendering equation. Compare the representation figure in 3D Gaussian Splatting.

Companions: 3D Gaussian Splatting — Kerbl et al 2023.

Explain aloud. Why does the ability to render a new camera view not establish the ability to predict a new physical action?

Practice. Run notebook 05 Part B: project a small 3D scene, reconstruct it using depth, and change the camera pose. No full NeRF training is required.

Lecture 16 Dynamic scenes and persistent identity

Learn. Track scene elements through time and separate scene reconstruction from future dynamics. Examine occlusion, rigidity assumptions, and camera-versus-object motion.

Watch and listen. Watch the Dynamic 3D Gaussians trajectory and occlusion demonstrations. Budget 45 to 60 minutes.

Resources: Dynamic 3D Gaussians official demonstrations.

Required paper. Dynamic 3D Gaussians — Luiten et al 2023 preprint and 3DV 2024. Read the persistent Gaussian representation, local rigidity constraints, and tracking evaluation. Identify whether the method reconstructs observed motion or predicts unseen futures.

Explain aloud. What extra model and data would turn a dynamic reconstruction into an action-conditioned simulator?

Practice. Choose a moving point in a demonstration and sketch its identity through occlusion. For HW3 mark which geometry variables your predictor actually represents.

Lecture 17 Learning physical dynamics

Learn. Build relational dynamics from nodes and pairwise interactions. Compare a numerical simulator with a learned update, and measure error accumulation under a changed physical parameter.

Watch and listen. Inspect the graph simulator’s normal rollouts and failure videos. Then draw the graph. Budget 45 to 75 minutes.

Resources: Graph network simulator official videos and failure cases.

Required paper. Learning to Simulate Complex Physics with Graph Networks — Sanchez Gonzalez et al 2020. Read the graph construction, encode-process-decode model, and long-rollout/generalization experiments. Optional: compare the function-to-function view in Fourier Neural Operator.

Companions: Fourier Neural Operator — Li et al 2020 preprint.

Explain aloud. What can go wrong after many steps even when the average one-step error is small?

Practice. Run notebook 06 Part A. Compare a learned relational spring update with the reference simulator and a no-interaction baseline, including shifted stiffness.

Lecture 18 Robot dynamics and transfer

Learn. Relate system identification, learned models, feedback control, and sim-to-real assumptions. Separate robustness to observation changes from robustness to changed dynamics.

Watch and listen. Watch the DayDreamer author talk and a physical-robot example. Budget 60 to 90 minutes.

Resources: DayDreamer author talk; DayDreamer official project.

Required paper. DayDreamer — Wu et al 2022. Read the robot setup, online learning loop, and adaptation experiments. In Domain Randomization, read what is randomized and the transfer test.

Companions: Domain Randomization for Sim to Real — Tobin et al 2017.

Explain aloud. Does training on randomized images test robustness to a different friction coefficient?

Practice. For HW3 change one environment parameter at a time. For the PushT project, finish the offline action/state audit before extracting expensive features.

Lecture 19 Language conditioned robot actions

Learn. Compare VLA policies, latent action discovery, and models that predict both futures and actions. State exactly what is supervised by robot commands versus passive video.

Watch and listen. Watch MIT 6 S184 lecture 5 Generative Robotics and inspect RT-2 robot demonstrations. Use the world-action tutorial’s diagrams as an optional guide. Budget 75 to 90 minutes.

Resources: MIT 6 S184 lecture 5 Generative Robotics with Benjamin Burchfiel; RT 2 official robot demonstrations.

Required paper. RT 2 Vision Language Action Models — Brohan et al 2023. Read the action tokenization and training mixture. Compare its policy output with Dreamer 4’s model/agent diagram and Genie’s latent actions.

Companions: Training Agents Inside of Scalable World Models — Hafner et al 2025; Genie Generative Interactive Environments — Bruce et al 2024; From World Models to World Action Models — Zhang et al 2026 tutorial.

Explain aloud. What additional prediction would justify calling a VLA system a world model?

Practice. Draw observation, language, future state, and action targets for three systems. Specify a supervised target for each arrow.

Lecture 20 Language models and internal state

Learn. Test whether sequence representations support state recovery and intervention. Distinguish a fluent narrative, a decodable state, and a reliably executable transition prediction.

Watch and listen. Watch the state/interaction framing in Shunyu Yao’s Language Agents defense. Budget 45 to 75 minutes.

Resources: Language Agents — Shunyu Yao PhD defense.

Required paper. Emergent World Representations — Li et al 2022 preprint and ICLR 2023. Read the synthetic game setup, probe results, and intervention experiments. Identify evidence specific to this task and limits on generalizing to language models.

Explain aloud. What would distinguish knowing the current board from merely predicting likely next moves?

Practice. Run notebook 06 Part B. Compare current observation with remembered state on a small partially observed digital environment. The core experiment uses no LLM API.

Lecture 21 Deliberation and inference budgets

Learn. Separate generating a candidate, scoring it, verifying it, and obtaining new evidence. Compare branching search with recurrent computation at a fixed resource budget.

Watch and listen. Watch the Tree of Thoughts portion of Shunyu Yao’s defense. Budget 60 to 90 minutes.

Resources: Language Agents — Shunyu Yao PhD defense.

Required paper. Tree of Thoughts — Yao et al 2023. Read the thought decomposition, search procedure, and budget comparisons. In Universal Transformers, inspect recurrence over depth and adaptive halting as a different use of computation.

Companions: Universal Transformers — Dehghani et al 2018 preprint.

Explain aloud. When does more reasoning repeat the same wrong transition assumption?

Practice. Use notebook 06’s exact checker to audit candidate plans and notebook 03 to understand selection bias. Submit HW3 after this lecture; the capstone checkpoint is due just before it.

Lecture 22 Tool using digital agents

Learn. Model a digital environment’s state, actions, feedback, and partial observations. Evaluate success with an external checker and separate imagined outcomes from executed ones.

Watch and listen. Watch the ReAct portion of Yao’s defense and inspect its failure example. Optional: the OpenApps session with Mark Ibrahim and Karen Ullrich in Mila day 2. Budget 60 to 90 minutes.

Resources: Language Agents — Shunyu Yao PhD defense; ReAct official examples code and failure analysis; Mila workshop day 2 recording; Mila World Modeling Workshop 2026 program and recordings.

Required paper. ReAct — Yao et al 2022 preprint and ICLR 2023. Read the interaction loop, examples, and failure discussion. Trace which observations are returned by the environment and which text is generated by the agent.

Explain aloud. Why is a convincing predicted tool response insufficient evidence that the task succeeded?

Practice. Extend notebook 06 Part B with one additional hidden variable or failed action. Log proposed action, actual result, and corrected state separately.

Lecture 23 Evaluation and research questions

Learn. Choose metrics matched to the claimed utility. Test calibration, action response, distribution shift, latency, and failure modes without conflating visual quality with control.

Watch and listen. Inspect WorldScore’s visual evaluation examples. Explain one useful and one missing metric for your own domain. Budget 45 to 75 minutes.

Resources: WorldScore official visual benchmark.

Required paper. WorldScore — Duan et al 2025. Read the benchmark decomposition, evaluation dimensions, and limitations. In Statistical Precipice, read uncertainty and aggregate reporting guidance.

Companions: Deep Reinforcement Learning at the Edge of the Statistical Precipice — Agarwal et al 2021.

Explain aloud. Which result would falsify your main claim, and which baseline could explain it away?

Practice. Freeze the capstone protocol before test evaluation. Use whole-episode resampling in notebook 03; prepare the presentation and final self-assessment.

The three assignments

These are original self-study assignments designed for this course adaptation.

HW1 State inference and representation audit

Release: After lecture 6. Due: After lecture 9. Workload: 8 to 10 hours across three study weeks. Compute: CPU.

Start. Notebook 05 Part A, notebook 02, and the trajectory-generation section of notebook 01.

Implement. A: Compare raw observations, a Kalman filter, and a smoother under noise and missing observations. Label the smoother as an offline reference. B: Compare single-frame and history features on the paired visual dataset. C: Create an episode-level split and a data card that defines every action and timestamp. Include an encoder objective comparison grounded in SimCLR and I-JEPA.

Deliver. One notebook entry point, split record, three plots, a two-page analysis, and a three-minute explanation. State which improvements arise from more information rather than a larger model.

References: Deep Kalman Filters — Krishnan Shalit and Sontag 2015; A Simple Framework for Contrastive Learning of Visual Representations — Chen et al 2020; Self Supervised Learning from Images with a Joint Embedding Predictive Architecture — Assran et al 2023.

HW2 Generative objectives and sampling

Release: After lecture 10; warm-ups begin at 7. Due: After lecture 12. Workload: 10 to 14 hours including warm-ups. Compute: CPU for toy tasks; optional Colab GPU for image extension.

Start. Notebook 07 and GENERATION_ASSIGNMENT.md contain original scaffolds. Use MIT 6 S184 2025 Lab 2 for flow and score matching; the course page links its Colab.

Implement. A: Fit a small autoregressive token model on quantized trajectories; compare teacher-forced loss with free-running behavior. B: Complete the toy-distribution flow-matching portion of MIT Lab 2; plot quality versus solver steps with fixed initial noise. C: Implement and invert one 2D affine coupling layer with a correct log determinant. D: Compare VAE, GAN, autoregressive, DDPM, flow-matching, and finite-flow objectives and diagnose one failure mode for each. Training a small image VAE or GAN is an optional extension.

Deliver. Generated samples, rollout plots, a Jacobian check, measured sampling time, a six-method comparison, and a three-minute explanation. Do not compare raw likelihood values when data parameterizations differ.

References: Auto Encoding Variational Bayes — Kingma and Welling 2013; Generative Adversarial Networks — Goodfellow et al 2014; Pixel Recurrent Neural Networks — van den Oord et al 2016; Denoising Diffusion Probabilistic Models — Ho et al 2020; Flow Matching for Generative Modeling — Lipman et al 2022 preprint; Density estimation using Real NVP — Dinh et al 2016 preprint; MIT 2025 Lab 2 Flow Matching and Score Matching.

HW3 Predictive models and decision quality

Release: After lecture 13. Due: After lecture 21. Workload: 12 to 16 hours spread across application units. Compute: CPU core; optional small neural Colab extension.

Start. Notebook 01 planning and notebook 03 selection bias are the common core. Choose notebook 06 Part A physics or Part B digital state as your application extension. Notebook 05 Part B is the geometry preparation.

Implement. Common core: compare random shooting and CEM with fixed initial states, and evaluate selected actions using the true environment. Separate the learned-model experiment in notebook 01 from notebook 03’s deliberately biased illustrative surrogate. Application: quantify multi-step errors and a controlled shift in spring stiffness, or compare memory and transition prediction in the digital environment. Add one failed-case visualization and a short mapping to a video or robotics paper.

Deliver. A results table with baselines, error-versus-horizon plot, measured runtime, episode-based uncertainty, one failure demo, and a three-page analysis. Choose one application deeply; broad lecture coverage does not require implementing every architecture.

References: PETS — Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models — Chua et al 2018; When to Trust Your Model — Janner et al 2019; Learning to Simulate Complex Physics with Graph Networks — Sanchez Gonzalez et al 2020; Emergent World Representations — Li et al 2022 preprint and ICLR 2023; Deep Reinforcement Learning at the Edge of the Statistical Precipice — Agarwal et al 2021.

Project milestones

Milestone	Self-study timing	Evidence
Proposal	After lecture 11	C1–C4 plan, PushT episode split, baselines, fixed evaluation seeds, and compute budget.
Checkpoint	Before lecture 21	C1–C3 results, baseline comparisons, failure cases, and C4 training/evaluation progress.
Presentation	Session 24 in week 15	Eight-minute talk or narrated video plus questions; distinguish reproduced methods from your measurements.
Report and code	End of week 16	A 4 to 6 page report, reproducible entry point, split and seed records, figures, and a one-page next-experiment proposal.

Course capstone

Build a mini robot foundation model on PushT. This is the course capstone, delivered through four guided notebooks. Complete C1 through C4 and submit one combined report and code package. The representation study below is an optional research extension after the capstone.

Preparation

Use notebook 04 to explore the PushT data, confirm the train/validation/test split, and establish baselines before C1. Read capstone/README.md for the evaluation protocol and submission checklist.

The four stages

C1: Flow-matching action policy and regression baseline. Start after lecture 9.

C2: Frozen DINOv2 vision, comparing direct and modular policies. Continue after lecture 11.

C3: A latent world model and policy-proposal verification experiment. Continue after lecture 13.

C4: Fine-tune SmolVLA with LeRobot and compare it with the specialist policies. Continue after lecture 19.

Evaluation and submission

Keep the episode split and evaluation seeds fixed across stages. Report the shared PushT score, uncertainty, baselines, failure cases, and compute use. C1-C3 use a T4-class GPU; budget an A100/L4-class GPU for C4.

Submit a proposal after lecture 11 and a checkpoint before lecture 21. Present in week 15 and submit one 4-6 page report, code, split/seed records, results and failure figures, and a short demonstration video in week 16. Use the eight-minute course presentation for assessment; the three-minute video is a portfolio summary.

Optional research extension protocol

A focused study within the broader curriculum

Optional extension after the capstone: representation sufficiency for action-conditioned prediction on offline PushT. Ask whether retaining a small spatial grid of frozen DINOv2 features, or adding one previous frame, improves multi-step physical-state prediction under a matched small predictor budget. This remains a tractable way to practice research while the lectures cover more domains.

A falsifiable hypothesis

On held out PushT episodes, retaining a coarse spatial feature grid or a short observation history reduces future physical state prediction error relative to a global pooled single frame representation, under a fixed small predictor budget. A null result is useful: it may show that these changes add little in this dataset, or that predictor capacity and data coverage dominate the effect.

Dataset and initial scope

Use the publicly released simulated PushT demonstrations from the Diffusion Policy project. The archive contains image, state, action, and episode boundary arrays. Inspect the schema rather than assuming conventions from another PushT release. Notebook 04 checks the archive structure and samples whole episodes. No new environment design is required for the offline study. Demonstrations limit the action distribution; they do not reveal the outcome of arbitrary new plans.

One week to establish the baseline

Start with 30 to 60 training episodes and separate validation and test episodes; use a fixed shuffled episode split such as 70 percent 15 percent 15 percent. Save the episode IDs before any feature extraction. Plot several image state action sequences. Fit a persistence baseline, an action aware true state predictor, and a low resolution pixel probe. Confirm action alignment from the source implementation and a visualized episode. Do not infer it from whichever alignment improves test scores.

The minimal experiment matrix

Use one frozen DINOv2 ViT S 14 checkpoint and the same preprocessed frames for every condition. Primary matrix: global mean pooled features versus a 2 by 2 spatial pooling grid, each with either one frame or two frames of history. The four conditions are small approximations; a 2 by 2 grid is not the full patch representation used in DINO WM. First train a common dimension projection and matched small predictor. Include a capacity sensitivity check so a larger input or parameter count is not mistaken for better information.

What to train

Freeze the encoder and cache features in evaluation mode with no gradients. Fit the projection or PCA using training episodes only. A simple ridge or small MLP predicts the next feature conditioned on current features and actions; for history conditions concatenate the previous feature or use a small GRU as a later extension. Train a separate state probe on observed training features. Compare probe predictions for observed test features with probe predictions for rolled out features. A direct state prediction head is a diagnostic baseline, not a latent world model by itself.

Baselines and controls

Required controls are persistence, a predictor using physical state and action as an information rich reference, a low resolution pixel representation, a predictor without actions, and a shuffled action evaluation. The state predictor is not an oracle simulator and can have approximation error. The no action control exposes datasets where observations already predict most of the next frame. Do not assume shuffled actions must hurt when motion is slow or demonstrations make actions redundant.

Evaluation protocol

Use three model seeds on the same locked episode split for neural predictors. Evaluate horizons 1 5 and 10 using recorded action sequences. For open loop rollout, feed predicted features back in; only initialize with observed history. Do not use future observed frames to refresh the recurrent state. Report observed feature state probe error separately from rolled out feature state probe error. Decode block angle as sine and cosine, then use circular angle error. Report position error in source coordinate units.

The primary and secondary metrics

Choose mean block position error at horizon 5 as the primary metric and report rotation error separately. Secondary diagnostics are error versus horizon, no action and action shuffled gaps, and physical state differences among near neighbors in feature space. Average transition errors within each episode before averaging episodes. Bootstrap whole paired test episodes for a 95 percent interval; also report variation across model seeds. A bootstrap over episodes does not replace model seed variability.

Two errors to keep separate

Representation limitation means even observed features fail to support a sufficiently expressive state probe. Dynamics limitation means a probe works on observed features but fails on predicted rollouts. This is evidence, not a complete mathematical decomposition: the probe itself may be too weak or extrapolate poorly. Compare a linear and a small nonlinear probe before claiming the representation lacks information. Use the same physical unit metric across encoders; raw latent MSE from different representations is not directly comparable.

An honest optional planning extension

After the offline study works, use a released DINO WM PointMaze checkpoint with its matching code and data. Keep it frozen. Test nested candidate budgets and measure selected plan outcomes in the actual supplied simulator; compare random candidates with planner selected candidates. This extends the core project from recorded prediction to closed-loop decision evaluation. If PointMaze setup is blocked, defer the extension or use the official matched PushT setup. New candidate actions cannot be evaluated from unrelated offline demonstration endpoints.

Compute allocation

Treat a 24 GPU hour project budget as a ceiling to enforce after profiling, not an expected measured runtime. Reserve 2 hours for setup and a small timing run, 4 for one time feature extraction, 8 for the four condition predictor matrix, 4 for repeats and the capacity check, 2 for final evaluation, and 4 for failures or one limited extension. CPU data inspection and cached feature probes need not consume GPU allocation. If a 1000 frame profile forecasts an overrun, reduce episodes, frame resolution, or conditions before launching; keep three seeds and the locked test set where feasible.

Optional extension timing

Begin this optional representation study after completing the course capstone. Define its own hypothesis, baseline, compute budget, and evaluation plan; it adds no extra course deadline.

What you deliver

Deliver code with an environment record and one command or notebook entry point; a split manifest; configs and seeds; a results table with uncertainty; three figures covering information retention, rollout horizon, and representative failures; a short report; and a narrated demonstration. List your compute use and exactly which pretrained weights and data were used. An advisor should be able to distinguish reproduced behavior, your measurements, and your proposed next step.

What would justify scaling

A useful pilot reveals a reproducible failure, rules out a simple baseline explanation, and proposes a specific fix with a measured benefit or clear test. Before claiming novelty, compare against DINO WM representation ablations, memory aware models, and other representation evaluations. A funding pitch should ask for a named follow up, such as a second dataset or matched full patch predictor, with a cost estimate from your measured throughput. Funding is a possible outcome, not a promise of the capstone.

Reading anchors: DINO WM — Zhou et al 2024 preprint and ICML 2025; DINOv2 Learning Robust Visual Features without Supervision — Oquab et al 2023; When to Trust Your Model — Janner et al 2019; Deep Reinforcement Learning at the Edge of the Statistical Precipice — Agarwal et al 2021; Diffusion Policy — Chi et al 2023.

Further optional research ideas

These are optional research ideas after completing C1–C4. They are not additional capstone requirements. Use the same controls and evaluation discipline if you explore one.

Physical dynamics

Use notebook 06 Part A to study whether a relational inductive bias improves rollout stability under a held-out stiffness or larger graph. Add noise and a nonlinear model only after the analytic and learned linear baselines work.

Language and digital agents

Use notebook 06 Part B to measure state-tracking and transition errors under longer histories or missing feedback. Add a fixed small language model as an extension and compare history-only, explicit-state, and verifier-assisted conditions on identical episodes. Log model/version, prompts, costs, and actual tool outcomes.

Video and spatial prediction

Study one released small model or cached predictions under camera motion versus object motion. Separate reconstruction, future extrapolation, and action-conditioned tests. New-view quality alone cannot establish physical predictive accuracy.

Session 24 Project presentation

Give an eight-minute narrated presentation with a question, method figure, baseline comparison, failure case, and next experiment. Reserve five minutes for questions or record yourself answering likely objections.

Session 25 Final self-assessment

Suggested format: 90 minutes, eight open-ended questions, followed by a spoken defense. These are original practice prompts. Use Final_Assessment_Notes.md after attempting the questions.

Question 1 Observation and state

Give two histories with the same latest image and different futures. Specify a state or belief that resolves the ambiguity.

Question 2 Filtering

Write a linear transition and observation model. Explain why smoothing has information that online filtering lacks.

Question 3 Representations

Compare a pixel reconstruction objective with a joint-embedding objective. Give a task-specific reason to preserve spatial structure.

Question 4 Generative objectives

Explain one training objective each for a VAE, GAN, autoregressive model, and flow model. Distinguish generative time from environment time.

Question 5 Planning

Design a controlled experiment where you increase candidate search budget. State how you will distinguish optimization gains from model exploitation.

Question 6 Space and physics

Explain why a dynamic 3D reconstruction need not predict a new action. Specify what additional data, model, and test would be required.

Question 7 Robots and language agents

Contrast a VLA policy, an action-conditioned world model, and a ReAct agent. Identify the source of ground-truth feedback in each.

Question 8 Research evaluation

Propose a falsifiable claim, baseline, capacity control, distribution shift, uncertainty estimate, and compute-bounded next experiment.

Glossary

Observation. What the agent receives, such as an image; it may hide some of the physical state.

State. Information sufficient for predicting the environment transition under an action, in the assumed model.

Latent representation. A learned or designed code for observations or history; it need not be a true physical state.

Belief or predictive state. A summary of past observations and actions used to reason about hidden information.

Dynamics model. A predictor of a future state, observation, or representation conditioned on the available context and usually actions.

Policy and planner. A policy maps context to actions; a planner evaluates or optimizes candidate future actions.

MPC. Model predictive control: plan ahead, execute a short action prefix, observe, and replan.

CEM. Cross entropy method: repeatedly sample candidate sequences and refit a distribution to the better ones.

Teacher forcing. Training or predicting with actual prior observations or states rather than only the model own previous predictions.

Open loop rollout. Predict repeatedly without correcting from new real observations.

Closed loop control. Act, receive a new observation, update the estimate, and act again.

Representation collapse. The encoder emits nearly identical features for different inputs, making prediction easy but uninformative.

State aliasing. Different underlying states map to the same or very similar representation even though their relevant futures differ.

Probe. A small supervised model used to test what information is accessible in a representation.

Out of distribution. Inputs or action sequences that differ materially from those used to fit the model.

Ablation. A controlled change to one component that helps explain an observed effect.

Filtering. Estimating current hidden state from observations available so far.

Smoothing. Estimating a state using a longer observation sequence that can include later observations.

Radiance field. A representation queried at spatial locations and viewing directions to render images.

Scene dynamics. How spatial structure and properties evolve over environment time.

Flow matching. Training a continuous generative flow by fitting a vector field along specified probability paths.

External verifier. A checker whose evidence comes from a defined task or environment, rather than only from the generator’s own judgment.