Probabilistic Programming Seminar 2023-2024

Description

Probabilistic programming languages (PPLs) use the syntax and semantics of programming languages to define probabilistic models. Using computer programs as a representation of probabilistic models yields two benefits. First, any computable process can be modelled as a probabilistic program. Second, PPLs enable a diverse audience – data scientists, systems designers, medical doctors, etc. – to design and reason about probabilistic systems. PPLs are becoming one of the central topics in probabilistic reasoning and programming languages, with increasing interest from industry and academia.

This course will:

1. introduce the core ideas of probabilistic programming – probabilistic inference, language design, and applications – as well as state-of-the-art ideas in the field that make PPLs more reliable, usable, and faster.
2. teach you how to critically analyse state-of-the-art ideas, their strengths and weaknesses, and propose improvements.

At the end of the course, you will be able to:

• read and write probabilistic programs
• understand how probabilistic programming languages are implemented and how they can be extended
• understand the basic and state-of-the-art probabilistic inference procedures (which are applicable beyond probabilistic programs)

Prerequisites

The target audience for this course are Master’s and PhD students. The are no formal prerequisites, but the students are expected to be comfortable with programming and mathematical notation. Basic familiarity with probability theory, artificial intelligence and machine learning is expected (equivalent to a Bachelor’s level course). We will not revise these topics during the lectures, but refresher materials are listed below.

Course Format

The course consists of several learning activities, alongside lectures by the intrstructor, with the points distributed as follows:

• 0%: Paper reviews
• 25%: Presentation
• 65%: Research report
• 10%: Participation

### Course deliverables

Paper reviews. The course will take the format of a research seminar. This means that there will be no textbook and blackboard lectures; instead, we will be reading and discussing state of the art papers in the field. Consequently, you are expected to come prepared for the class by reading one of the papers covered in the lecture. As a preparation for the class, you have to write a review of that paper. The reviews consist of a few questions that help you understand the paper to sufficient extend; they will point out important points, what you should pay attention to, and which part you can avoid. Scientific articles are not always easy to read as they are typically written for people that already know a lot about the field; the review questions help you navigate that. The reviews are not graded, but are essential preparation for the classes. After the lectures, you will receive prototypical answers that help you to track your progress. These questions are in the Miro boards associated with the paper, and you should submit them in Brightspace.

Paper presentation. Only the classes in the first two weeks will be typical lectures; the goal of these lectures is to set all of you at the same starting point. Starting in week 3, you will take the lead role.

In every lecture, we will cover 2 papers on arelated topic. Each paper will be presented by a student taking the course. You choose which paper you present. for every lecture, you are expect to read only one paper out of the 2 we cover; you will learn about the other one from the presentation of your classmate.

For the paper you are responsible, you will understand the paper in details, including searching for additional literature that helps you to understand the paper and why it works, and present it to your colleagues. You are allowed to use any material from the Web for your presentations.

Your objective for the presentation: explain the core idea behind the assigned paper in clearest terms possible. Don’t try to cover everything in the paper, identify important and essential parts. Convey the intuitions before the math.

Lastly, schedule a meeting with me at least 2 days before your presentation for the feedback.

Research report. The largest part of your grade and efforts goes to a research report. In contrast to standard courses, the goal of this course is not for you to just soak up what you have been thought; instead, the goal is to go beyond the materials and think about strengths and weaknesses, and how to overcome them.

You main task in the report is to design a research proposal, without execuyting it. You will have to dig deeper into your topic, studying papers that we will not cover in the class. The report should consist of 5 parts:

1. Description of the topic. Briefly describe your topic and contextualise it within the probabilistic programming field. Describe what the problem is and why is it challenging.
2. Literature study. Starting from a paper you have been assigned to, you should find several related papers (which either propose alternative solutions to the same problem, or use similar techniques for other problems). You should briefly summarise them and outline their strengths and weaknesses compared to the main paper you started with.
3. Relation to other topics in the course. You cannot pass the course by only investing time into your own topic. In the report, you have to describe how it connects to every other topic we have covered (does topic X solve the same problem as your topic? Does X employ different assumptions? Does it use different techniques? …).
4. Experimental test. Each paper we cover either comes with code implementing it or is easy to implement on top of existing probabilistic programming languages. You are expected to play with your method and stress-test it. You can, for example, design probabilistic programs that test the complexity of problems the inference procedures can handle, you can change parts of the techniques to see if they improve their performance or make it worse, etc. If you are re-implementing papers, you don’t have to implement them fully. Start from a simplified versions, explain your motivation for the simplification, and add complexities later on (given the time restraints).
5. Research proposal(s). Describe one or more research projects you think would advance the field, focusing on your topic. The proposal should
   • Clearly describe the problem you are addressing.
   • How would you address it technically, and why does that approach make sense
   • How would you experimentally test it
   • what could go wring with your plan. Note that I do understand that, for many of you, this is the first time encouring open-ended research project. I do not expect you to design entirely new projects no one has taught of before. While you shoudl certainty try to do that, research papers often contain ‘Future work’ sections; you are allowed to start from these suggestions and work them out in details.

Starting in week 5, you will have some time avaialble to request my feedback on your drafts. I will reserve a certain amount of time per week to read your drafts. Each of you will also get a fix amount of time, which you can divide as you wish (e.g., all at once, or distribute it through the weeks). The details will be known once I know how many people will be taking this course.

Class participation. As this is a seminar course, you are expected to actively participate in class. During the lectures, we will be clarifying the details from the papers, analysing their strengths, weaknesses, and evaluation; in other words, exactly what you are supposed to do in your reports. Your participation will be graded in two ways:

• Sending questions in advance. To start up discussion, I ask you to think about the papers in advance and post questions you have about it. What is a good question? Anything that helps you to understand the paper and is not of the form “What is X?” where X is something explained in the paper. Put these question in the Miro board and don’t forget to put your name next to the question.
• Helping others. Our shared goal is to understand the papers, and the field of probabilistic programming itself, together. We will maintain a shared PDF in which you can post and answer questions about papers. Answer the question in the Miro board and, again, dont’ forget to put your name.

See how to do well in a seminar course.

Schedule

You do not have to read every paper from the required literature. You will be divided in groups (group split is on Brightspace) and each group reads only one paper for the class. You will learn about the other paper from the presentation of your colleagues.

Use review questions to focus on the important parts. Of course, you are not discouraged to go into depths of every paper.

Other materials

Additional materials

The course does not have a required textbook, it instead relies on selected papers. Some materials that might help you better understand probabilistic programming:

• Probabilistic Models of Cognition: the book uses probabilistic programs to formulate models of human cognition. The book is a great introduction to using probabilistic programming and contains plenty of example to practice your modelling skills.
• The Design and Implementation of Probabilistic Programming Languages: a high-level book covering the basic inference procedures.
• An Introduction to Probabilistic Programming: a work-in-progress introductory textbook. Covers many topics of the course. Very good introduction to basic inference procedures, implementation strategies, and variational inference.
• Statistical Relational Artificial Intelligence: Logic, Probability, and Computation: a high-level introduction to various paradigms of probabilistic logic programming.
• Foundations of Probabilistic Logic Programming: a deeper introduction to probabilistic logic programming.
• Types and programming languages: good introduction to foundations of programming languages.
• Probabilistic Programming and Bayesian Methods for Hackers: an easy to follow book, great at explaining key ideas intuitively.

Additional topics

Probabilistic programming is rapidly developing field. There are many topics we will not be able to cover in the class. Here is a snapshot of such topics, if you want to know more:

• Other inference tasks: The inference task we look at was the characterisation of the posterior distribution. What if we are interested in other tasks, e.g., maximising the posterior?
  • Bayesian Optimization for Probabilistic Programs
  • Expectation Programming: Adapting Probabilistic Programming Systems to Estimate Expectations Efficiently
• Learning probabilsitic programs: a standard PP pipeline focuses on inference; but what if we don’t know the program?
  • Human-level concept learning through probabilistic program induction
  • Bayesian Synthesis of Probabilistic Programs for Automatic Data Modeling
  • Neural Sketch Learning for Conditional Program Generation
• Semantics of probabilistic programming languages: how to define the meaning of a probabilistic program?
  • Semantics of Probabilistic Programming: A Gentle Introduction
  • Semantics for probabilistic programming: higher-order functions, continuous distributions, and soft constraints
  • Semantics of Probabilistic programs
• Deep generative models: an alternative framework for generative modelling based on neural networks. It has less flexibility than PPL but better learning properties.
  • Deep Generative Modelling
• Analysis of probabilistic programs:
  • Conditional independence by typing
  • Probabilistic Termination: Soundness, Completeness, and Compositionality
• Nested probabilistic programs: probabilistic programs that call probabilistic programs
  • Nesting probabilistic programs
  • Reasoning about reasoning by nested conditioning: Modeling theory of mind with probabilistic programs
  • On the Pitfalls of Nested Monte Carlo
• Stochastic conditioning: in the course, the observations always came in form of the exact value of a variable. What if we want to condition on a distribution?
  • Probabilistic Programs with Stochastic Conditioning
• Deep probabilistic programming: we have touched upon various interaction between probabilistic programming and deep neural networks. Deep probabilistic programming is a paradigm that combines the two, often with murky boundaries from e.g. variational or amortised inference.
  • Neural probabilistic logic programming
  • Deep Probabilistic Programming

Courses at other universities

• Probabilistic programming by Frank Wood at University of British Columbia, Canada
• seminar course by Steven Holtzen at Northeastern, USA
• seminar course by Dan Roy at University of Toronto, Canada, which focuses on theoretical aspects of probabilistic programming
• Deep Probabilistic Programming course by Thomas Hamelryck and Ahmad Salim Al-Sibahi at University of Copenhagen, Denmark
• Probabilistic Programming Languages course by Norman Ramsey at Tufts University, USA
• Probabilistic Programming and Relational Learning bu Guy Van den Broeck at UCLA, USA.
• Probabilistic Programming course by Hongseok Yang at KAIST, South Korea

Practicalities

How to do well in a seminar course

A seminar course is different from a standard course in two ways: (1) “just” learning the topics covered by the materials is not enough; and (2) you are expected to actively participate in the lectures. The active participation includes giving a presentation (see Preparing your presentation) and discussing the materials during the lecture. Actually, the lectures will consist only of these two activities. The exceptions are week 1, which contain two lectures by the instructor, and week 2, in which we discuss papers but no one has to prepare the presentation. Each student will present one paper and is expected to actively contribute in discussions.

If understanding the materials is not enough, what is? In a seminar course, we teach you to think beyond what you have been served. In fact, if you only master the materials, that will get you a mere 6. Instead, you are expected to critically analyse every paper you read. For every paper you read, think about the following questions:

• What is the main idea proposed in the paper?
• Which problem does it solve?
• What are its strengths?
• What are its weaknesses?
• Which parts of the paper were/and perhaps still are confusing to you?
• Do you have an impression that the idea works just because the authors made some particular choices?
• What are the implicit assumption the authors are making?
• Do you see multiple options to solve a subproblem X and are wondering why the authors chose
• Is the idea properly evaluated? If not, what is missing? How would you do it?
• Are the arguments for the idea convincing?
• How would you extend the work?

During the discussion part of the lectures, we will discuss these questions and compare your observation. Not all questions are applicable to every paper, so don’t worry if you cannot relate some of them to some papers. The quality of the discussion will therefore depend on you. Importantly, you have to come prepared for each lecture.

It is important that you actively participate in the discussion. As a rule of thumb, the more you talk, the higher your grade will be in the end. However, be constructive and avoid saying something just to say something. I don’t want you to be quiet the entire quarter, but that does not mean that you have to say something in every discussion. Sometimes you would have something brief to say, sometimes you would have some deeper insight. That is fine. As a rule of thumb, I expect you to say something at least every few lectures. Don’t be upset if I respond to you or even correct you. For most of you, this is the first time you are participating in a seminar. It is expected that you need some guidance. Moreover, I tend to jump in when you mention something interesting without (perhaps) realising it.

An important difference between a regular course and a seminar is that you will not be penalised for not understanding something. That does not mean that this is a free-pass course, but rather that it is ok to build up towards an understnading during the course. You will be reading research papers, often in depth.
In contrast to textbooks, research papers are concise and expect certain knowledge from a reader. This has two consequence. First, it is likely that you will not know some of the necessary concepts and would have to fill the gap to understand the paper. You should go through that process. Second, scientists are often not great writers – they might explain simple concepts very confusingly because they have different audience in mind (peers, not students). For any of these reasons, you might not understand a part of the paper. This is normal and you should not hide it – confusing parts of the paper are great starters for discussions!

Here are a few situations that are acceptable in a seminar course, but perhaps not in a regular course:

1. You misunderstand something in a paper and provide a wrong answer in a review. If you realise that during the lecture, you can correct you answer after the lecture. That is fine.
2. You misunderstand something that leads you to an interesting idea for an extension. This is perfectly fine if your reasoning taking the misunderstanding as a fact is correct and interesting.
3. You just can’t understand what the paper is trying to say. Voice it in the lecture! Try to phrase it as a detailed question; point out as specifically as possible where you lost it

Preparing your presentation

You will have to present one paper to the group. Student presentation start in week 3.

Your presentation should take 15 mins. This is an estimate, some papers might require less while other more time. If you think you have to go beyond this limit, discuss it with me.

Approach your presentation as a short lecture. You goal is to explain the topic to your colleagues so that they learn from the lecture; only half of the student in the class will have read the paper. Think about the right way to explain the topic; this might not be the way it was explained in the paper. For instance, if you found that you had to look a lot of other concepts, explain them before the topic you are covering. Use illustrations and animations, you can find lots of them online (if not for the exact topic, then for something very related that could help you explain the intuition). Provide a working example, especially if such an example is not present in the original paper.

Feel free to use “non-academic” materials for your preparations: blogs, videos, informal notes… I don’t care what you use as long as you understand the topic. Importantly, if you encounter materials that you found more useful than the one I proposed, share them with me.

Your main focus in the presentation should be to convey the idea/concept to your colleagues. This usually means that you have to think about not only how to convey the idea effectively but also which parts of the paper to include or skip. There will be no correlation between the amount of content squeezed in a lecture and a grade. Often it makes sense to skip something to optimise for clarity.

Important: schedule a meeting with me at least 2 days before your presentation; this is to ensure that your presentation is of sufficient quality to provide a valuable learning experience to your colleagues.

Project - implementation

Project - report

Course feedback

This is the first edition of the course. As you can imagine, this is kind of a beta version of the course. To make sure the course offers the best learning experience to you, I would appreciate if you provide feedback throughout the course. What works for you? What doesn’t? Do you have an idea how to improve something?

You can submit the feedback anonymously HERE.