Prerequisites

• DS-GA-1001: Intro to Data Science or its equivalent
• DS-GA-1002: Statistical and Mathematical Methods or its equivalent
• Solid mathematical background, equivalent to a 1-semester undergraduate course in each of the following: linear algebra, multivariate calculus (primarily differential calculus), probability theory, and statistics. (The coverage in the 2015 version of DS-GA 1002, linked above, is sufficient.)
• Python programming required for most homework assignments.
• Recommended: Computer science background up to a "data structures and algorithms" course
• Recommended: At least one advanced, proof-based mathematics course
• Some prerequisites may be waived with permission of the instructor
• You can also self-assess your preparation by filling out the Prerequisite Questionnaire

Grading

Homework (40%) + Midterm Exam (20%) + Final Exam (20%) + Project (20%)

Many homework assignments will have problems designated as “optional”. At the end of the semester, strong performance on these problems may lift the final course grade by up to half a letter grade (e.g. B+ to A- or A- to A), especially for borderline grades. You should view the optional problems primarily as a way to engage with more material, if you have the time. Along with the performance on optional problems, we will also consider significant contributions to Piazza and in-class discussions for boosting a borderline grade.

Important Dates

• Midterm Exam (100 min) Tuesday, March 6th, 5:20–7pm.
• Final Exam (100 min) Tuesday, May 15th, 6-7:50pm (confirmed).
• See Assignments section for homework-related deadlines.
• See Project section for project-related deadlines.

Textbooks

The Elements of Statistical Learning (Hastie, Friedman, and Tibshirani)

This will be our main textbook for L1 and L2 regularization, trees, bagging, random forests, and boosting. It's written by three statisticians who invented many of the techniques discussed. There's an easier version of this book that covers many of the same topics, described below. (Available for free as a PDF.)

An Introduction to Statistical Learning (James, Witten, Hastie, and Tibshirani)

This book is written by two of the same authors as The Elements of Statistical Learning. It's much less intense mathematically, and it's good for a lighter introduction to the topics. (Available for free as a PDF.)

Understanding Machine Learning: From Theory to Algorithms (Shalev-Shwartz and Ben-David)

Last year this was our primary reference for kernel methods and multiclass classification, and we may use it even more this year. Covers a lot of theory that we don't go into, but it would be a good supplemental resource for a more theoretical course, such as Mohri's Foundations of Machine Learning course. (Available for free as a PDF.)

Pattern Recognition and Machine Learning (Christopher Bishop)

Our primary reference for probabilistic methods, including bayesian regression, latent variable models, and the EM algorithm. It's highly recommended, but unfortunately not free online.

Bayesian Reasoning and Machine Learning (David Barber)

A very nice resource for our topics in probabilistic modeling, and a possible substitute for the Bishop book. Would serve as a good supplemental reference for a more advanced course in probabilistic modeling, such as DS-GA 1005: Inference and Representation (Available for free as a PDF.)

Other tutorials and references

• Carlos Fernandez-Granda's lecture notes provide a comprehensive review of the prerequisite material in linear algebra, probability, statistics, and optimization.
• Brian Dalessandro's iPython notebooks from DS-GA-1001: Intro to Data Science
• The Matrix Cookbook has lots of facts and identities about matrices and certain probability distributions.
• Stanford CS229: "Review of Probability Theory"
• Stanford CS229: "Linear Algebra Review and Reference"
• Math for Machine Learning by Hal Daumé III

Software

• NumPy is "the fundamental package for scientific computing with Python." Our homework assignments will use NumPy arrays extensively.
• scikit-learn is a comprehensive machine learning toolkit for Python. We won't use this for most of the homework assignments, since we'll be coding things from scratch. However, you may want to run the scikit-learn version of the algorithms to check that your own outputs are correct. Most people will use it for their final projects. Also, studying the source code can be a good learning experience.

Overview

The project is your opportunity for in-depth engagement with a data science problem. In job interviews, it's often your course projects that you end up discussing, so it has some importance even beyond this class. That said, it's better to pick a project that you will be able to go deep with (in terms of trying different methods, feature engineering, error analysis, etc.), than choosing a very ambitious project that requires so much setup that you will only have time to try one or two approaches.

Key Dates

• Feb 26 (Mon 10pm): Deadline for choosing project groups
• March 2 (Fri 10pm): Email short description (few sentences) of project idea(s) to adviser, along with personal intros
• March 7 (Wed, Lab time): First meeting with advisers. Each group will give a 5-minute presentation of their project idea to their assigned project adviser, followed by brief discussion.
• March 22 (Thurs 10pm): Project Proposals Due
• Apr 18th (Wed, Lab time): Second meeting with advisers
• May 2nd (Wed, Lab time): Third meeting with advisers
• May 17th or May 18th, depending on adviser: Final Project Reports Due to Advisers

Guidelines for Project Topics

A good project for this class is one that's a real "problem", in the sense that you have something you want to accomplish, and it's not necessarily clear from the beginning the best approach. The techiques used should be relevant to our class, so most likely you will be building a prediction system. A probabilistic model would also be acceptable, though we will not be covering these topics until later in the semester.

To be clear, the following approaches would be less than ideal:

1. Finding an interesting ML algorithm, implementing it, and seeing how it works on some data. This is not appropriate because I want your choice of methods to be driven by the problem you are trying to solve, and not the other way around.
2. Choosing a well-known problem (e.g. MNIST digit classification or the Netflix problem) and trying out some of our ML methods on it. This is better than the previous example, but with a very well-established dataset, a lot of the most important and challenging parts of real-world data science are left out, including defining the problem, defining the success metric, and finding the right way to encode the data.
3. Choosing a problem related to predicting stock prices. Historically, these projects are the most troubled. Interestingly, our project advisers who have worked in this field are the ones who advise against this most strongly.

Project proposal guidelines

The project proposal should be roughly 2 pages, though it can be longer if you want to include figures or sample data that will be helpful to your presentation. Your proposal should do the following:

1. Clearly explain the high-level problem you are trying to solve (e.g. predict movie ratings, predict the outcome of a court case, ...).
2. Identify the data set or data sets that you will be using. You should give a clear description of the characteristics of the data (how many examples, what kinds of features do we have for each example, are there issues with missing data or bad data, etc.).
3. How will you evaluate performance? In certain settings, you may want to try a few different performance measures.
4. Identify a few "baseline algorithms". These are simple algorithms for solving the problem, such as always predicting the majority class for a classification problem, using a small set of decision rules designed by hand, or using a ridge regression model on a basic feature set. Ideally, you will be able to report the performance of a couple baseline algorithms in your proposal. The goal will be to beat the baseline, so if the baseline is already quite high, you will have a challenge.
5. What methods do you plan to try to solve your problem, along with a rough timeline. Methods include data preprocessing, feature generation, and the ML models you'll be trying. Once you start your investigation, it's best to use an iterative approach, where the method you choose next is based on an understanding of the results of the previous step.

Project writeup guidelines

The main objective of the project writeup is to explain what you did in a self-contained report. No strict guidelines on the format of the report, but the goal is to make it something you'd be proud to share with a potential employer. Some of the content will resemble your project proposals. Make sure to:

1. Clearly explain the high-level problem you are trying to solve (e.g. predict movie ratings, predict the outcome of a court ase, ...).
2. Identify the data set or data sets that you will be using. You should give a clear description of the characteristics of the ata (how many examples, what kinds of features do we have for each example, are there issues with missing data or bad data, etc.).
3. How did you evaluate performance and measure success?
4. What did you use for features, and explain any feature engineering that you did.
5. What did you do to attempt to improve performance over your baseline algorithms (e.g. error analysis, new features, new parameter tuning,...)
6. What challenges did you encounter? What insights into your problem did you get?
7. What would be good next steps to take if you were to continue this work?
8. If you got ideas from other sources, please cite them.

Some Previous Projects

• Social attitudes cannot be predicted from federal court decisions and judge characteristics.
• What Matters: Agreement between U.S. Courts of Appeals Judges
• Probability Estimation for Online Education System
• Using Predictive Models to Determine Judge Bias in Asylum Refugee Court Cases
• E-Commerce Recommender Modeling: A Case Study of Ponpare Coupon Purchase Prediction.
• Predicting Job Markets in India using heavy tailed regression
• Resampling the Spectral Time Series of a SN in the Time Dimension

Some Public Data Sets (just to get you thinking)

• Datasets on Amazon's AWS cloud
• Yelp Dataset Challenge
• NYC Open Data
• Data.gov
• UN Data
• Kaggle
• Quandl financial, economic, social datasets
• Rating data sets from MovieLens
• Congress voting records
• Quota's meta list of datasets

Instructor

David Rosenberg

[email protected]

David is a data scientist in the office of the CTO at Bloomberg L.P. Formerly he was Chief Scientist of YP Mobile Labs at YP.

Section Leader

Ben Jakubowski

Ben is a 2017 NYU Data Science MS graduate. He currently works as a data scientist for the University of Chicago's Crime Lab New York (CLNY), where his portfolio includes several prediction problems that arise in criminal justice and social policy.