Principles of Machine Learning (STATS 302)

Fall 2021-2022 / Session 1 (7 weeks, 35 + 14 hours)

Course Period: August 23 - October 14, 2021

  • Lectures: Monday / Tuesday / Wednesday / Thursday @ 13:15-14:30 (Classroom: TBA + Zoom)
  • Recitations / Labs: Tuesday / Thursday @ 14:45-15:45 (Classroom: TBA + Zoom)
Instructor: Mustafa MISIR (Office: WDR 2106), mustafa.misir [at] dukekunshan.edu.cn
Lab Instructor: Xue Chen (Office: TBA), xue.chen240 [at] dukekunshan.edu.cn
Teaching Assistant: Cirun Zhang (Office: TBA), cirun.zhang [at] duke.edu

Machine Learning (ML) is a popular field with interdisciplinary characteristics relating to various fields including Computer Science, Mathematics and Statistics. ML aims at learning without being explicitly programmed, through data and experience. The target applications are the complex tasks which are challenging, impractical or unrealistic to program. ML can be used to address those sophisticated activities that humans or animals can routinely do such as speech recognition, image understanding and driving. The other functions to learn that ML concentrates on, are concerned with the ones requiring capabilities beyond human capacities, in terms of speed and memory.

This course will identify the major ML problems while introducing the fundamental ML algorithms to solve them. To be specific, the topics to be covered are maximum likelihood estimation, linear discriminant analysis, logistic regression, support vector machines, decision trees, linear regression, Bayesian inference, unsupervised learning, and semi-supervised learning. The course will require basic programming skills (Python) and introductory level knowledge on probability and statistics besides benefiting from certain linear algebra concepts.

By the end of this course, you will be able to:
  1. specify a given learning task as a ML problem
  2. determine the appropriate ML algorithms for addressing an ML problem
  3. manipulate the given data concerned with a learning task so that the preferred ML algorithm can be effectively applied
  4. construct generalizable ML models that can address a given ML problem of unseen data
  5. analyze the performance of the ML algorithms while revealing their shortcomings referring to the nature of the data
  6. build complete ML workflows in Python together with the relevant libraries / frameworks / tools besides effectively communicating your methods and results using Jupyter notebooks
+++ Follow Sakai for announcements and discussions

Pre-requisites

  • COMPSCI 201: Introduction to Programming and Data Structures
  • STATS 210: Probability, Random Variables and Stochastic Processes

Co/Pre-requisites

  • MATH 304: Numerical Analysis and Optimization
  • MATH 305: Advanced Linear Algebra
The following chart shows how STATS 302 fits to the DKU curriculum, where the abbreviations indicate the course types, i.e. D: Divisional, DF: Divisional Foundation, ID: Interdisciplinary and E: Elective. Refer to the DKU Undergraduate Bulletin for more details.




There is no official textbook for this course. Still, the following books can be used as references.

Reference Books


Lecture Notes / Slides

  • Week 0 (Reviews)

  • Week 1   [23/08 - 26/08]   (Keywords: History, Terminology and Basics; Supervised Learning; Regression Problem; Gradient Descent)

  • Week 2   [30/08 - 02/09]   (Keywords: Supervised Learning; Classification Problem; Regression Problem)
    • Logistic Regression
      • External Lecture Notes: CS229: Machine Learning (Andrew Ng, Stanford U) [Part II] - Logistic Regression
      • External Video: CS229: Machine Learning (Andrew Ng, Stanford U) - Logistic Regression
    • k-Nearest Neighbors (kNN)
      • External Lecture Notes: CS4780: Machine Learning for Intelligent Systems (Kilian Weinberger, Cornell U) - kNN
      • External Video: CS4780: Machine Learning for Intelligent Systems (Kilian Weinberger, Cornell U) - kNN
    • Naive Bayes
      • Review: Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville, Chapter 3 - Probability
      • Article: Hand, D.J. and Yu, K., 2001. Idiot's Bayes-not so stupid after all?. International statistical review, 69(3), pp.385-398
      • External Lecture Notes: CS4780: Machine Learning for Intelligent Systems (Kilian Weinberger, Cornell U) - Bayes Classifier and Naive Bayes
      • External Video: CS4780: Machine Learning for Intelligent Systems (Kilian Weinberger, Cornell U) - Naive Bayes
    • Recitation / Lab: Pandas, Logistic Regression, kNN, Naive Bayes
    • Homework 1: TBA

  • Week 3   [06/09 - 09/09]   (Keywords: Supervised Learning; Feed-forward Neural Networks; Regression Problem; Classification Problem)

  • Week 4   [13/09 - 16/09]   (Keywords: Supervised Learning; Regression Problem; Classification Problem)
    • Decision Trees
      • Book Chapter: Introduction to Statistical Learning by Gareth James, Daniela Witten, Trevor Hastie, Rob Tibshirani (2014), Chapter 8 - Tree-based Methods
    • Ensembles: Bagging and Boosting
      • Book Chapter: Pattern Recognition and Machine Learning by Christopher Bishop (2006), Chapter 14 - Combining Models
    • Support Vector Machines (SVM)
      • External Lecture Notes: CS229: Machine Learning (Andrew Ng, Stanford U) [Part VI] - SVM
      • External Video: CS229: Machine Learning (Andrew Ng, Stanford U) - SVM
      • External Lecture Notes: CS4780: Machine Learning for Intelligent Systems (Kilian Weinberger, Cornell U) - SVM
      • External Video: CS4780: Machine Learning for Intelligent Systems (Kilian Weinberger, Cornell U) - SVM
    • Recitation / Lab: Decision Trees, Ensembles, SVM
    • Homework 3: TBA
    • MIDTERM (Date: TBA)

  • Week 5   [17/09 - 18/09; 22/09 - 23/09]   (Keywords: Un/-supervised Learning; Dimensionality Reduction; Clustering (+ Evaluation and Analysis))
    • Dimensionality Reduction
    • Principal Component Analysis (PCA)
      • External Lecture Notes: CS229: Machine Learning (Andrew Ng, Stanford U) - PCA
      • External Lecture Notes: CS4786: Machine Learning for Data Science (Karthik Sridharan, Cornell U) - PCA
    • Linear Discriminant Analysis (LDA)
      • Book Chapter: Introduction to Statistical Learning by Gareth James, Daniela Witten, Trevor Hastie, Rob Tibshirani (2014), Chapter 4.4 - LDA
      • Book Chapter: Probabilistic Machine Learning: An Introduction by Kevin P. Murphy (2021), Chapter 9 - LDA
    • k-means Clustering
    • Recitation / Lab: PCA, LDA, k-means
    • Homework 4: TBA

  • Week 6   [27/09 - 30/09]   (Keywords: Graphical Models)
    • Bayesian Networks
    • Markov Random Fields (MRF)
      • Book Chapter: Pattern Recognition and Machine Learning by Christopher Bishop (2006), Chapter 8.3 - Bayesian Networks
      • External Lecture Notes: CS228 - Probabilistic Graphical Models (Stefano Ermon, Stanford U) - MRF
    • Factor Graphs
      • Book Chapter: Pattern Recognition and Machine Learning by Christopher Bishop (2006), Chapter 8.4.3 - Factor Graphs
    • Recitation / Lab: Graphical Models
    • Homework 5: TBA

  • Week 7   [11/10 - 14/10]   (Keywords: Sequential Decision Making; Generative Models; Semi-supervised Learning)
    • Sequential Data
      • Book Chapter: Pattern Recognition and Machine Learning by Christopher Bishop (2006), Chapter 13 - Sequential Data
    • Hidden Markov Models (HMMs)
      • Book Chapter: Pattern Recognition and Machine Learning by Christopher Bishop (2006), Chapter 13.2 - HMMs
    • Generative Models
    • Semi-supervised Learning
    • Recitation / Lab: HMMs
    • Homework 6: TBA
    • Project Presentations (Date: TBA)
    • FINAL (Date: TBA)

Grading

  • Homework: 20%
    • Mathematical, Conceptual, or Programming related
    • Submit on Sakai; 6 in total, the lowest score is dropped
  • Weekly Journal: 10%
    • Each week, write a page or so about what you have learned
    • Submit on Sakai; 2 points off for each missing journal, capped at 10
  • Midterm: 20%
  • Final: 30%
  • Project: 20%
    • Report Rubric (TBA)
    • Presentation Rubric (TBA)

Reference Courses


Sample Projects


Other Books

Quick / Easy Reads:
Python Programming:
Python Programming for Data Science / Analytics:
Data Visualization:

Other Materials / Resources