Machine Learning

Classical machine learning from a computing science perspective, with most algorithms implemented from scratch in NumPy on invented, seeded data: the learning problem and empirical risk minimization, the bias–variance decomposition, PAC bounds and VC dimension, linear regression with ridge and lasso regularization, the perceptron, logistic and softmax regression, support vector machines and kernels, nearest neighbors and decision trees, bagging, random forests, AdaBoost and gradient boosting, naive Bayes and discriminant analysis, cross-validation, ROC and precision–recall analysis, calibration and model comparison, feature engineering and leakage-safe pipelines, k-means, Gaussian mixtures and EM, hierarchical and density-based clustering, PCA, a first neural network trained by backpropagation, and a capstone project with subgroup evaluation and US rules on automated decisions.

CMPUT-367 Year 3 3 credits 12 lessons · 7 h 1 min

Offered by HSM University · Computing Science

This is the American version of this course. See the Canadian version.

What you'll learn

Formulate learning problems in terms of task, performance measure and experience, and describe the supervised learning setup of distribution, sample, hypothesis class, loss and risk.
Derive the bias–variance decomposition for squared loss, explain underfitting and overfitting, and apply finite-class PAC bounds and VC dimension to reason about sample size.
Fit linear regression by the normal equations and by gradient descent, interpret it as maximum likelihood, and apply ridge and lasso regularization with validation-chosen strength.
Implement the perceptron and prove its mistake bound, and train logistic and softmax regression by gradient methods, interpreting weights as changes in log odds.
Explain maximum-margin classification, the soft margin and hinge loss, support vectors and the dual, and use kernels, Mercer's condition and kernel ridge regression.
Apply k-nearest neighbors and decision trees, choose k, depth and pruning by validation, compute impurity measures, and explain the curse of dimensionality.
Build bagged ensembles, random forests, AdaBoost and gradient boosting, explain why averaging and boosting reduce error, and interpret permutation feature importance.
Construct naive Bayes and Gaussian discriminant classifiers, contrast generative and discriminative learning, and make cost-sensitive decisions from posterior probabilities.
Evaluate models with appropriate cross-validation schemes, confusion-matrix measures, ROC and precision–recall curves, calibration and paired statistical comparisons.
Engineer features, build leakage-safe preprocessing pipelines, handle missing values and class imbalance, and tune hyperparameters reproducibly.
Cluster data with k-means, Gaussian mixtures fitted by EM, hierarchical and density-based methods, and assess clusterings critically.
Reduce dimension with PCA, train a small neural network by backpropagation, and carry out an end-to-end project that includes subgroup evaluation and awareness of Canadian and US rules.

Course content

12 lessons · 7 h 1 min total · final exam

1 The learning problem 39 min · quiz
2 Generalization, bias–variance and learning theory 35 min · quiz
3 Linear regression and regularization 35 min · quiz
4 Linear classifiers: the perceptron and logistic regression 34 min · quiz
5 Support vector machines and kernels 35 min · quiz
6 Nearest neighbors and decision trees 34 min · quiz
7 Ensembles: bagging, random forests and boosting 35 min · quiz
8 Probabilistic classifiers: naive Bayes and discriminant analysis 33 min · quiz
9 Evaluating and comparing models 36 min · quiz
10 Feature engineering, pipelines and practical machine learning 34 min · quiz
11 Unsupervised learning: clustering and mixture models 34 min · quiz
12 Dimensionality reduction, neural networks and a capstone project 37 min · quiz
E Final exam 20 questions

Requirements

Recommended background. You can start any course at any time.

Description

Part of the Science stream, typically taken in year 3.