Course Information

Short Intro

The objective of this course is to provide a complete introduction to deep machine learning. General artificial intelligence and machine learning techniques will be quickly reviewed in a historical connections fashion, but the focus of this course is machine learning. So the main theme here is to understand modern techniques that specifically handle deep neural network: how to design it, how to train it and how to evaluate it on real problems.

Description

The course aims at providing an overview of existing approaches and methods, at teaching how to design and train a deep neural network for a given task, and (depending on the audience,) at providing the theoretical basis to go beyond the topics directly seen in the course.

The course will touch on the following topics:

• What is deep learning, and how is it derived from general artificial intelligence and machine learning.
• Tensors, multi-layer Perceptrons, gradient descent, back-propagation.
• Generalized networks, automatic differentiation, batch processing, convolutional networks.
• Initialization, optimization, and regularization.
• Drop-out, BatchNorm, ResNets.
• Deep models for computer vision.
• Deep models for sequence processing.
• Analysis and visualization of deep models, geometric explanation of deep learning.
• Auto-encoders, embeddings, and generative models.

Concepts will be illustrated with examples in the PyTorch framework.

Keywords: deep learning, neural networks, python, PyTorch

Course group

Previous offers: [2022]

Closely related: [Artificial Intelligence], [Machine Learning], [Natural Language Processing]

Prerequisites

We expect you to have the following skills before taking this course:

本课程建议掌握如下专业知识：

Required：

• Linear algebra (vector, matrix computation, Euclidean spaces).
• Differential calculus (Jacobian, Hessian, chain rule).
• Probabilities and statistics (common distributions, law of large numbers, conditional probabilities, Bayes)
• Python programming.

Recommended：

• Numerical optimization (notion of minima, gradient descent).
• Algorithm analysis (computational costs).
• Visual computing (image processing).

Teaching Plan

This course is organized into a 12-week session (4 hours per week). The main contents are listed below:

本课程计划安排12周内容，主要讲授内容分别如下：

1. Introduction 导言：什么是深度学习
2. Preliminaries 预备知识
3. Linear Neural Networks 线性神经网络
4. Multilayer Perceptrons 多层感知机
5. Deep Learning Computation 深度学习计算
6. Convolutional Neural Networks 卷积神经网络
7. Modern Convolutional Neural Networks 现代卷积神经网络
8. Computer Vision: Applications I 计算机视觉应用I
9. Computer Vision: Applications II 计算机视觉应用II
10. Recurrent Neural Networks 循环神经网络
11. Modern Recurrent Neural Networks 现代循环神经网络
12. Natural Language Processing: Applications I 自然语言处理应用I

For classes that also take Natural Language Processing 2023F, NLP part will be changed as follows:

对于同时学修《自然语言处理》的班级，课程关于NLP的部分将调整为如下内容：

10. Attention Mechanisms 注意力机制
11. Transformers and Pretraining Transformers及预训练
12. Natural Language Processing: Applications II 自然语言处理应用II

Schedule

Multiple offerings; Multiple locations.

Methodology

Problem-solving oriented, equal emphasis on lecture and practice.

以解决实际问题为导向，教学与实践并重。

Each lecture is roughly organized into 3 progressive units:

1. Core Concepts 核心概念: provides elementary knowledge of the topic
2. Advanced Discussion 进阶讨论: provides in-depth understanding, mathematical formulations
3. Practical Skills 实践技巧: provides problem-solving skills through hands-on programming training

Evaluation

• Attendance & participation: 20%
• Understanding of topics: 30%
• Final exam/project: 50%

Textbook

Not mandatory but recommended:

以《动手学深度学习》为主。其他辅助教材：

• Chollet, Deep Learning with Python
• Courville et al., Deep Learning
• 周志华, 机器学习
• Friedman et al., The Elements of Statistical Learning

Resource

• 动手学深度学习¹
• HUNG-YI LEE (李宏毅), MACHINE LEARNING 2022 SPRING²