李宏毅——一天搞懂深度学习PPT学习笔记

李宏毅一天搞懂机器学习PPT，SildeShare链接：https://www.slideshare.net/tw_dsconf/ss-62245351?qid=108adce3-2c3d-4758-a830-95d0a57e46bc&v=&b=&from_search=3
也可以在csdn下载中下载（资源附学习笔记全文）：https://download.csdn.net/download/wozaipermanent/11998637

1 Introduction of Deep Learning

1.1 Three Steps for Deep Learning

• Step1: define a set of function (Neural Network)
• Step2: goodness of function
• Step3: pick the best function

1.2 Step1: Neural Network

1.2.1 Fully Connect Feedforward Network

1.2.2 Output Layer(Option)

• Softmax(归一化指数函数)：它能将一个含任意实数的k维向量Z“压缩”到另一个k维向量$\sigma (Z)$中，使得每一个元素的范围都在(0, 1)之间，并且所有元素的和为1。

1.2.3 Example Application

• Handwriting Digit Recognition
  

1.3 Step2: Goodness of Function

1.3.1 Learning Target

1.3.2 Loss

• Total Loss:

1.4 Step3: Pick the Best Function

1.4.1 Gradient Descent

• RBM(Restricted Boltzmann Machine): 受限玻尔兹曼机，这部分可以参考链接：https://zhuanlan.zhihu.com/p/22794772

• Then Compute $\partial L/\partial w$ , if Negative then Increase w; elif Positive then decrease w

• $\eta$ is called “learning rate”

Gradient Descent Diagram:

• Randomly pick a starting point

1.4.2 Gradient Descent Difficulty

• Backpropagation(反向传播算法)：an efficient way to compute $\partial L/\partial w$ , link below:
  • http://speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2015_2/Lecture/DNN%20backprop.ecm.mp4/index.html
  • https://www.jianshu.com/p/2e02bc6384a8

1.5 Deep is Better

1.5.1 Universality Theorem

• Any continuous function $f:R^N\to R^M$ can be realized by a network with one hidden layer (given enough hidden neurons). Ref: http://neuralnetworksanddeeplearning.com/chap4.html

1.5.2 Thin + Tall is Better

• Neural network consists of neurons

• A hidden layer network can represent any continuous function

• Using multiple layers of neurons to represent some functions are much simper

• Less parameters, less data

1.5.3 Modularization

1.6 Toolkit

1.6.1 Keras

• Documentation: https://keras.io or https://morvanzhou.github.io/tutorials/machine-learning/keras/

1.6.2 Example of Handwriting Digit Recognition

Step1: define a set of function

Step2: goodness of function

Step3: pick the best function

Testing

score = model.evaluate(x_test, y_test)
print('Total loss on Testing Set: ', score[0])
print('Accuracy of Testing Set: ', score[1])
result = model.predict(x_test)

1.6.3 GPU to Speeding Training

• Way1

  THEANO_FLAGGS=device=gpu0 python YourCode.py
  

• Way2

  import os
  os.environ["THEANO_FLAGS"] = "device=gpu0"
  

2 Tips for Training Deep Neural Network

2.1 Good Results on Training Data

2.1.1 Choosing Proper Loss

2.1.2 Mini-Batch

2.1.3 New Activation Function

Vanishing Gradient Problem

ReLU

model.add(Activation('sigmoid'))
model.add(Activation('relu'))

ReLU - variant

2.1.4 Adaptive Learning Rate

Learning Rates

• If learning rate is too large, total loss may not decrease after each update
• If learning rate is too small, training would be too slow

Adagrad

Notes:

• Learning rate is smaller and smaller for all parameters
• Smaller derivatives, larger learning rate, and vice versa

2.1.5 Momentum

• Adam: RMSProp (Advanced Adagrad) + Momentum. Adam (Adaptive Moment Estimation)本质上是带有动量项的RMSprop，它利用梯度的一阶矩估计和二阶矩估计动态调整每个参数的学习率。Adam的优点主要在于经过偏置校正后，每一次迭代学习率都有个确定范围，使得参数比较平稳。
  

2.2 Good Results on Testing Data

2.2.1 Early Stopping

Why Overfitting

• Learning target is defined by the training data.
• The parameters achieving the learning target do not necessary have good results on the testing data.

Early Stopping

2.2.2 Weight Decay

Weight decay is one kind of regularization.

• Our brain prunes out the useless link between neurons.
• Doing the same thing to machine’s brain imporves the performance.
  

2.2.3 Dropout

Training

• Each time before updating the parameters

  • Each neuron has p% to dropout
    • The structure of the network is changed.
  • Using the new network for training

• For each mini-batch, we resample the dropout neurons

Testing

Dropout - Intuitive Reason

Drop is a Kind of Ensemble

Try It

2.2.4 Network Structure

e.g. CNN is another good example.

3 Variants of Neural Network

3.1 Convolutional Neural Network (CNN)

3.1.1 Why CNN for Image

• When processing image, the first layer of fully connected network would be very large.
• Some patterns are much smaller than the whole image. A neuron does not have to see the whole image to discover the pattern.
• The same patterns appear in different regions.
  
• Subsampling the pixels will not change the object, so we can subsample the pixels to make image smaller.

3.1.2 Three Steps

Step1: Convolutional Neural Network

Convolution

Max Pooling

• Smaller than the original image.
• The number of the channel is the number of filters.

Flatten

Summary

Step2: goodness of function & Step3: pick the best function

3.2 Recurrent Neural Network (RNN)

Step1: Recurrent Neural Network

LSTM

Step2: goodness of function

Step3 : pick the best function

4 Next Wave

4.1 Supervised Learning

4.1.1 Ultra Deep Network

4.1.2 Attention Model

4.2 Reinforcement Learning

4.2.1 Scenario of Reinforcement Learning

4.2.2 Supervised v.s. Reinforcement

4.2.3 Difficulties of Reinforcement Learning

• It may be better to sacrifice immediate reward to gain more long-term reward.
• Agent’s actions affect the subsequent data it receives.

4.3 Unsupervised Learning

4.3.1 Image: Realizing what the World Looks Like

4.3.2 Text: Understanding the Meaning of Words

• Machine learn the meaning of words from reading a lot of documents without supervision
• A word can be understood by its context

4.3.3 Audio: Learning Human Language Without Supervision

• Audio segment corresponding to an unknown word (Fixed-length vector)
• The audio segments correspondsing to words with similar pronunciations are close to each other.