Fast.ai v3 2019课程中文版笔记

[Daniel (深度碎片)]

Jeremy 在课程中是如何讲解学习率的

lesson 2

如何从图中读取最优学习率区间?

@Daniel

如何从图中读取最优学习率区间

27:36-29:39
视频节点

• 怎样的下坡才是真正有意义的最优学习率区间？
  • “bumpy"起伏不平的不好，”平滑坡陡”更好？
  • 主要靠实验来积累感官经验，构造良好直觉
• 怎么选择的 (3e-5, 3e-4)?
  • 确定好了3e-5后，通常选择1e-4或3e-4
  • 依旧是依靠实验和经验累积

[quote=“Daniel, post:44, topic:39325”]

why do we need learning rate at all?

@Daniel

why do we need learning rate at all?

101:10-105:47
why do we need learning rate at all

• derivative tells us direction and how much
• but it may not best reduce the loss
• we need learning rate to help get loss down appropriately

Lesson 3

How to pick learning rate carefully?

@Daniel

How to pick learning rate carefully?

48:37-50:27
How to pick learning rate carefully

• How to read the lr for fine-tuning?
• How to read the lr for full training?
• What is discriminative learning rate?

Lesson 5

What is fit-one-cycle

@Daniel

What is fit-one-cycle

120:00-123:30
What is fit-one-cycle

• fastai takes care of the optimization details for us
• what does fit-one-cycle do?
• make learning rate start low
• as we know very little of the world
• go up about half of the time
• knows the world better, direction is correct
• then go down about half of the time
• finetune to get closer to convergence
• right : momentum graph
• when steps are small and momentum is accumulating, suggest we can just make big steps
• when steps are big, momentum is small, suggesting we change flexibly
• when steps are getting smaller, the momentum is increasing, we can make big steps again
• make it super fast convergence
• an inspiring story