Lesson 7 Resnet Mnist
Jeremy的视频解读1:59-11:02
MNIST CNN
三行魔法代码
%reload_ext autoreload
%autoreload 2
%matplotlib inline
所需library
from fastai.vision import *
Data
下载完整MNIST
下载完整MNIST
path = untar_data(URLs.MNIST)
path.ls()
[PosixPath('/home/ubuntu/.fastai/data/mnist_png/training'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/testing')]
利用data block api 分布实现DataBunch构建
利用data block api 分布实现DataBunch构建
il = ImageList.from_folder(path, # recursively 提取文件夹中图片和label信息
convert_mode='L' # 图片按照黑白取色)
ImageList 的内容可以通过items查看
ImageList 的内容可以通过items查看
il.items[0]
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/4/44688.png')
将fastai的图片颜色设置为二元
将fastai的图片颜色设置为二元
defaults.cmap='binary' # 通常是RGB
展示ImageList的内容
展示ImageList的内容
il
ImageList (70000 items)
[Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28)]...
Path: /home/ubuntu/.fastai/data/mnist_png
展示ImageList的第一个内容(图片)
展示ImageList的第一个内容(图片)
il[0].show()
如何查分数据集成训练,验证和测试集
如何查分数据集成训练,验证和测试集
sd = il.split_by_folder(train='training', valid='testing')
# 这里的'testing' 文件夹里是含有标注的数据(也就是验证集),不是真正意义上的无标注测试集
查看分割后的ImageList
查看分割后的ImageList
sd
ItemLists;
Train: ImageList (60000 items)
[Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28)]...
Path: /home/ubuntu/.fastai/data/mnist_png;
Valid: ImageList (10000 items)
[Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28)]...
Path: /home/ubuntu/.fastai/data/mnist_png;
Test: None
查看‘training’文件夹内容
查看‘training’文件夹内容
(path/'training').ls() # 一个类别一个子文件夹
[PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/4'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/6'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/8'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/0'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/9'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/1'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/3'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/2'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/5'),
PosixPath('/home/ubuntu/.fastai/data/mnist_png/training/7')]
如何基于文件夹提取标注
如何基于文件夹提取标注
ll = sd.label_from_folder()
ll # 注意查看 哪些是LabelLists, CategoryList, ImageList
LabelLists;
Train: LabelList
y: CategoryList (60000 items)
[Category 4, Category 4, Category 4, Category 4, Category 4]...
Path: /home/ubuntu/.fastai/data/mnist_png
x: ImageList (60000 items)
[Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28)]...
Path: /home/ubuntu/.fastai/data/mnist_png;
Valid: LabelList
y: CategoryList (10000 items)
[Category 4, Category 4, Category 4, Category 4, Category 4]...
Path: /home/ubuntu/.fastai/data/mnist_png
x: ImageList (10000 items)
[Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28), Image (1, 28, 28)]...
Path: /home/ubuntu/.fastai/data/mnist_png;
Test: None
如何提取一个training 样本来查看
如何提取一个training 样本来查看
x,y = ll.train[0]
x.show()
print(y,x.shape)
4 torch.Size([1, 28, 28])
如何为训练和验证集图片做变形设置
如何为训练和验证集图片做变形设置
tfms = ([*rand_pad(padding=3, size=28, mode='zeros')], [])
# rand_pad = 做random padding, 注意args
# * 代表接受多个输出值
# 第一个[]针对训练集图片,第二个 []针对验证集图片
ll = ll.transform(tfms)
构建DataBunch时,在什么情况下不用Imagenet.stats
构建DataBunch时,在什么情况下不用Imagenet.stats
bs = 128
# 在不使用pre-trained model时,不用Imagenet.stats
data = ll.databunch(bs=bs).normalize()
如何展示一个DataBunch的训练样本
如何展示一个DataBunch的训练样本
x,y = data.train_ds[0]
x.show()
print(y)
4
如何展示训练样本的变形效果
如何展示训练样本的变形效果
- 从LL到DataBunch,图片被加入了形变,批量和normalization
- 因为是随机padding,每次变形都不太一样
def _plot(i,j,ax): data.train_ds[0][0].show(ax, cmap='gray')
plot_multi(_plot, 3, 3, figsize=(8,8))
如何获取一个批量的训练样本?
如何获取一个批量的训练样本?
xb,yb = data.one_batch()
xb.shape,yb.shape
(torch.Size([128, 1, 28, 28]), torch.Size([128]))
如何展示一个批量中的图片和标注
如何展示一个批量中的图片和标注
data.show_batch(rows=3, figsize=(5,5))
如何手动打造CNN
如何手动打造CNN
Basic CNN with batchnorm
特制conv层(特定kernel大小,stride,padding大小)
特制conv层(特定kernel大小,stride,padding大小)
def conv(ni,nf): return nn.Conv2d(ni, nf, kernel_size=3, stride=2, padding=1)
手动构建模型
手动构建模型
model = nn.Sequential(
conv(1, 8), # num_input = 1 channel, num_filters = 8 (channels out), feature_map_size = 14
nn.BatchNorm2d(8),
nn.ReLU(),
conv(8, 16), # num_filter自主选择,fm_size = 7, 因为stride=2, 14/2=7
nn.BatchNorm2d(16),
nn.ReLU(),
conv(16, 32), # 4 = 7/2 = 3.5 四舍五入,得到4; 选择filter数量为32
nn.BatchNorm2d(32),
nn.ReLU(),
conv(32, 16), # 2 = 4/2 = fm_size; 选择性降低filter的数量到16
nn.BatchNorm2d(16),
nn.ReLU(),
conv(16, 10), # 1, 在下降到10,因为我们要分出10个类别, 获得(10,1,1),但损失函数只接受vector
nn.BatchNorm2d(10),
Flatten() # remove (1,1) grid, 生成含有10个值的vector向量
)
如何从模型结构变为Learner
如何从模型结构变为Learner
learn = Learner(data, model, loss_func = nn.CrossEntropyLoss(), metrics=accuracy)
如何打印出Learner的内部细节
如何打印出Learner的内部细节
print(learn.summary())
# 注意shape,param数量,注意哪些层是可训练的
======================================================================
Layer (type) Output Shape Param # Trainable
======================================================================
Conv2d [128, 8, 14, 14] 80 True
______________________________________________________________________
BatchNorm2d [128, 8, 14, 14] 16 True
______________________________________________________________________
ReLU [128, 8, 14, 14] 0 False
______________________________________________________________________
Conv2d [128, 16, 7, 7] 1168 True
______________________________________________________________________
BatchNorm2d [128, 16, 7, 7] 32 True
______________________________________________________________________
ReLU [128, 16, 7, 7] 0 False
______________________________________________________________________
Conv2d [128, 32, 4, 4] 4640 True
______________________________________________________________________
BatchNorm2d [128, 32, 4, 4] 64 True
______________________________________________________________________
ReLU [128, 32, 4, 4] 0 False
______________________________________________________________________
Conv2d [128, 16, 2, 2] 4624 True
______________________________________________________________________
BatchNorm2d [128, 16, 2, 2] 32 True
______________________________________________________________________
ReLU [128, 16, 2, 2] 0 False
______________________________________________________________________
Conv2d [128, 10, 1, 1] 1450 True
______________________________________________________________________
BatchNorm2d [128, 10, 1, 1] 20 True
______________________________________________________________________
Flatten [128, 10] 0 False
______________________________________________________________________
Total params: 12126
Total trainable params: 12126
Total non-trainable params: 0
如何启用模型生成预测值
如何启用模型生成预测值
xb = xb.cuda() # 让数据在GPU上运行
model(xb).shape # 生成预测值
torch.Size([128, 10])
寻找学习率并作图
寻找学习率并作图
learn.lr_find(end_lr=100)
LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.
learn.recorder.plot()
训练模型 (挑选最优学习率)
训练模型 (挑选最优学习率)
learn.fit_one_cycle(3, max_lr=0.1)
Total time: 00:30
epoch |
train_loss |
valid_loss |
accuracy |
1 |
0.223167 |
0.217859 |
0.930500 |
2 |
0.136179 |
0.078651 |
0.976400 |
3 |
0.072080 |
0.038664 |
0.988600 |
Refactor
如何将BatchNorm, ReLU 也融入到一个conv创建函数里?
如何将BatchNorm, ReLU 也融入到一个conv创建函数里?
def conv2(ni,nf): return conv_layer(ni,nf,stride=2) # fastai提供了conv_layer
refactor后,建模变得更简练
refactor后,建模变得更简练
model = nn.Sequential(
conv2(1, 8), # 14, 非常便捷,只用在意输入输出channel数量即可
conv2(8, 16), # 7
conv2(16, 32), # 4
conv2(32, 16), # 2
conv2(16, 10), # 1
Flatten() # remove (1,1) grid
)
learn = Learner(data, model, loss_func = nn.CrossEntropyLoss(), metrics=accuracy)
训练10次后,效果很好,虽然中间准确率有波动,但损失值一直在下降
训练10次后,效果很好,虽然中间准确率有波动,但损失值一直在下降
learn.fit_one_cycle(10, max_lr=0.1)
Total time: 01:12
epoch |
train_loss |
valid_loss |
accuracy |
1 |
0.228332 |
0.206325 |
0.937500 |
2 |
0.189966 |
0.192558 |
0.940800 |
3 |
0.156765 |
0.092810 |
0.969100 |
4 |
0.135871 |
0.083914 |
0.973300 |
5 |
0.108844 |
0.071582 |
0.978000 |
6 |
0.105887 |
0.128586 |
0.960200 |
7 |
0.080699 |
0.052754 |
0.983200 |
8 |
0.066007 |
0.037588 |
0.988600 |
9 |
0.047513 |
0.030255 |
0.990200 |
10 |
0.044705 |
0.028373 |
0.991600 |
如何手写Resnet
简易增大模型方法
简易增大模型方法
- 每个conv2 layer后面加一个conv1 layer,这样不会改变feature map的大小,可以模型层数可以无限扩大
简易增大模型的弊端
简易增大模型的弊端
- 对比plain 20 vs 45 层模型,居然浅层模型损失值降得更低,很反常,为什么?
Kaiming He的解决方案
Kaiming He的解决方案
- 新设计让56层模型效果至少应该和20层模型一样
- 所有人都应该尝试将Resnet之前有趣的论文,尝试套入Res-block设计,看看有多少增效
为什么res-block有奇效?
为什么res-block有奇效?
Resnet-ish
手写ResBlock
手写ResBlock
class ResBlock(nn.Module):
def __init__(self, nf):
super().__init__()
self.conv1 = conv_layer(nf,nf)
self.conv2 = conv_layer(nf,nf)
def forward(self, x): return x + self.conv2(self.conv1(x))
fastai res_block函数只需填入filter 数量即可
fastai res_block函数只需填入filter 数量即可
help(res_block)
Help on function res_block in module fastai.layers:
res_block(nf, dense:bool=False, norm_type:Union[fastai.layers.NormType, NoneType]=<NormType.Batch: 1>, bottle:bool=False, **kwargs)
Resnet block of `nf` features.
用resblock将模型层数提升到原来的3倍
用resblock将模型层数提升到原来的3倍
model = nn.Sequential(
conv2(1, 8),
res_block(8), # 不改变原feature mapd饿大小
conv2(8, 16),
res_block(16),
conv2(16, 32),
res_block(32),
conv2(32, 16),
res_block(16),
conv2(16, 10),
Flatten()
)
为什么要经常做refactor?
为什么要经常做refactor?
如何将conv2 layer 与resblock合二为一
如何将conv2 layer 与resblock合二为一
def conv_and_res(ni,nf): return nn.Sequential(conv2(ni, nf), res_block(nf))
其他流程不变
其他流程不变
model = nn.Sequential(
conv_and_res(1, 8),
conv_and_res(8, 16),
conv_and_res(16, 32),
conv_and_res(32, 16),
conv2(16, 10),
Flatten()
)
learn = Learner(data, model, loss_func = nn.CrossEntropyLoss(), metrics=accuracy)
learn.lr_find(end_lr=100)
learn.recorder.plot()
LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.
learn.fit_one_cycle(12, max_lr=0.05)
Total time: 02:00
epoch |
train_loss |
valid_loss |
accuracy |
1 |
0.248807 |
0.121582 |
0.972800 |
2 |
0.120927 |
0.360583 |
0.890300 |
3 |
0.104013 |
0.074916 |
0.977800 |
4 |
0.081181 |
0.065717 |
0.980000 |
5 |
0.068514 |
0.096448 |
0.967200 |
6 |
0.061274 |
0.087955 |
0.971800 |
7 |
0.051673 |
0.033911 |
0.989400 |
8 |
0.048090 |
0.033234 |
0.988800 |
9 |
0.039095 |
0.024638 |
0.992400 |
10 |
0.023670 |
0.021215 |
0.993400 |
11 |
0.019128 |
0.016159 |
0.994500 |
12 |
0.021365 |
0.016120 |
0.995200 |
ResNet在MNIST中的效果比较
ResNet在MNIST中的效果比较
- 12次训练就打到数年前的顶级水平
- 因为resent的简易和奇效,得到偏爱,主流library花费大量时间优化Resnet,让其速度更快;而新出炉的设计往往训练较慢
如何融合DenseBlock and Resblock?
如何融合DenseBlock and Resblock?
- 什么是x.orig? -> original input
- 什么是fastai的Resblock 设计
如何理解Dense net工作原理和特点?
如何理解Dense net工作原理和特点?
- 占用很多内存
- 但参数很少,需要较少图片
- 适用于image segmentation
print(learn.summary())
======================================================================
Layer (type) Output Shape Param # Trainable
======================================================================
Conv2d [128, 8, 14, 14] 72 True
______________________________________________________________________
ReLU [128, 8, 14, 14] 0 False
______________________________________________________________________
BatchNorm2d [128, 8, 14, 14] 16 True
______________________________________________________________________
Conv2d [128, 8, 14, 14] 576 True
______________________________________________________________________
ReLU [128, 8, 14, 14] 0 False
______________________________________________________________________
BatchNorm2d [128, 8, 14, 14] 16 True
______________________________________________________________________
Conv2d [128, 8, 14, 14] 576 True
______________________________________________________________________
ReLU [128, 8, 14, 14] 0 False
______________________________________________________________________
BatchNorm2d [128, 8, 14, 14] 16 True
______________________________________________________________________
MergeLayer [128, 8, 14, 14] 0 False
______________________________________________________________________
Conv2d [128, 16, 7, 7] 1152 True
______________________________________________________________________
ReLU [128, 16, 7, 7] 0 False
______________________________________________________________________
BatchNorm2d [128, 16, 7, 7] 32 True
______________________________________________________________________
Conv2d [128, 16, 7, 7] 2304 True
______________________________________________________________________
ReLU [128, 16, 7, 7] 0 False
______________________________________________________________________
BatchNorm2d [128, 16, 7, 7] 32 True
______________________________________________________________________
Conv2d [128, 16, 7, 7] 2304 True
______________________________________________________________________
ReLU [128, 16, 7, 7] 0 False
______________________________________________________________________
BatchNorm2d [128, 16, 7, 7] 32 True
______________________________________________________________________
MergeLayer [128, 16, 7, 7] 0 False
______________________________________________________________________
Conv2d [128, 32, 4, 4] 4608 True
______________________________________________________________________
ReLU [128, 32, 4, 4] 0 False
______________________________________________________________________
BatchNorm2d [128, 32, 4, 4] 64 True
______________________________________________________________________
Conv2d [128, 32, 4, 4] 9216 True
______________________________________________________________________
ReLU [128, 32, 4, 4] 0 False
______________________________________________________________________
BatchNorm2d [128, 32, 4, 4] 64 True
______________________________________________________________________
Conv2d [128, 32, 4, 4] 9216 True
______________________________________________________________________
ReLU [128, 32, 4, 4] 0 False
______________________________________________________________________
BatchNorm2d [128, 32, 4, 4] 64 True
______________________________________________________________________
MergeLayer [128, 32, 4, 4] 0 False
______________________________________________________________________
Conv2d [128, 16, 2, 2] 4608 True
______________________________________________________________________
ReLU [128, 16, 2, 2] 0 False
______________________________________________________________________
BatchNorm2d [128, 16, 2, 2] 32 True
______________________________________________________________________
Conv2d [128, 16, 2, 2] 2304 True
______________________________________________________________________
ReLU [128, 16, 2, 2] 0 False
______________________________________________________________________
BatchNorm2d [128, 16, 2, 2] 32 True
______________________________________________________________________
Conv2d [128, 16, 2, 2] 2304 True
______________________________________________________________________
ReLU [128, 16, 2, 2] 0 False
______________________________________________________________________
BatchNorm2d [128, 16, 2, 2] 32 True
______________________________________________________________________
MergeLayer [128, 16, 2, 2] 0 False
______________________________________________________________________
Conv2d [128, 10, 1, 1] 1440 True
______________________________________________________________________
ReLU [128, 10, 1, 1] 0 False
______________________________________________________________________
BatchNorm2d [128, 10, 1, 1] 20 True
______________________________________________________________________
Flatten [128, 10] 0 False
______________________________________________________________________
Total params: 41132
Total trainable params: 41132
Total non-trainable params: 0