Q about train_imagenette.py

Seb · May 29, 2019, 7:55pm

It feels like I shouldn’t be opening a new thread for this but here we go:

fastai/fastai/blob/master/examples/train_imagenette.py

from fastai.script import *
from fastai.vision import *
from fastai.callbacks import *
from fastai.distributed import *
from fastprogress import fastprogress
from torchvision.models import *
from fastai.vision.models.xresnet import *
from fastai.vision.models.xresnet2 import *
from fastai.vision.models.presnet import *

torch.backends.cudnn.benchmark = True
fastprogress.MAX_COLS = 80

def get_data(size, woof, bs, workers=None):
    if   size<=128: path = URLs.IMAGEWOOF_160 if woof else URLs.IMAGENETTE_160
    elif size<=224: path = URLs.IMAGEWOOF_320 if woof else URLs.IMAGENETTE_320
    else          : path = URLs.IMAGEWOOF     if woof else URLs.IMAGENETTE
    path = untar_data(path)

    n_gpus = num_distrib() or 1

This file has been truncated. show original

In both train_imagenette and train_imagenet, we have the following line:
bs_rat = tot_bs/256.

Is 256 a hard coded value that should actually be our chosen batch size per GPU?

That would explain why I got better results when replicating results from the Imagenette leaderboard to use as my baseline (I hadn’t used that line). Jeremy used a bs=64, so his lr would have been divided by 4 unintentionally.

sgugger · May 30, 2019, 1:03pm

You can definitely make it a parameter if you want. It’s just the best learning rate was computed for bs=256 in our case, so we adapt it with this rule of thumb.

Seb · May 30, 2019, 3:00pm

Thanks for the clarification!