Lesson 1 - results change on re-run?

Part 1 (2018) Beginner (2018)
1

Trying dogs vs cats, but on tulips vs. roses, with 10-20 images in each sample, I got ~70% accuracy with 3 epochs, tried 5 epochs and got 90% accuracy, then did 3 epochs again and got 50%. Repeated with 3 several times and got different results each time. In contrast, re-running on the dogs vs cats data from the Lesson 1 seemed to give precisely the same results each time. Any pointers toward where in the code I can dig to understand this? Thanks!

2

There could be many reasons for this. It is always best practice to provide as much info as possible, posting a gist would be best but at a minimum you should post the code you are using and the results so others can view it and help you out.

If posting code use [ code] my code [/code] tags (without the space in the first tag).

1 Like
3

Thanks, Amrit, for the help getting started. The results I am seeing are now posted at this gist. Looking at it more closely, I saw that if I re-run the dogs vs. cats training, the results do in fact change slightly from run to run, so it no longer seems something’s “wrong” with my results. I am still curious to learn where the randomness appears in the training process. I expect that will get clearer as I work through the course.

4

@dangoldner. You only need to use this portion of the code once:

arch=resnet34
data = ImageClassifierData.from_paths(PATH, tfms=tfms_from_model(arch, sz))
learn = ConvLearner.pretrained(arch, data, precompute=True)

and then you can use

learn.fit(0.01, 3)

until you start to overfit.

Your results depend on a number of things:

  • the size of each sample. If your sample size is 10-20 images you will get alot more variation when compared to larger datasets. Also 20% of your sample is being used for validation.
  • the size of your images
  • the learning rate. As you have such a small sample I am assuming the learning rate finder is blank.

I would recommend using a larger dataset and see what results you get. Hope that helps.

5

Makes sense - thanks!

6

Hi dan,

There is some randomness to these networks. first big area is that the initial weights are randomized. also, you’ll see some randomness whenever computing is distributed among threads. this happens on both the pytorch and fast.ai side.

Here’s some things I tried to do to get it to run in a deterministic fashion.

manual_seed = 555
random.seed(manual_seed)
np.random.seed(manual_seed)
torch.manual_seed(manual_seed)
torch.cuda.manual_seed_all(manual_seed)
torch.backends.cudnn.deterministic = True

This should cover most of it from the pytorch side. However, I think that the thread use in fast.ai was still introducing some amount of invariability. you would have to make everything run single threaded or do something to ensure things run in the same order each time.

2 Likes
7

Thanks, Dave - this gives a lot of insight quickly into the different random selections through the process. Much obliged!