Why Do we get Different Results on Different Runs?

Part 1 (2020)
1

Can any one kindly give like an overview answer as to why two runs on the same model return different accuracy or losses when run a different time?

Or is there a way to get the same results even after a different run? :thinking:

2

Reproduciability is one of the biggest problem in machine learning. No two runs have the same result. There is lots of randomness in the learning process like splitting train test data, choosing which set of data being used in each batch and so on. This is expected. As long as the two runs have similar values instead of the same value, we should be good.

2 Likes
3

I agree with what @nareshr8 wrote. On top of that, random weight initialization and dropout are additional sources of randomness in the training procedure of neural networks.

@jimmiemunyi I also recommend having a look at this thread: [Solved] Reproducibility: Where is the randomness coming in?

2 Likes
4

thanks @nareshr8 and @stefan-ai.

Thanks for the thread suggestion too

5

random_seed(0,use_cuda=False ) before tabular_learner and also fit_one_cycle

in V1 was helpful in reproducing results.

1 Like
6

You get different results when making inference? Or when training?
Because if it is when training it is normal and that is why they have put you up, but if it is when making inference it should not happen

1 Like
7

I’d suggest looking here: [Solved] Reproducibility: Where is the randomness coming in? - #28 by harikrishnanrajeev

but what I found to reproduce is just the line

set_seed(42)

before the dataloaders.

Here is my theory as to why you need this. If anyone can confirm or correct this please do:

  • Before each epoch the training set is randomly shuffled.
  • The weights are updated every mini-batch.
  • If you do not have a seed for this your weights will be updated in a different order which will give you different weights and therefor a model which will perform differently on the validation set
  • Using set_seed(42) you set a seed for this random shuffling of the training set so it will shuffle it the same way each time giving you identical runs for each epoch when you run the whole thing again.

is this right?

if so, are we handicapping the the model training because its shuffling the training set the same way for each epoch?

8

@daveramseymusic your understanding is perfectly correct. Coming to your question, in the production setting different random sequences of ‘set_seed()’ are chosen to find the better minima, then the ensembling technique is used on best-performing models.

1 Like