My training is not repeatable

Part 1 (2019)
1

I was experimenting with different learning rates for a tabular model and I noticed that my training was not repeatable.

For example, I would initialize the learner:
learn = tabular_learner(data, layers=[60], metrics=accuracy)

Then use fit_one_cycle for a given number of epochs and a certain learning rate.

If I repeat this process (initialize the learner and run fit_one_cycle) for the exact same inputs I am getting different results. Sometimes the accuracy may be 10% different than the previous case.

I assume there would be some variability because the weights are randomly initialized although I haven’t figured out where to set the seed for this. But should the deviation from one training to the next be so high?

1 Like
2

try setting the seed using a code snippet like this:

def seed_everything(seed):
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.deterministic = True

SEED = 0
seed_everything(SEED)
1 Like
3

Somehow, even after calling seed_everything(), I’m getting different results every time.

I’m calling seed_everything() twice; once before creating the Learner, and once before calling Learner.fine_tune(). But the results are different every time. (I’ve used random_state=42 in train_test_split() as well…)

4
1 Like