splitter = RandomSplitter(), that has a seed (may be try setting that)
I haven’t succeeded with getting reproducible results either. The module is really helpful @stas thank you 
1 Like
FWIW RandomSplitter’s default seed is 42, so setting it 42 manually shouldn’t (to me) make a difference
1 Like
Finally, I am able to reproduce results !
In a nutshell:
- In order to reproduce your results for demonstration purposes etc. use fastaiV1 not fastaiV2
- @stas provided a code snippet that you must place at the top of your notebook (original post at the top)
- call: random_ctl(bad_seed) first in every cell when randomness comes into play which is: before you create your databunch, before you create your learner, before you call fit_one_cycle —> HINT: set bad_seed != 0 for reproducibility
- If done correctly you can then recreate your learner and run fit_one_cycle
A working example is shown in my example notebook:
Best regards, thank you all and special thanks to @stas !
3 Likes
def RandomSplitter(valid_pct=0.2, seed=None, **kwargs):
"Create function that splits `items` between train/val with `valid_pct` randomly."
def _inner(o, **kwargs):
if seed is not None: torch.manual_seed(seed)
rand_idx = L(int(i) for i in torch.randperm(len(o)))
cut = int(valid_pct * len(o))
return rand_idx[cut:],rand_idx[:cut]
return _inner
where are we setting the seed set to 42 ?
not sure if this changed recently or we have been setting 42 regularly 
1 Like
Nice!!
Not sure what we are missing in v2.
Here is the other thread [Solved] Reproducibility: Where is the randomness coming in? (success in v1 but not in v2) 
2 Likes
Honestly I can’t recall. Looking through all this seed stuff may have done it for me. I think it’s the call to set_seed sets that particular seed but having the 42 at the end of the day may also be needed. (And I think that’s where I got confused). IE if we set the manual seed I don’t believe we need to state it twice
The random_ctl function has one very important feature, which I think is being overlooked.
Have you ever had a situation that every 50th run the code fails or you get a terrible outcome? And then you have to re-run another 50+/-50 times to get it repeated, or it only appears when you are doing a live demo? But even if you do you won’t be able to fix it. That’s where knowing the seed saves the day. So, normally, you always run this function but w/o forcing the seed, i.e.:
random_ctl(0)
So by default it doesn’t go against the fastai philosophy. It just sets a new random seed every time you run it. So you can just have it in your nb template and forget about it.
if, however, you get a bad outcome, voila, this function reported the seed it randomly picked when it was run.
Using seed 999
Now you can re-run the code with:
random_ctl(bad_seed)
(bad_seed is the one reported by the function) and look at the unique situation that caused the problem, debug it, fix your code and then go back to random_ctl(0).
1 Like
I’m glad you were able to isolate that this is a bug in fastai2 - next step is to report this issue https://github.com/fastai/fastai2/issues
@AMusic discovered:
- In order to reproduce your results for demonstration purposes etc. use fastaiV1 not fastaiV2
- @stas provided a code snippet that you must place at the top of your notebook (original post at the top)
- call: random_ctl(bad_seed) first in every cell when randomness comes into play which is: before you create your databunch, before you create your learner, before you call fit_one_cycle —> HINT: set bad_seed != 0 for reproducibility
- If done correctly you can then recreate your learner and run fit_one_cycle
The fact that in fastaiv1 you have to reset the seeds in three places seems to imply that these operations can leave the RNGs in different states between runs. There must be an indeterminacy after the operations. Maybe there is another seed we have not identified and reset.
fastaiv2 shows a bigger problem - even with seed resets, results are not reproducible. The implication is that the RNGs are altered after the seed reset and before or during the operations.
Please excuse me if this logic is faulty. It’s late and I have been watching episodes of Dollhouse.
There was indeed a problem with fastai2’s DataLoader, which was not setting its seed in a reproducible fashion. Fixed and confirmed that the following code gives the same results all the time:
set_seed(42)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
path = untar_data(URLs.PETS)
fnames = get_image_files(path/"images")
pat = r'^(.*)_\d+.jpg'
dls = ImageDataLoaders.from_name_re(path, fnames, pat, bs=64, item_tfms=Resize(224))
learn = cnn_learner(dls, resnet34)
learn.fit_one_cycle(1)
5 Likes
Just wanted to inform you that I updated the fastai2 library using:
pip install fastai2 --upgrade
Then I restarted the kernel and I still get slightly different error_rates when I run your conde snippet from above.
Any idea?
You need to use a developer install to have the latest fixes/features. This won’t be available via pip install fastai2 --upgrade until we make a new release of fastai2 (probably before next lesson on Tuesday).
In the meantime, you do a dev install with cloning the fastai2 repo then:
cd fastai2
pip install -e .
Thanks for the quick reply. As I am using paperspace I will wait for the new release.
1 Like
Glad it got resolved.
Meanwhile, if you’re already using ipyexperiments, you can now enable an automatic RNG seed setting by passing cl_set_seed=SEED, e.g. cl_set_seed=42. Here is an example:
# cell 1
import numpy as np
from ipyexperiments import IPyExperimentsPytorch
# cell 2
exp13 = IPyExperimentsPytorch(exp_enable=False, cl_set_seed=42, cl_compact=True)
rnd1 = np.random.random()
# cell 3 (the seed gets reset automatically before the cell is run)
rnd2 = np.random.random()
assert rnd1 == rnd2, f"values should be the same rnd1={rnd1} rnd2={rnd2}"
Of course, here it’d be a train loop cell that can be re-run again and again and get reproducible results, if that’s what you’re after.
The new version ipyexperiments-0.1.17 is on pypi and conda servers.
1 Like
If you want to install the latest fastai2 version (from master), you can run the following command in a a jupyter cell at the beginning of your notebook:
!pip install git+https://github.com/fastai/fastai2.git
make sure to also install the fastcore latest version (from master too) in order to have both packages aligned:
!pip install git+https://github.com/fastai/fastai2.git
You need to run these 2 commands each time you start a new virtual machine. Therefore you can save me at the top of your notebook.