I’m occassionally getting this warning/error when I fit() or run lr_find() on my learner:
Exception in thread Thread-25:
Traceback (most recent call last):
File “/home/ubuntu/anaconda3/envs/fastai/lib/python3.6/threading.py”, line 916, in _bootstrap_inner
File “/home/ubuntu/anaconda3/envs/fastai/lib/python3.6/site-packages/tqdm/_tqdm.py”, line 144, in run
for instance in self.tqdm_cls._instances:
File “/home/ubuntu/anaconda3/envs/fastai/lib/python3.6/_weakrefset.py”, line 60, in iter
for itemref in self.data:
RuntimeError: Set changed size during iteration
I have been experiencing and ignoring this error, as my LRs seem to work out correctly. But I’m guessing a deep dive into the code will find that at some point the code is not waiting on a global lock before updating the underlying set.
I got the same problem with personal setup (1080 ti). The problem is that if you ignore it and run again, it may run the remaining steps easily has to do it one by one. (I tried to run all to keep a record but stopped by this error.)
Seem you have to stop the kernel. Then run one or two steps and the try to run “all steps below”.