We’re experimenting with Lesson 3 and everything works fine when we use it with the dataset that comes through URLs.CAMVID_TIDY but when we download the larger CAMVID dataset from U. of Cambridge we get the error “CUDA error: device-side assert triggered”.
The only substantial difference we saw is that the y variable for CAMVID_TINY is greyscale, whereas U. of Cambridge label images are colorized.
Any idea how to get out of this impasse? We have pathology slide data waiting for FASTAI to be applied and we are getting the same error
Set the CUDA_LAUNCH_BLOCKING environment variable to 1 (In jupyter you can use os.environ['CUDA_LAUNCH_BLOCKING'] = '1'). This will cause it to stop at the actual line causing the issue (otherwise as GPU operations are asynchonous it will stop at a later point). That might give a better idea.
Or try running it on the CPU if possible, some operations aren’t supported, you can’t use `to_fp16 for instance. But this will give a proper error message. You may want to reduce your dataset size if it’s at the end of the epoch as happens with errors in the loss function.
