Hi, I am new using fastai. The first time, I ran the code, it works, no problem, but the second time I ran the code, it shows me the next error. Any suggestions on how to solve it?
This is the environment
PyTorch: 1.6.0
fastai: 1.0.61
torchvision: 0.7.0
python: 3.6
cuda: 10.1.105
Tesla k40c
I ran
>>> import torch
>>> from torchvision import models, transforms
>>> from fastai.vision import *
>>> torch.cuda.is_available()
True
>>> ...
>>> learner = cnn_learner(data=data, base_arch=model, pretrained=True, loss_func=loss, metrics=accuracy)
File "main.py", line 63, in <module>
learner = cnn_learner(data=data_full, base_arch=model, pretrained=True, loss_func=loss, metrics=accuracy)
File "/python3.6/site-packages/fastai/vision/learner.py", line 105, in cnn_learner
if init: apply_init(model[1], init)
File "python3.6/site-packages/fastai/torch_core.py", line 257, in apply_init
apply_leaf(m, partial(cond_init, init_func=init_func))
File "/python3.6/site-packages/fastai/torch_core.py", line 253, in apply_leaf
for l in c: apply_leaf(l,f)
File "/python3.6/site-packages/fastai/torch_core.py", line 252, in apply_leaf
if isinstance(m, nn.Module): f(m)
File "/python3.6/site-packages/fastai/torch_core.py", line 247, in cond_init
if (not isinstance(m, bn_types)) and requires_grad(m): init_default(m, init_func)
File "/python3.6/site-packages/fastai/torch_core.py", line 241, in init_default
if hasattr(m, 'weight'): func(m.weight)
File "/python3.6/site-packages/torch/nn/init.py", line 420, in kaiming_normal_
return tensor.normal_(0, std)
RuntimeError: CUDA error: no kernel image is available for execution on the device
Also can reproduce the same error as the following code:
>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.tensor([1.0, 2.0])
tensor([1., 2.])
>>> torch.tensor([1.0, 2.0]).cuda()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/python3.6/site-packages/torch/tensor.py", line 153, in __repr__
return torch._tensor_str._str(self)
File "/python3.6/site-packages/torch/_tensor_str.py", line 371, in _str
return _str_intern(self)
File "/python3.6/site-packages/torch/_tensor_str.py", line 351, in _str_intern
tensor_str = _tensor_str(self, indent)
File "/python3.6/site-packages/torch/_tensor_str.py", line 241, in _tensor_str
formatter = _Formatter(get_summarized_data(self) if summarize else self)
File "/python3.6/site-packages/torch/_tensor_str.py", line 89, in __init__
nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
RuntimeError: CUDA error: no kernel image is available for execution on the device
The first time I ran it, it works. I have been looking around but I didn’t find anything about it. As I said before, the first day I ran the code and it works and the second day it got me that error. Please any suggestion 