Weird exception when language modeling with new fastai.text code

Getting close to having language modeling working with the new fastai.text namespace, but running into error …

Can anyone give me some insight as to what I need to investigate or what exactly I may be doing wrong here??? I’m not sure what to make of this exception given the stack trace below.

My code:

md = LanguageModelData(PATH, 1, len(trn_ds.vocab), trn_dl, val_dl)
opt_fn = partial(optim.Adam, betas=(0.7, 0.99))

learner = md.get_model(opt_fn, em_sz, nh, nl,
               dropouti=0.05, dropout=0.05, wdrop=0.1, dropoute=0.02, dropouth=0.05)
learner.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)

batch_iter = iter(md.val_dl)
b = next(batch_iter) # x, y
p = learner.model(V(b[0]))       # predictions

print(b[0].size(), b[1].size(), p.size())b = next(batch_iter) # x, y
p = learner.model(V(b[0]))       # predictions

print(b[0].size(), b[1].size(), p.size())

Throws a CUDA exception on the p = learner.model(V(b[0])) line :

RuntimeError: cuda runtime error (48) : no kernel image is available for execution on the device at /opt/conda/conda-bld/pytorch_1518244421288/work/torch/lib/THC/generic/

Partial stack trace:

So debugging this shows that*sz) returns a rank 1 tensor of 0’s. Not sure if the rest of the data coming back here is right but this definitely looks off.

For others that come across this …

It appears that this is happening because pytorch no longer plays nice with my NVIDIA 960M GPU. Yah, I know that is underpowered for just about everything, but it’s what came with my Dell XPS 15 and has enabled me in the past to do a lot of development on it with a small sample dataset before moving working code to AWS.

Anyhow, you see something like this … it likely means its time to upgrade your GPU.

1 Like