Strange error creating Dataloaders

[Mark_F]

I am getting a strange error when I try to create a Dataloaders. In the screenshot below, I’ve used factory method, but same happens with DataBlockapi. The error that I"m getting is:

RuntimeError: cusolver error: CUSOLVER_STATUS_EXECUTION_FAILED, when calling `cusolverDnSgetrf( handle, m, n, dA, ldda, static_cast<float*>(dataPtr.get()), ipiv, info)`

What’s odder is that if I rerun the cell a second time, there is no error. Any idea what is causing this? The full trace is long but here is the end of it:

[mike.moloch (github:aeamaea)]

Popped it into the search box and what do I know, someone else had the exact same problem a week ago!

Walk-thru 10

In case others run into this - I was getting an error on Paperspace - not a CUDA memory issue but “Could not do one pass in your dataloader, there is something wrong in it. Please see the stack trace below” and the bottom of the trace was “cusolver error: CUSOLVER_STATUS_EXECUTION_FAILED, when calling `cusolverDnSgetrf( handle, m, n, dA, ldda, static_cast<float*>(dataPtr.get()), ipiv, info)”. I found that reducing the image size in the resize property (not the size) avoided this. Despite dropp…

[nikem (Niyazi Kemer)]

I had exactly the same error. It is Pytorch Bug. If you upgrade your driver it will be sorted out. Which driver do you use currently?

Walk-thru 10

Finally solved the problem. It turns out it is not about the walk-thru code/data or anything else. It is all about a PyTorch bug that appears on certain Nvidia drivers.

[Mark_F]

I’m on Paperspace, so presumably using whatever is there. Not sure that I can do an upgrade. But if I rerun the cell, the problem seems to go away. Thanks.

[nikem (Niyazi Kemer)]

As far as I understand, it happens occasionally, may be some servers are up to date some others are not.