Custom Dataloader for "npy" files

Part 1 (2019)
1

Hello Everyone,

I am trying to create a custom data loader for “.npy” files. Here is what the data loader looks like:

def open_npy(fn):
    x = np.load(str(fn))
    return vision.Image(torch.Tensor(x))

class NpyItemList(vision.ImageList):
    def open(self, fn): return open_npy(fn)
    
class NpyNpyList(NpyItemList):
    _label_cls = NpyItemList

def get_y_fn(x):
    parent = 'train' if 'train' in str(x) else 'valid'
    fn = data_dir/'t1'/parent/f'{str(x.stem)}.npy'
    return fn

idb = (NpyNpyList.from_folder(data_dir/'low_res', extensions=('.npy'))
       .split_by_folder()
       .label_from_func(get_y_fn)
       .databunch(bs=2))

When I try to run the learning rate finder, I get the following error:

RuntimeError: Expected 5-dimensional input for 5-dimensional weight 64 1 9 9, but got 4-dimensional input of size [2, 64, 64, 64] instead

What could be the cause of this error?

Thanks,

Kishan

2

I was able to figure out the issue, and it was unrelated to the dataloader.

I realized the issue was not with loading npy itself. The issue was with the size of the tensor before inserting it in the neural network. After changing the tensor size, I was able to resolve the issue. This seems quite obvious to me in hindsight, but since I was deal with a new datatype I assumed the issue was with it.

Note: For anyone else interested in designing a npy dataloader you can refer to this thread.