DataBlock with Images generated by a function

I would like to create a DataBlock for Images. However, the images are not stored on disc in a folder, but are created on request by a function (as a 3D tensor). Can someone explain to me how this works?

Thank you very much and best regards!


1 Like

Hi @badassgenetics,

I also have this problem. I have had mixed results - I have explained my process so far below.

I am retrieving images from a remote server based on an id number. I am passing in a list of id numbers in as my “source”, and have defined get_x and get_y functions which retrieve the image and labels.

Running dblock.summary(train_ids) shows that the get_x and get_y functions seem to work - they are successfully returning data and fastai can format them as tensors properly, however it then fails to create a batch from the items. I get the following error:

Error! It's not possible to collate your items in a batch
memoryview is not collatable

With the following traceback

     46     b = t[0]
     47     return (default_collate(t) if isinstance(b, _collate_types)
---> 48             else type(t[0])([fa_collate(s) for s in zip(*t)]) if isinstance(b, Sequence)
     49             else default_collate(t))

NotImplementedError: multi-dimensional sub-views are not implemented

Any ideas? Does the memoryview error indicate that the issue is due to holding all the data in memory rather than on disk?

What is your application?