I am trying to get started with using data sets larger than my GPU memory but one problem I am running into atm is that I cannot reduce the batchsize as BcolzArrayIterator complains: batch_size needs to be a multiple of X.chunklen
Is there a way to reduce the chunklen of a bcolz carray?
It's very simple:
arr_hr_16 = bcolz.carray(arr_hr, expectedlen=16)
arr_lr_16 = bcolz.carray(arr_lr, expectedlen=16)
Not sure why arr_lr_16.chunklen returns 4, though
I think you are confusing the expectedlen parameter with the chunklen parameter (both optional)see: http://bcolz.readthedocs.io/en/latest/reference.html#the-carray-class
The expectedlen parameter helps the bcolz library to determine how many objects in your iterable to compress into one chunk where the chunlen paramter will set the ammount of objects that compress into the same chunk.
For our purposes it is generally a good idea to make the chunklen the same or a multiple of the batchsize.