I’ve a image dataset properly labeled and split into train, test and valid folders but when I create an ImageDataBunch I’m prompted with the this error;
IndexError: index 0 is out of bounds for axis 0 with size 0
Although there’s another post in the forums about this same issue and there a possible solution to it but there’s no explanation to the error. Here’s the link to the other forum post.
I have the same problem on Kaggle
For me it worked when i changed the path specification from ‘content…’ to ‘\content…’
In my case, it is about the file name.
MNIST use the folder name ‘training’ and ‘testing’ instead of ‘train’ and ‘valid’ which is the default of from_folder.
It worked after I assign the folder name:
ImageDataBunch.from_folder(path, train=‘training’, valid=‘testing’, ds_tfms=tfms, size=26)
Reference from help(ImageDataBunch.from_folder):
from_folder(path: Union[pathlib.Path, str], train: Union[pathlib.Path, str] = ‘train’, valid: Union[pathlib.Path, str] = ‘valid’, valid_pct=None, classes: Collection = None, **kwargs: Any) -> ‘ImageDataBunch’ method of builtins.type instance
Create from imagenet style dataset in path with train,valid,test subfolders (or provide valid_pct).
What worked for me is I specified my path as follow (working on colab):
root_dir = ‘/content/drive/My Drive/fastai/’
path = Path(root_dir)
path = root_dir + ‘sleeves_lenghts/train’
I have added the test dataset to my databunch as follows:
data = ImageDataBunch.from_df(path_train_img, train_clf, ds_tfms=get_transforms(), size=256, bs=bs, test=path_test_img ).normalize(imagenet_stats)
I now want to get the predictions on the test set, so I’m using
preds, y, losses = learn.get_preds(ds_type=DatasetType.Test, with_loss=True)
which return the same error as you get:
IndexError: index 0 is out of bounds for axis 0 with size 0
is there any insight as to why this happens and how to fix it? BTW - my test data is unlabelled.
I had the same problem. The path that I had specified was incorrect. It was pointing to the train folder instead of the parent of the train folder.
You are a saviour !
Thanks for this
I had the same error when trying to load the MNIST dataset. I was following this tutorial: https://docs.fast.ai/tutorial.data.html.
What worked for me was to change the folder names inside the mnist_png folder:
cd /home/jupyter/.fastai/data/mnist_png
mv training/ train/
mv testing/ valid/
Then, loading with
path = untar_data(URLs.MNIST)
data = (ImageList.from_folder(path)
.split_by_folder()
.label_from_folder()
.transform(tfms, size=32)
.databunch()
.normalize(imagenet_stats))
worked.
Please note that this leaves the dataset with no samples for testing.
I also got stuck in the same error.
It appears this thread has been gaining a lot of attention lately. Although I’m not actively developing on fastai for now or probably until v3 is live but hopefully I can share my inputs into these problem people are facing.
You see ImageDataBunch creates train, test & valid sets automatically but at times, datasets like MNIST already comes preprocessed in separate folders like Train & Test. If that’s the case, ImageDataBunch will thow an error.
Besides, you can try troubleshooting by checking the path to the dataset or like mentioned previously, check if the “Train” & “Test” folders exists already. Then you’ll have to manually create a Valid set from the “Test” set.
Anyway, if there’s something missing out, do let me know, I’ll update the post accordingly, since I’m not caught up with the fastai library for a while.
Thanks a lot for this. I was feeling so helpless till I came across this. I did go through the documentation but I wish they had given more examples.
can you show with an example, i read the docs and still cant figure out how to load data.
my data is in the following structure:
similarly for test and pred.
pla help
Although I can understand reading documentations can be confusing but please make a habit of doing so. The fast.ai docs are probably one of the best out there among many but you’re still welcome to ask for help.
So in order for us to help you out better, you need to provide us with a minimal reproducible code snippet that you got working until stuck at some point. That way it’s efficient for us to guide you through & it’s a good practice for you in the long run too. So I request you to please share a code snippet that you got working until being stuck.
Besides, you should probably check out the snippets showcased in the Quickly get your data ready for training section of the vision.data documentation. I assume you’re a beginner, hence I’m obliged to mention that you can check out a plethora of techniques to load in an image dataset into the DataBunch class.
It’s understandable if it was a mouthful to pick up but don’t hesitate to questions on this thread or create a new thread if it’s something else that you’re confused with.
I couldn’t get it working either. Looks like lot needs to be learning about how data_block object work but couldn’t it be simpler like sklearn style.
As seen in the code bellow:
df_train = pd.read_csv(’/home/gkarmakar4969/data/fashion/training/fashion-mnist.csv’)
df_train.head()
data = ImageDataBunch.from_folder(PATH, train=‘training’, ds_tfms=tfms, valid_pct=0.2).normalize(imagenet_stats)
/anaconda/envs/py37_pytorch/lib/python3.7/site-packages/fastai/data_block.py:454: UserWarning: Your training set is empty. If this is by design, pass ignore_empty=True to remove this warning.
warn(“Your training set is empty. If this is by design, pass ignore_empty=True to remove this warning.”)
/anaconda/envs/py37_pytorch/lib/python3.7/site-packages/fastai/data_block.py:457: UserWarning: Your validation set is empty. If this is by design, use split_none()
or pass ignore_empty=True when labelling to remove this warning.
or pass ignore_empty=True when labelling to remove this warning.""")