Camvid Lesson 3 - IndexError

I am trying to run the ‘Subset classes’ code block in the lesson3-camvid jupyter notebook:

path = Path('./data/camvid-small')`

def get_y_fn(x): return Path(str(x.parent)+'annot')/x.name

codes = array(['Sky', 'Building', 'Pole', 'Road', 'Sidewalk', 'Tree',
    'Sign', 'Fence', 'Car', 'Pedestrian', 'Cyclist', 'Void'])

src = (SegmentationItemList.from_folder(path)
       .split_by_folder(valid='val')
       .label_from_func(get_y_fn, classes=codes))

bs=8
data = (src.transform(get_transforms(), tfm_y=True)
        .databunch(bs=bs)
        .normalize(imagenet_stats))

But when I run it, the code throws an IndexError:

IndexError: index 0 is out of bounds for axis 0 with size 0

Upon further investigation by printing the SegmentationItemList.from_folder(path) portion, apparently the folder that the path points to - ‘data/camvid-small’ - is empty.

Which means that the untar_data function, which is supposed to download the camvid data from URLS. is not downloading anything in the first place.

Anyone know what to do? Much appreciated.

I had this error before, and it is because there was no pictures in the folder.

1 Like

I see. Alright, I’ll probably do a manual download of the images then. Thanks!

hi, Could you at least post the picture of the directory of where you store your picture?
In addition, I want to see what is inside the variable “path” by running the “path” variable?
If this still bothering you, I suggest just run it in colab using google’s GPU to train for free.

1 Like

This is the directory in which I store my pictures. It worked well enough with the code for Lesson 1, when we were building a teddy bear detector:

image

But in Lesson 3, everytime I ran a cell with a mkdir command, nothing would appear in the directory. Maybe there’s a change made in Lesson 1’s notebooks that wasn’t ported over to Lesson 3, I don’t know.

See cell under ‘Subset classes’ for path:
image

Appreciate the advice - I may check out Colab. Although I am already using GCP.

Here is another try: Try to create another block to run this code:
path = untar_data(URLs.CAMVID)
Of couse, don’t forget to run those import fastai blocks

If this still fails, I suggest just use colab or use colab to solve your problem from below:
I can’t totally see what is inside of the resolution picture.
Please check that under the data folder, it has these files.


Or please just use my colab notebook run it step by step to see how the folder should be structured.
After that check it with your own local folder, I think you will find the error over there

Click here to see my colab notebook

Don’t forget to use click runtime->change runtime type to use GPU.
After using your colab notebook, try to see these folder on your computer or not after running these codes:

from fastai.vision import *

from fastai.callbacks.hooks import *

from fastai.utils.mem import *
path = untar_data(URLs.CAMVID)

Here is my files after running the code in above: If you don’t see them in your computer, it may has issue.

2 Likes

Thanks. These are the lines I get when I run path = untar_data(URLs.CAMVID) and then print out the path variable.

[PosixPath('/home/jupyter/.fastai/data/camvid/valid.txt'),
 PosixPath('/home/jupyter/.fastai/data/camvid/labels'),
 PosixPath('/home/jupyter/.fastai/data/camvid/images'),
 PosixPath('/home/jupyter/.fastai/data/camvid/codes.txt')]

which I think is what we expect.

Now, after running untar_data, I still don’t see the files in my folder. But this is likely because my data folder is in the path tutorials/fastai/course-v3/nbs/dl1. So the pictures are not downloading into my folder since the path inside URLs.CAMVID are different.
Thus far, I have assumed that this is supposed to be the case. Whenever I run the jupyter notebook instance (as advised in the fastai webpage), the ‘root’ folder I have access to is only tutorials and not home or jupyter. So, I have assumed that (due to resource constraints?) that pictures downloaded by untar_data are downloaded into some other folder home/jupyter/.fastai ... in which I am not supposed to have access to.
Sorry I am not a data engineer so I have no idea how the structure is supposed to work.

Anyhow, just FYI, I tried running your Colab notebook. I ran into an OS error in the cell:

codes = np.loadtxt(path/'codes.txt', dtype=str); codes

The OS error is :

data/camvid-small/codes.txt not found.

But that’s because the data was originally downloaded to data/camvid and not data/camvid-small. That should mean that the path variable got modified to data/camvid-small after untar_data was called. I can’t see which line of code modified that though. Anyhow, once I manually changed the path to data/camvid/codes.txt, it ran fine.

Regardless, I also just tried running my own jupyter notebook in GCP again. For some strange reason, everything works now. I don’t know if running your Colab notebook downloaded the pictures into my computer/whichever destination folder. But it works.

One guess as to what is causing all this is that sometimes the connection to the web server to download the images may not be stable, or that not all the data/files get downloaded, and that untar_data does not check if all files are downloaded properly. The user needs to check it himself/herself. But that’s just my guess.

Finally, something that I did forget to mention, is that the ‘Subset classes’ block in the jupyter notebook was commented out in the first place, so I wasn’t meant to run it. I only uncommented it because I was experiencing OSError (similar to what I just experienced in your Colab notebook) where the codes.txt file was not being located.

Thanks again for all of your help!

Just a reminder.
You may forget to run this code before running everything. If so, it will cause the data folder error.:
!curl https://course.fast.ai/setup/colab | bash

I see, ok will run that line first. Thanks!