I have only rudimentary computer science knowledge so I apologize if this is a simple question, but I am stuck on untar_data(). Specifically, the first argument is (url:str,…). If I pass an external url to untar_data, I get an error. It seems that the example used in the course (“URLs.PETS”) is part of the class “URLs”, and when I call help, it provides a list of available datasets. Why can I not pass an external URL to untar_data? How do I modify an external URL (eg. CIFAR-10 [https://www.cs.toronto.edu/~kriz/cifar.html]) so that I can use it in the Jupyter notebook? I am getting an error that it is not a tar zip file (sorry I forget the exact errror).
Also, the paths for the pets dataset is ‘/home/ubuntu/.fastai/data/oxford-iiit-pet’. I have found this path on the terminal but cannot find it in the Salamander GUI directory. I am seeing “cifar-100.tgz.tgz” and cifar-10-python.tar.gz.tgz" in this directory, so it must have uploaded at some point when I entered the URL.
Finally, in the Image_Data_Bunch.from_name_re () method, why do we need to specify both path and fnames? Both seem to be path objects, and the fnames argument is just longer, with the file names. Does Image_Data_Bunch subtract path from fnames to determine the file name and apply the regular expression?
Sure! Let’s say your city is New York City. You’d have 2 labels: nyc and not_nyc. As far as the model is concerned, that’s no different than the labels dog and cat. In fact, you could find New Yorkers who’d say the difference between nyc and not_nyc is bigger than between cars and dogs.
Another ex: hotdog and not_hotdog, which was featured on an app on the TV show Silicon Valley.
Here’s an example of someone recreating it using fastai:
Thank you very much Anders, never thought of it this way, training non_city with images of different cities. I was more thinking in terms of not training non_city at all.
Beginner question: When I use the imageDataBunch method from folder, classes are numbered alphabetically (i.e. clas1 is 0 in the predictions, clas2 is 1 etc). How could I change this order? In my case (anomaly detection), I have 2 classes (‘good’ and ‘bad’) and I would like ‘bad’ to get index 1 and ‘good’ index 0 as it is usually the convention. Thanks a lot!
You’re welcome! I’m still very new at this, but I think one of the major pieces of doing deep learning or any machine learning is figuring out, how do I get turn my problem into something computers are good at doing?
It’s like cooking on an outdoor grill. Grills are great at cooking food that’s a certain size, like a burger. They’re not great at cooking small pieces of meat or vegetables — they’ll fall through the grill. That’s why people use skewers/kabobs: they use the skewer to “transform” several too-small pieces into one big enough piece, and now you can grill it.
Similarly, if a deep learning method needs more than 1 value, you change the way you’re defining your problem to give it 2 values.
I’m totally new at this so please forgive me if I’m asking obvious questions. I’ve gone through the first lesson a few times now and although I can shift+enter to run the code, I don’t feel like I know what I’m doing. I feel like I need an introductory course, maybe with some history of how deep learning works. That aside, I do have a specific question.
Regarding most_confused(), if the system is able to tell me which ones it got wrong, why didn’t it just adjust and get them right? And how does it know which ones it got wrong?
Really sorry if this is obvious but I don’t see this explained anywhere in the first lesson.
Hi Ted, I’m a beginner as well but I’ll try to explain things the best I can.
most_confused() gives us the instances it got wrong from the validation set. The model can not adjust and get them right, because it does not “read” (train) from the validation set. Validation set is kept aside to determine how good our model is doing.
Regarding how it knows which ones are wrong: Items in the validation set actually have their label. Our model first predicts what the item label is (with a certain confidence value) then looks at the actual label and determines whether it predicted right or wrong.
To sum it up, we have two sets of data :
Training set
Validation set
The model first looks at the training set items with its labels. Then predicts the labels for items in the validation set. Then checks whether it predicted correctly and how wrong (or right) it was in predicting.
5.7% error prior to unfreezing is pretty decent. Now I have a few question of my own, when you say unfreezing made everything worse, exactly how much worse did things become? Was that an observation you made after 1 epoch or several epochs? You could try something like learn.fit_one_cycle(5,slice(1e-5,1e-4). Yes eyeballing the dip and taking value that is somewhat smaller seems to work, but then again its not exactly a science per se, you could try messing around with it and see what works best for your dataset.
If you feel a little overwhelmed, don’t be. The way things are done around here is that we will first show you how things are done, then gradually let you in on how things actually work behind the scenes. Its sorta like how you would teach a kid to play basketball, you’d play the game with him and skip the technical details like the physics of a ball’s trajectory etc
Thanks for the encouragement, @dreambeats, I need it!
I’ve just finished lesson one (went through it twice now) and would like to create my own data set. I have a set of images I’d like to classify into two types. In type one there is a blue dot on a device and in type two, the dot is purple. Several questions here (using Crestle):
Must I use notebooks to experiment or is there command line I can access somewhere?
How do I (where do I) upload my dataset to?
Should I be doing some kind of two-pass approach (images that have a dot followed by a distinction of their color)?
If I’ve understood lesson 1 correctly, it seems like fastai has some tools for cropping the input. (to make it square) Is this true? Did I understand correctly?
Again, lots of thanks in advance. I really, really appreciate the pointers!
You do not necessarily need to use a notebook to do all this, but I think it is an amazing platform for us to mess around and see what works, and even to do serious stuff. The developers of fastai use it too! But if you really want to, you can toss all the same code in your notebook into a Python script and run it, that works too.
This depends on where you are doing your development, is it on your local machine? Or is it on a cloud instance? You sound like you already have that dataset ready in your PC, if you want to upload it to a virtual instance that you are using you can do it here.
I’ve just finished lesson one (went through it twice now) and would like to create my own data set. I have a set of images I’d like to classify into two types. In type one there is a blue dot on a device and in type two, the dot is purple. Several questions here (using Crestle):
I might be late to pitch in my suggestion, but if you could, watch the first 30 minutes(?) of Lesson 2 or just refer to the Homework in the viewer resources of Lesson 1. Concretely, read through Lesson2-download.ipynb. The process goes through curating your own dataset(from google images) using the included scripts from which you could experiment with the ImageDataBunch or even the Data Block API. Hopefully you can make your own image classifier and then readily deploy it using the production guide
Thanks to @dreambeats again and to @gembancud. I think I just need to keep plowing through. Thanks to your answers I’m understanding better and better. I’m just beginning lesson 2 and hope that by the end of it I can do a basic pass of my own data.
I’m pretty new to this and have been watching the first lesson.
How come the files that are in the first lesson (such as lesson1-pets.ipynb) are not in the github pull that the fast.ai website directed me in ‘Cloning fastai repository’ to:
I’m guessing it should be to the following github page instead as this does contain ‘lesson1-pets.ipynb’:
My git pull is linked to the first link that I’ve posted. Do I need to change it - if so, how?
When I tried using my own dataset (uploaded to Azure blob storage), I got an error saying it wasn’t a valid zip file. I was able to finally make it work, but I have a couple of questions about the design and expectations of fastai dataset libraries:
I notice that download_dataappends the extension to the given URL in order to download. This indicates that the URL we pass to download_data (or untar_data) should be given without the file-extension, even though the uploaded file itself has the extension. This isn’t apparent from the docs. I initially called untar_data with the full URL of file exactly as it was uploaded - but the code appended an extra ‘.tgz’ at the end, and so it couldn’t find the file to download. Why is the extension appended to the URL? I would expect the method to take in the full URL; instead it worked only when I stripped off the extension from the URL untar_data.
If the file isn’t found, shouldn’t download_url indicate failure (by throwing an exception, for example)? This didn’t happen - instead, download_url created an empty file, and tarfile.open failed when it tried to unzip the file - saying it isn’t a valid zip file. This a bit misleading, since the error had to do with the URL not being found.
-I might use it some day-
