Lesson 1 official topic

sswam · October 13, 2022, 2:58pm

I want to respectfully give the feedback that I don’t like the initial “bird detector” example in the 2022 part 1 course.

The XKCD cartoon asks us to “check whether the photo is of a bird”. The notebook says that “today, we can do exactly that, in just a few minutes”, but the model we train doesn’t come anywhere close to meeting that goal.

For example, I tested it on an image of a cow, and it told me it was likely a bird. This makes sense, as a cow is more similar to a bird than to a forest, but that’s not a very reliable bird detector! I certainly wouldn’t trust any similar model to detect potentially dangerous creatures such as bears.

I spent a few hours trying to train a more reliable “bird vs anything else” detector, and it wasn’t trivially easy, and I didn’t succeed to my satisfaction. My last attempt gave 60% false positives when I tested it against a collection of 100 random animals, so I think it has a ways to go yet. I don’t think it would take five years and a research team, but I don’t think we can do it in just a few minutes either, at least not using this method.

I did like the initial examples from previous versions of the course, especially the breeds of cats and dogs example, as it was very impressive and I can’t distinguish all those breeds myself. Those classification examples had a limited domain of acceptable input photos, and they worked well.

bencoman · October 13, 2022, 3:54pm

Hey Sam, Thanks for sharing your experience, and sorry to hear its been less than great. Can you share your notebook? …in case the community can spot how to help you improve your result. Respectfully, there is a reasonable chance you are doing something wrong.

To share my own experience (and btw this is my first exposure to Deep Learning), after only Lesson 2 I created “WhichWatersport - a Hugging Face Space by bencoman”. Please check it out! And let me know what you think. Months later, It still blows me away how accurately the model distinguishes between the 37 watersports I trained it on. These are much similar than a bird and a forest. In particular, the six divers are very similar. See if you can guess them before you get the model to do it.

btw, I notice you say “tested it against a collection of 100 random animals” but you don’t indicate how you “trained” the model to distinguish those 100 animals. Can show how you did that training?
If that training didn’t happen, then there may be a gap in your understanding the learning/inference process. So lets see if we can work out what is going wrong for yours.

hobby · October 14, 2022, 2:14am

When I was preparing for the first chapter of the course, several other examples (forest and bird, tables, etc.) were able to be trained, but this collaborative filtering example was not able to be completed. It has been updated to the night version of pytorch and it still doesn’t work. Does anyone know why?

It’s my code,named test.py

import os
os.environ['OMP_NUM_THREADS'] = "1" #for m1 mac

from fastcore.all import *
from fastai.vision.all import *


from fastai.collab import *

path = untar_data(URLs.ML_SAMPLE)
dls = CollabDataLoaders.from_csv(path/'ratings.csv')

learn = collab_learner(dls,y_range=(0.5,5.5))
learn.fine_tune(10)

Output

python test.py
epoch     train_loss  valid_loss  time
-:8:10: error: invalid input tensor shape: updates tensor shape and data tensor shape must match along inner dimensions% [0/75 00:00<?]
-:8:10: note: see current operation: %5 = "mps.scatter_nd"(%0, %arg0, %4) {batch_dims = 0 : ui32, mode = 0 : i32} : (tensor<101x1xf32>, tensor<64x50xf32>, tensor<64x1xi64>) -> tensor<101x1xf32>
/AppleInternal/Library/BuildRoots/a0876c02-1788-11ed-b9c4-96898e02b808/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphExecutable.mm:1267: failed assertion `Error: MLIR pass manager failed'
zsh: abort      python test.py

jeremy · October 14, 2022, 2:31am

If you replace the softmax with a sigmoid it would do what you want, but the goal here was to keep things simple, rather than be perfectly accurate. I think you may have taken things a little too literally!

sswam · October 14, 2022, 2:33pm

Hi Jeremy, I’m sincerely a great fan of your work, and I love these courses, and I don’t want to annoy you or waste your time for no good reason, but I don’t think I’m taking things too literally here.

The XKCD cartoon says “check whether the photo is of a bird”. That’s the problem statement. In the notebook you say “we can do exactly that, in just a few minutes”, and you conclude that creating classification models is “trivially easy”, but the model we train doesn’t know the difference between a cow and a bird.

In my testing it misidentifies all sorts of things as birds with >99% confidence, including a butterfly, a lion, camels in the desert, two platypus, two wolves, a cow, and a chair. It’s very far from perfectly accurate.

Here are four photos which the model says are birds with 100% confidence (to 4 significant figures):

When you think about it, we’ve trained it symmetrically so it’s either going to say that an image is a bird or a forest, and it’s going to be wrong one way or the other for any other image. The “forest” images were I guess more similar than the “bird” images and had more “significant pixels”, so it tends to guess that anything other than a forest is a bird.

All of the previous examples were good (cat and dog breeds, cat vs dog, black/brown/teddy) because they did what they set out to do. But I think that the “bird” vs “not bird” problem is not a good fit for an introductory image classifier. It seems to be more difficult than “dog” vs “cat”, because “not bird” might include all sorts of different things.

I guess you probably know some way to fix it so it will do a good job of detecting birds while keeping things simple enough for an introductory example, and I’d like to see it. I don’t understand how to fix it by using sigmoid as you suggested. Do you mean that we should require a higher confidence threshold before saying that something is a bird?

Here’s a little gallery with some more test images for the original model. Out of all the photos I tested, it thinks everything is a bird except for a “forest” picture, so it might be more accurate to call it a “forest” detector or a “green” detector. I didn’t exclude any photos were it got it right, to make it look bad.

Things in my test gallery that aren’t birds which it misidentifies as birds, with percent confidence:

butterfly: 100%
a colourful lion picture: 100%
desert with camels: 100%
two platypus: 100%, 99.98%
two wolves: 99.98%, 99.70%
two chairs: 99.83%, 98%
two cows: 99.42%, 92.45%
empty desert: 94.43%
a lion: 71.23%

suvash · October 18, 2022, 6:16pm

Hey Sam, perfectly reasonable question. If you go further in the course (and the book chapters), you’ll grok a bit more about how to implement what Jeremy has mentioned above. It depends a lot on what your training+validation dataset looks like and how your output losses are set up.

In the book, Chapter 6 - Multi label classification to be exact. Let us know how it goes.

sswam · October 22, 2022, 6:56pm

I found that the infamous FileUpload widget does not work, in the newest ipywidgets (8.0.2). I did a git bisect and made a comment on a bug report here.

A workaround is to install an older version, e.g 7.7.1 which is the default on colab:

!pip install ipywidgets==7.7.1

bencoman · October 23, 2022, 3:38am

Awesome work to discover this. Its a long topic here though. If you know which items in the wiki at the top that this applies to, it might be good to add a link there referring to your post, that can be modified again once the issue has been addressed in ipywidgets latest release.

jeremy · October 23, 2022, 5:21am

@willingc is this something you might have any visibility into?

willingc · October 24, 2022, 4:55am

Hi @jeremy, @sswam, @bencoman,

ipywidgets 8.x is significantly different from 7.x. If using classic jupyter, collab, or nteract, the 7.x series has better support.

Here’s a migration guide from 7 to 8: Migrating user code — Jupyter Widgets 8.0.2 documentation

For my everyday work, I am still using 7.x.

jeremy · October 24, 2022, 5:21am

Thanks for explaining @willingc . I guess our book is broken now Oh well! I’ll pin the package in the fastbook metapackage so at least anyone using that will have it continue to work.

aditya.swami · October 24, 2022, 7:30am

Hey folks,

I’ve tried to build a simple classifier from Lesson 1 and most of it works as intended.

I extend the bird classifier to a couple more

Wallaby vs Kangaroo (both very similar in appearance)
The error rate increased with training. I wonder if this is because the images are too similar. For a model that took a few seconds to train it’s amazing because I couldn’t tell the difference between the two animals most of the time. (Kangaroos are taller)

Wallaby vs Forest
This trained like the model in the lecture.

I got tripped up a little bit in the learn.predict section.

is_wallaby,_,probs = learn.predict(PILImage.create('wallaby.jpg'))
print(f"This is a: {is_wallaby}.")
print(f"Probability it's a wallaby: {probs[0]:.4f}")

This is a: wallaby.
Probability it’s a wallaby: 0.0000

When I examined the tuple

learn.predict(PILImage.create('wallaby.jpg'))

I get,
(‘wallaby’, tensor(1), tensor([2.9118e-05, 9.9997e-01]))

Wondering if the order of the folders (alphabetical) is the same order as the probabilities in the tensor.

In which case shouldn’t it be

print(f"Probability it's a wallaby: {probs[1]:.4f}

Thanks!
Adi

benkarr · October 24, 2022, 11:37am

Yes, at least by default. To be shure, you can check the order of your labels with learn.dls.vocab, the order here is the same as in the probabilities. To get the index of a particular label you could also call learn.dls.vocab.o2i['wallaby'], so:

label='wallaby'
label_idx = learn.dls.vocab.o2i[label]
print(f"Probability it's a {label}: {probs[label_idx]:.4f}")

should make it failsafe

intrtz · October 24, 2022, 2:01pm

I’m having problems with the book.

When I try to run the first cell I get the following error message. The same happens both in jupyter and when running Jupyter through Visual Studio Code.

#hide
! [ -e /content ] && pip install -Uqq fastbook
import fastbook
fastbook.setup_book()

'[' is not recognized as an internal or external command,
operable program or batch file.

bencoman · October 24, 2022, 2:19pm

It works for me in Jupyter Lab…

and Jupyter Notebook…

This is running on WSL in this environment…
(base) ben@OFI-PC-0004:~/jupyter$ jupyter --version
Selected Jupyter core packages…
IPython : 8.5.0
ipykernel : 6.15.3
ipywidgets : 8.0.2
jupyter_client : 7.3.5
jupyter_core : 4.11.1
jupyter_server : 1.18.1
jupyterlab : 3.4.7
nbclient : 0.6.8
nbconvert : 7.0.0
nbformat : 5.5.0
notebook : 6.4.12
qtconsole : 5.3.2
traitlets : 5.4.0

(base) ben@OFI-PC-0004:~/jupyter$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION=“Ubuntu 20.04.4 LTS”

What is your environment?

btw, Have you confirmed the same works from your shell command line?

stewart1981 · October 24, 2022, 4:51pm

Sorry if posting in the wrong area.

I seem to be having issues with either Kaggle or duckduck_search.

I am trying to build a model on Kaggle and here is the issue I am facing. Any help will be appreciated. I have tried Google but that just links back to these tutorials:

stewart1981 · October 24, 2022, 5:29pm

Never mind, found the function declaration for search_images.

aditya.swami · October 24, 2022, 10:21pm

Thanks a lot @benkarr!

That does make it fail-safe. I learnt a few more little functions along the way which is fantastic.

Appreciate your help!

silience · October 25, 2022, 8:18am

This chunk of code in from Lesson 1 is showing an error. Do we have to create a directory somewhere?

searches = 'forest', 'bird'
path = Path('bird_or_not')

if not path.exists():
	for o in searches:
		dest = path/o
		dest.mkdir(exist_ok = True)
		results = search_images_ddg(f'{o} photo')
		download_images(dest, urls = results[:200])
		resize_images(dest, max_size = 400, dest = dest)

The error is this:

intrtz · October 25, 2022, 8:30am

Thanks that helped me to fix the issue, now I’m having another one with just the formatting.

edit: nevermind, I found in another post that it’s supposed to be incorrectly formatted when viewed in the notebook, only the book has correct formatting.

There a cell with [[chapter intro]] where the square brackets just stay there and then under the headline Deep Learning Is for Everyone it says [[myths]] and I guess there’s supposed to be a table there. Above this I see it says asciidoc, so it looks like maybe asciidoc is not setup correctly or something, but Googling this I only find converter between jupyter and asciidoc files