I think this is a great suggestion @ilovescience but I always feel I get overwhelmed.
If one were to start digging through the code, where should one start? I can do just enough Python to write small scripts, so I feel like I don’t have the chops to pick up a codebase and sort of intuitively know which source file to start in which would be at the right abstraction level.
If we have a way of explaining where the model is focusing in coming to a conclusion then we can use this fact to make it more accurate. For example two breeds which are almost same but differ prominently in the shape of the nose but we detect that the model is not focusing on nose then if we give more images where nose is visible clearly then model may improve.
Last Fastai course covered heatmap for detecting the focus, but that was not too fine grained to arrive at such a specific conclusion. In case a more refined heatmap can be produced, that can be an incredible tool for improving classification of similar looking classes.
Well, its more of seeing how something is implemented. Then trying it out on your own. Chapter 17 A Neural Net from the Foundations.
Very thought provoking quesiont!
I’d go about it this way: if we look at how “human experts” differentiate between these rocks, we may find that their examination doesn’t just consist of visual comparison. Humans have multiple models running. One does visual classification, the other may compare weights (when the expert picks each equally sized rock), and the other might compare the sounds that each rock makes when they hit it with a hammer. They may break it and see how the crystal structure parts. They may then take out a magnifying glass and look at each sample a little closer.
All these features, when presented to a large enough model (or a collections of models that make parallel predictions) should give better classification. I’m a big believer of not using one giant function to do everything. Because none of the biological systems evolved this way. And there must be a reason as to why.
Looks really interesting. Earlier I found some implementation of Bayesian neural Network which used customized dropout to generate the entropy score for each prediction. Entropy is supposed to be larger for untrained class or when there is no clear winning class. However I found that the performance was not very consistent.
You’ll have to build a dataloader out of the files that you’d like to infer, assign it to the test dataloader, then ask the learner to get predictions on it.
Following your code, I’d assume this to work. You might have to run same argmax things from the dataloader vocab to get back the exact labels.
learn = load_learner("file.pkl")
png_files = get_image_files("images")
test_dl = learn.dls.test_dl(png_files)
pred_prob, pred_index, decoded = learn.get_preds(dl=test_dl, with_decoded=True)
In fact, if you dig into the learn.predict, you’ll find that it uses get_preds under the hood with a test dataloader of just the 1 item that you’ve provided. You can use that function as a guide to customize further things.
https://github.com/fastai/fastai/blob/master/fastai/learner.py#L268-L269
I really loved that this section was included in the course today. Maybe in future lectures we could have some more (short) sections that explored a this topic a bit further, esp. in terms of how easily do these models fine tune to a different dataset.
I’d really like to use something more modern for quick runs/baselines, will try using convnext family models in the coming weeks.
In case someone is following along with the questions over on aiquizzes.com, it’s a bit counterintuitive but by the end of lesson 3 you should have answered the following sets of questions (listed in settings):
I think this is because the questions refer to an older version of the course, so the match isn’t identical. Once the course is done it might be worth compiling / sorting a list of ‘which questions came up in which (2022) class’ for when the course gets released and we can pass that list on to @radek for listing somewhere on the site?
Yeah, I agree with you.
I was very impressed with the convnext on my raptor model. I’m using WSL and have a 6G GTX1660 and found I needed to drop the image down to 160 from 192 to avoid CUDA issues (cudablas was the error - but went away with smaller images…) and even on smaller images the error rate dropped from 6.6% to 3.5% for the same training set compared to resnet.
I just saw something very cool:
Its a library that can automatically optimise your models and they have a FastAI notebook example!
I am trying to recreate results for this challenge:
I got about ~ 0.87 DICE score on the validation set with Fast AI!
This repo was the winner but its not clear what these two scores mean.
Does anyone know what Image Based DICE vs Dataset Based DICE would mean?
In the titanic example the error is based on the average of loss between rows 662 and 714. I’ve found the error, and the optimisation, is quite sensitive to what rows are selected to form the average. For example, if you pick rows 21 to 541 then your optimised error is 0.14. If you add one row to the original set (i.e. 661 to 714) the error drops by 1%. If you pick the lot, then the optimised error is 0.13.
I understand the point of the exercise is to compare the error of a slightly more complex model vs a linear one, so the rows selected to calculate error need to be the same. However considering that we have a test dataset, how come we don’t use the entire training dataset to calculate training error? I assume there’s a good explanation for this, so it’s handy to ask in case someone else encounters the same doubt.
It’s also interesting how volatile the error is. The way I found this is when I was trying to play around and use the length of people’s names (as a proxy for poshness) as a factor. When you look at a boxplot of death status vs name length, it does show a relationship (longer names have a higher median and interquartile range). But when I tried incorporating it into the model my error wasn’t performing as well as Jeremy’s. It seems as if a different sample has a big influence on the training error, and that despite name length (divided by 82 which is the maximum name length, and for a bit of fun subtract the mean of the set to make it hover around 0) appearing to have a correlation with death, it just made the model perform worse. I’m assuming this is because I didn’t do the same exploratory analysis with other variables and the correlation between name length and death is weaker and drags the model down.
I welcome everyone’s thoughts!!
very cool indeed … according to the fastai example, it seems to be very straight forward to use. thanks for sharing!
that’s really neat. I’m also planning to run some tests later today on some notebook I have in place to test this. what about training time ? is it comparable too, i’d assume it to be slightly longer ?
That was a bug on my part - well spotted! Fixed now in the repo.
Search Youtube about an interview that Jeremy gave to Lex Fridman. He talks extensively about this, mainly in nontechnical terms.
Firstly, as you progress in the book, you’ll use Pyorch quite a lot. As Jeremy said elsewhere, you will actually implement a lot of fastai’s core features from scratch.
Anyway, if you are enthusiastic and impatient about it… There is a 60-minutes Pytorch blitz written by Jeremy. It’s on Pytorch’s site. The official tutorials are also good.
Our @VishnuSubramanian has actually written a book about Pytorch, search for it.
I would not recommend reading it though 
it is outdated and there are way better resources now.
