Here is a great crowd sourcing site in partnership with Harvard business school and Boston Children’s (unfortunately US info only)
4 Likes
I don’t think I’ve seen a counting task being done with either classification or regression.
The way I’d frame this would be as an object detection problem where the model puts a bounding box around each subject. And then you count the number of bounding boxes.
6 Likes
Just want to point out that currently we are still discussing chapter 1, which you can read here
1 Like
It’s a particular type of regression, because the number of cats is an integer while normally regression problems deal with continuous variables, but it’s still regression.
You could make this into a classification problem if you were to label examples as “one”, “two”, “three”, “many” and then teach the model to recognize those situations as classes.
2 Likes
Here’s a post I wrote on validation sets: How (and why) to create a good validation set
11 Likes
If there is overfitting, I guess there must be underfitting as well. I guess that means the the predictions are not accurate enough. Is that right?
I’ve heard people describe overfitting as training error being below validation error. Does this rule of thumb end up being roughly the same as yours (validation error increasing)?
5 Likes
Is k-fold cross-validation a way to avoid validation set overfitting? Should it be used systematically if yes?
1 Like
Yes, under-fitting just means that your predictions are not as accurate as they could be. “accurate enough” is subjective.
Yep underfitting also exists and is defined as:
when a statistical model or machine learning algorithm cannot adequately capture the underlying structure of the data
Typing Jeremy’s response as this is important. As you start overfitting, your training loss will continue to get better, but your validation loss will start to go up. It won’t start to get better, it will start to go worse. As we’ll see, this is not necessarily a bad sign in itself. The important thing to look at is your metric getting worse, not your loss function getting worse.
23 Likes
It is a good technique for that, but it’s also expensive to run. If your model takes days to train, training 10 times is going to be painful. So in my opinion you have to use your judgement on when to use it. But if you have the possibility and it’s not too painful, then it’s probably a good idea to run it.
3 Likes
the reverse question – if training loss is more than validation loss – is the model underfitting ?
1 Like
I am bit confused on the differences between loss/error/metric. Can anyone explain it here. Thanks
1 Like
Not necessarily, it will most likely be the case if you use the techniaues against overfitting we will describe in the course.
Not necessarily. There is more regularization applied to the training vs validation data so the training loss can often be worse than your validation loss.
1 Like
How do you know when you have overfitting? Jeremy mentioned that validation “loss” will start to go up But what is that in comparison to? Are we comparing across epochs?
Separately, how do you figere out what is causing the overfitting (e.g. # of layers, parameters, epochs or something else?)
1 Like
I’m confused regarding the pre trained model vs a new model, when we selected a resnet34 in fast AI is it already a model that has been trained in data or how do we know when a model has been trained or is a new model?
1 Like
When I was in grad school I made the exact mistake Jeremy alluded to where 20% of data is randomly removed from a time series dataset. I gave presentations on what I did and nobody noticed. Eventually I figured out my mistake and my metrics got way worse. Lots of people make the same mistake and I suspect a large fraction of grad research falls into this trap. Proper validations sets are still not well understood broadly. So yes, be very careful with your validation set.
9 Likes
Note what Jeremy said about the separate test set at the end. You’d want to compare how the overall validation accuracy was on the moving validation (the training data) and look at the test data and compare their results.
2 Likes