Lesson 2: My predictions are not accurate

Part 1 2022
1

Hello All,

Use case: I have a web application for K-3 activities. I tend to break UI of one activity or other and testing UI is very tedious – requires eyeballing {Ido have way to generate automatically all screenshots though}

The aim is to let AI handle it and I was thinking of classifying the UI as “BrokenUI” or “OKUI”.

Current status: We have gone through Lesson-0 to Lesson-2 video and corresponding book chapters. The kaggle notebook is BrokenUI | Kaggle

After following the “Lesson-1, 2” steps, I loop through all the images in my DataSet and print whether “BrokenUI” or “OKUI”. By eyeballing we have determined whether UI is broken or not and renamed the images with “Broken” as “BD” and those with “OKUI” as starting with “D”

The predictions are mostly inaccurate and the loss/error_rate are as below
error_rate

Am I on the right track and should go through the next chapters to see improvements or am I on the wrong path? Any guidance/comments are appreciated.

Thanks

2

This sounds like a great real-life problem to work through as you are working through the course! A few observations after looking at your notebook:

  • resnet18 is not a great model, try resnet34 or import timm (which is a large collection of PyTorch models pretrained on image data) and try some of the convnext models. To use timm models you pass the string of the architecture to the vision_learner. For example if you are using the resnet50 model from timm you would write vision_learner(dls, 'resnet50').
  • Your validation loss is increasing each epoch which is a sign that your model is overfitting (or “memorizing”) your training data. One reason could be that there is not enough data. You could also try different data augmentations (which is introduced in chapter 2) by passing batch_tfms=aug_transforms() to your DataLoaders.
  • Once you get a model that’s not overfitting, you can try passing different item_tfms to the DataLoaders object and see if that makes a difference. For example does Resize(224, "squish") work better than Resize(224)? And so on. Jeremy has created a fantastic series of four notebooks where he works through an image classification problem from start to finish—I recommend following a similar approach.

If you can make your dataset public, others can experiment with it in your notebook and give you more suggestions.

3

Thankyou vbakshi for your response and suggestions. Will try them and get back with progress…

Dataset public — I would see if I can get permissions.

1 Like
4

Hi!

Add some weight decay, pass a key argument ‘wd=0.1’ into vision_learner.

5

Hello MIke,

I will try what you suggested and will keep you posted.

Thanks.

6

Using resnet34 and wd 0.1 etc. did not give better results.

I looked at my Training set and then changed it.
Now with resnet18, wd = 0, I get 100% accuracy.

I hope it is not the case of model learning everything as JH mentions in Chapter-1. I will know after some days after we have new screenshots.

Thanks for your response – it forced me to do somethings and helped learning

1 Like
7

Hi KBSharma74,

First of all, kudos to you for identifying the importance of data quality and its impact on model performance. It’s great to see that you took the initiative to improve your dataset and achieved 100% accuracy on your training set.

However, as you mentioned, there’s a risk of overfitting when a model learns everything from the training data. To verify whether this is the case and ensure your model generalizes well to unseen data, consider the following:

1. Test on a similar dataset:

If possible, obtain a dataset similar to the one you’re not allowed to share. Testing your model on this dataset will provide a more accurate assessment of its generalizability.

2. Analyze the types of errors:

Even if your model achieves high accuracy, it’s important to analyze the types of errors it makes. Are there specific types of images that it consistently misclassifies? Understanding these errors can help you identify potential biases or limitations in your model.

3. Consider the acceptable margin of error:

In most real-world applications, achieving 100% accuracy is not always necessary or even feasible. Instead, focus on determining an acceptable margin of error for your specific use case.

4. Don’t get stuck:

If you find yourself stuck on a particular problem, it can be helpful to take a break and work on something else. Sometimes, coming back to the problem with a fresh perspective can lead to new insights and solutions.

Finally, remember that machine learning is an iterative process. It’s normal to encounter challenges and setbacks along the way. Keep learning, experimenting, and iterating on your approach, and you’ll eventually find the best solution for your specific needs.

The pleasure was all mine. I appreciate the chance to share my thoughts. I gained some insights into my path while answering your post.

Best of luck!