This is interesting. A friend of mine told me a startup doing exactly this, and it is getting global biz. This is a legit practical problem. As forgers are very good in faking product, your 37% may even beat lot of human, including me. So 99% probably not a benchmark to beat for now. I suspect you have to get more data, and try to get multiple shots on the same bag from different angle, and emphasizing diff parts, and average the predictions.
1 Like
I am working on a project that involved counting, but nothing to do with fastai lesson. I am a bit confused since this looks to be an image recognition problem, and so wonder if you are labelling your images as “49”, “50”, “51”? But this will limit the range of object you can count. For my case, I have to use object detection model (which i think will be part 2 fastai)? You may have seen some videos such as yolo, ssd, etc. that draw bounding boxes around objects in some self-driving tech promo? a side product obviously you can count the objects. Potentially, you are not limited to the range of number of things you can count, up to a certain point.
Hi Cory,
Your project is really interesting. Could tell us more about how you created the dataset ?
I made a small notebook which determines whether a knife is present or not in an image. I wrote up a blog post on my method: https://hackernoon.com/recognising-a-knife-in-an-image-with-machine-learning-c7479f80525
I’d also be interested in how many images. I have a classifier that works across 16 sports, and I also had an ‘other’ option for non sports images. I had between 300 and 700 images per sport, depending on availability, and found that image size made a difference and ended up using 448x448 to get the detail needed to differentiate. Still I get around 95% accuracy. I’d expect two class baseball v football to easily get close to 100%. https://sportsidentifier.azurewebsites.net/ Quidditch was the last sport I added to my list.
1 Like
I think this is just the row,column nature of tensors (and matrices generally) compared to the column,row (x,y) approach in images.
Thank you for your comments.
- I originally ran 4 epochs with either default image size or image size of 224 and of course started thinking of simple ways to improve upon the results. From (data.show_batch), I saw that a number of the images were cropped to a level that was hiding large important sections of the birds, so I increased the size to 384 and results improved 4-5%. Need to continue to research to make improvements.
- In the initial stages, I was looking to just get an output and keep things simple, so did not try (.normalize(imagenet_stats)) or additional functionality that FastAI offers. Will continue to explore.
- Agree 100% with your statement regarding the researchers train_test_sets. Again, initial hope for me was to just get reasonable predictions and was surprised to how close (+/- 1-2%) to state-of-the-art my initial attempt performed. Making this comparison is my next project.
1 Like
@kechan - Thank you for your comments. I in no way wish to diminish the real work that the researchers performed.
My understanding is that FastAI’s accuracy rate is calculated on test holdout set that the trainer has never seen. So if the researchers performed their train_test splits properly and FastAI’s random test split is sound, then I believe the results should be fairly solid. However, you are absolutely correct that using the same test set to ensure that one of us is not working with an outlier test set is in order. This my next project.
Happy to answer any questions about my approach, but it is basically all in the notebook. I ran with resnet34 2-3 times, ran 2-3 times with different image sizes, and finally increased the number of epochs as hold out accuracy was still decreasing.
It wasn’t very sophisticated. First I found this LV fan page on instagram (just a warning, the ostentatiousness on display is disturbing). Whoever runs the account takes real vs fake seriously, so they’re only reposting legit products. Then I looked for replica hashtags. Often they were stores posting bags/wallets, when it was obvious they were posting their own photos (walls, countertops, backgrounds give this away) I used the photos for the fake category. I took screen shots, which is why the dataset is so small… it’s time consuming.
The biggest issue is being sure that a replica/fake is in fact a fake and not a real bag image to give the impression of a higher quality fake. This is the main reason I didn’t automate the process, I felt like I needed to be sure what was labeled fake was in fact fake. Here’s a dropbox link to the images I used if you’re interested.
FWIW, if I were to take this project seriously I wouldn’t use instagram as a source. It was either in V2 or V3 of the class where Jermey said that going through the process sorta turns you into a domain expert even if you had zero prior knowledge of the subject (he used dog/cat breeds as an example), that’s 100% true. Now that I know a lot about the counterfeit market I realize that instagram isn’t where the majority of buying/selling takes place.
3 Likes
I agree more data is likely the answer. Sourcing the counterfeit bag images is probably the hardest part of the problem. Lot’s of sellers of fake products use legit images. One idea I had to ensure that the images were counterfeit products was to find pictures used as evidence in cases brought against counterfeiters (in theory these should be available). Maybe someone with a legal background would have more success than me. I found it to be a dead end. I think one way of getting some quality images would be to put together an app that predicts if a bag is fake or not using a simple model like the one I came up with and have users upload pictures of their bags and see the quality of the fake. These new images can be used to further train the model.
Thanks for replying, I was really interested in your data collection process because you obviously have to be an expert to be able to tell apart the real thing and fakes, as counterfeiters have become pretty good.
And to the best of my knowledge, I don’t think the luxury industry has found a way yet to protect customers and their brands against counterfeits though they’re exploring things such as blockchain.
It’s definitely a cool project and if you were to continue, I would be happy to brainstorm with you about this.
FYI you’re under-fitting, so should train for longer.
That’s not quite what you said on twitter - what @sandhya says here is correct however. But more importantly, what @sandhya has done here is much more respectful of the work that @JCastle did.
Can you tell the difference from the photos? (If it’s a task that humans can’t do from your data, then generally a computer won’t be able to either, so you want some kind of human baseline to compare to.)
1 Like
That “downplaying” apology applies to me, you are doing the hard work here as well and I am just a bystander.
The problem is not so simple. There’s a 3 way split as far as I know. Train, validation and test set.
Train - seen by model and used to adjust weight
Validation - not seen by model and used to continuous evaluate progress, but lt is seen by you, the human decision maker.
Test - not seen by model, not used during training in any manner, not even printing out its loss or accuracy. It is used only at the end running only once. Ideally, this data is hidden under a rock until your finish all your work.
This is really the most honest way with probably no chance of accidental overfitting. And of course, this is how Kaggle work as well. You don’t see their test set while u work, and only get a number reported back to you.
This point may be explained further in fastai, I am not sure. But if not, you can easily google it up.
Having said all this, I get JH point. There’s a good chance you achieved equal or higher accuracy than the paper, cos u aren’t tuning the hyperparam like mad, only very mildly by LR or epoch num if so. But I would be careful if the claim is compared to a research paper, and then mentioning it on Twitter, then the result will have to be defended more. I perfectly understand we are all just learning, I just shared this cos I ran into bad things before by leaving out work.
The post you’ve cited is very interesting: I definitely want to take a closer look at it! I agree with the author that having “unkonwn” people in the validation set let you better measure how the model generalizes.
My notebook is an “initial trial” / “starter kit” for a generic object classification competition in kaggle with fast.ai, not an attempt to win it.
Take a look at your data augmentation:
In this sample you’re probably using a wrong label because the visible dots are less than 45.
I naively knew sourcing the data this is hard, but after hearing your experience, it looks even harder. Making an app to serve customers and then collect the data is probably the way, assuming you have no deep connection in that industry. I had thought about a cannabis identifying app, and u bet it’s not so easy to collect large diverse dataset, before legal trouble, even if you live in Canada. 
I googled sativa and some well know brand, and I already got discouraged. This is another area you probably need some connections with the right biz ppl. For counterfeit bags, it will help if you have a personal collection, and just take hell lot of photos.
I can’t tell the difference between a good knock off and an original. However, it seems that there are people who can. Often a fashion blogger/account will post a “can you tell the real from the fake” like this one the majority of people who comment are able to spot the fake, so I assume it has to be possible. Sending one of these bloggers the same images and seeing if they can beat the classifier would be an interesting exercise.
If you have time, take a quick look at course 3 of Andrew Ng Deep Learning specialization on coursera. He had a very in-depth discussion on human baseline, Bayes error, and super human performance. He used radiologists as a discussion case,