I am very excited about the v3 course and the shared work here! A lot of creative approaches! 
I also played around with the STONEFLY9 Image Database consisting of 3826 images of 9 taxa of Stoneflies for training a ResNet34 and ResNet50.
This is how the data looks like:
I was able to get the accuracy up to 0.99+ and the single misclassified image is one where the insect is only shown incompletely:
ResNet34: 0.995037
ResNet50: 0.998677
These results also beat the older papers as far as I have found comparable values. 
You can find my notebook here: https://github.com/MicPie/fastai_course_v3/blob/master/L1-stonefly.ipynb
Questions, comments, and suggestions are highly appreciated!
I will now look deeper into results with the methods posted by others in this thread (PCA, activations, etc.).
Kind regards
Michael
Edit: Added a post about visualizing the activations including a notebook.