Here Lesson 1 part 1 v2 custom images u/priyal was able to do it with 40 images. I got two datasets, with 500 (comparing windsor and chinese chairs) and 700 images (oak trees with bonsai trees), both had the same problem. Dogscats was ok.
I’m also not too concerned, and moved on 
Ended up just playing with different learning rates “by hand”.
EDIT: I see that @priyal reduced batch size, I will check later if this helps