Not enough training data for simple three color classification?

I have made some training images of SET cards and am just trying to classify color to start, using techniques from the first lesson of DL for coders. I notice two things: 1) I don’t get very good error rates - usually 30% or so. It seems plenty simple, so I feel like this SHOULD work. 2) The error rates vary a lot. Perhaps both of these things imply a poor training set? There are about 50 images. Given some discussion in the lesson about 30 images for cricket, it doesn’t seem an unreasonable size. But perhaps I need to hand craft the training/validation images? The original data and notebooks are here:

But I figured perhaps the Kaggle Kernel was an easier method of sharing. I’ve experimented with LR and such with no luck, but from the discussion in the video, I would think I would have better luck with “vanilla” code than I’m having, leaving only the training data as a suspect.

Thanks for any help!