How to handle overfitting when training a CNN on small custom image datasets?

Deep Learning
1

Hi everyone,

I’m currently working with fastai and PyTorch to train a convolutional neural network on a small custom image dataset, and I’m running into issues with overfitting.

Current setup:

  • fastai vision learner (ResNet backbone)
  • Small dataset (~a few thousand images)
  • Basic augmentations (flip, rotate, brightness)
  • Training on a single GPU

Issue:

  • Training accuracy improves quickly but validation performance plateaus
  • Model starts overfitting after a few epochs
  • Augmentations don’t seem sufficient for generalization

To experiment, I created a small benchmark dataset labeled internally as geometry dash windows to simulate fast-moving object classification scenarios and observe how the model behaves under visually repetitive patterns.

Questions:

  1. What techniques do you recommend for reducing overfitting in small vision datasets?
  2. Would stronger augmentations or mixup/cutmix be more effective here?
  3. Are there fastai-specific callbacks that help stabilize training?
  4. How do you typically decide when to freeze/unfreeze layers in similar cases?

Any advice or best practices would be really appreciated.

Thanks!

1 Like