I want to train a Neural Network to classify images of rooms into 1 or 10 categories (e.g. ‘bedroom’, ‘kitchen’, ‘living room’, ‘bathroom’ etc).
I have 15000 images online but none of them are labelled, so I am building an Active Learner process.
I have manually labelled 250 images to get started. Using Resnet34 as a start via Transfer Learning, I am already getting 70% accuracy. I found 3 epochs of training got the best results, after that the validation error begins to rise or plateaus.
The idea now is to download batches of 100 images at a time, have the learner predict for each of the 100, select the 10 images it is least certain about and present me the human with those 10 images to classify. Then, incrementally learning on the newly labelled 10 images, we restart the process by downloading a new batch of 100 images, picking the 10 most uncertain, presenting to me to label etc etc.
My question is - how do I do incremental learning?
Option a) start with a fresh Resnet34 and train three epochs using the entire available labelled dataset.
This is obviously the slowest method, but seems fairly simple and fool-proof. Its not really incremental learning though.
Option b) take the current learner model, use the newly labelled 10 images to use as a new dataset.
Obviously 10 images is insufficient for a training set, a test set and validation.
Option c) mix new batch with some of previous labelled data.
For example, I randomly select 50 previous images, add in the 10 new images, and split these 60 images into a train, test, validation set. This would result in over-sampling of 50 images which the network has seen before.
Option d) Use only new data, but wait for a larger batch
Four iterations would provide 40 newly labelled images. Only execute incremental training once 40 newly labelled images are available
My questions are:
a) What is the best way to incrementally train my model? Do I risk overfitting by not using Option a? I assume there are some best-practices for Incremental training
b) How do I programatically redefine the dataset to train on my new batch, whichever method I use to select it?