Does anyone know of any research into single-epoch or low epoch training with small batch sizes(1-10)?
I was thinking through what would be required to build a labeling tool which trains a model in the background as data is annotated. The labeling tool could could suggest a label to the annotator or order the unlabeled examples by confidence and suggest perplexing examples. This would allow the annotator to quickly build a dataset and a model without needing a data scientist in the loop.
There are some technical challenges here that come to mind though.
Would the model need to be retrained from scratch every time a new image(or set of images) are labeled? Typically, a dataset is fully constructed and then a model is trained for multiple epochs on the entire dataset. Ideally this training could happen on a cpu to allow this tool to be used with arbitrary hardware. Has any research been done on training a model where it is only allowed to look at each image once and where dataset statistics cannot be computed ahead of training?
Are there any other technical challenges to such a tool being developed? Does anyone know if this is being done by any labeling tools?