If I have a data loader that can supply an infinite number of batches per epoch using randomized data augmentation, I’d like to set the number of iterations/steps per epoch. In Keras this is done by setting steps_per_epoch parameter in fit_generator. Does fastai support this?
I was looking at keras-retinanet. It can be trained on VOC07 trainval with 5,000 images and 10,000 steps per epoch.
Thanks!
You can stop training after a given number of iterations with a Callback, yes.
My problem is that for VOC07 trainval with 5000 images and batch size being 32, each epoch goes through about 156 iterations (5000 // 32). I was looking for a way to increase the number of iterations to a large number, say 10000 per epoch.
As I think more about it, it is possible to achieve this objective by simply increasing the number of epochs. For example, with 50 epochs at 10000 iterations/epoch, training will go through 500K (50x10K) iterations. With fastai I would just use 500K/156 = 3,200 epochs.
if we have 1000 images and let say we apply 2 augmentations (crop and rotate). Can we say our training set is now consisting of 3000 images?
and therefore if we apply batch size of 16, can we say each epoch takes about 187 iterations?
(1000 original images + 2000 Augmented images)/16 = 187
I want to end the epoch and perform validation after certain number of iterations.
So that, the loss table gets formed quickly.
It takes more than 15 hours to see the first row of the table, because the dataset is huge.
Can I end the epoch after certain number of iterations? Is there any such flag, I could not find any.
Just return {'stop_epoch': True} in the event on_batch_end of your Callback.