During training, my current code will save the weights to a file if the new validation loss is better than the previous. This was done in a similar way to how Jeremy showed in the lectures and my code is shown below which initiates training.
There is a commented line which I uncomment after the first training run if I wish to continue training the same model (say for example I didn’t choose a large enough number of training epochs). The problem I have is I would like to reload not just the weights, but the entire training history. At the moment when I uncomment the line and resume training, the model doesn’t “see” the previous training history but assumes we are starting from scratch.
Is there a something in keras that is able to save the full state of the model and not just the weights?
early_stopping = EarlyStopping(monitor='val_loss', patience=7, verbose=1, mode='auto')
history = LossHistory()
checkpointer = ModelCheckpoint(filepath='tmp/weights.hdf5', verbose=1, save_best_only=True)
batch_size = 32
steps_to_take = int(len(fetcher.training_images_paths)/batch_size)
val_steps_to_take = int(len(fetcher.validation_images_paths)/batch_size)
# Uncomment below line when we want to reload and resume training
# model = load_model('tmp/weights.hdf5')
hist = model.fit_generator(BatchGenerator(fetcher),
samples_per_epoch=steps_to_take,
nb_epoch=10,
validation_data=ValBatchGenerator(fetcher),
nb_val_samples=val_steps_to_take,
verbose=2,
callbacks=[history,checkpointer,early_stopping],
)