This is a strange occurance I get on the StateFarm sample.
Since the learning rate starts off low, and after a couple of epochs, is set high again, I decided to use Keras’ LearningRateScheduler to automate it as follows:
def fit_model(model, epochs=1, lr=1e-3):
callbacks = []
if type(lr) is list:
lr_schedule = np.hstack([np.repeat(lr[i], epochs[i])
for i in range(len(lr))])
callbacks.append(LearningRateScheduler(lambda epoch: lr_schedule[epoch]))
else:
model.optimizer.lr = lr
return model.fit_generator(train_batches, train_batches.num_batches,
epochs=np.sum(epochs), verbose=0,
callbacks=callbacks, validation_data=valid_batches,
validation_steps=valid_batches.num_batches)
Now, if I do it the normal way (like it was done in class), in the simple linear model,
fit_model(model, 2, 1e-5)
fit_model(model, 4, 1e-3)
I get more or less what is expected:
Now, interestingly, if I use the LearningRateScheduler,
fit_model(model, [2, 4], [1e-5, 1e-3])
I get something wierd!
The accuracy goes up, like previously for the low learning rate (epochs 0 and 1).
As soon as the Scheduler changes the rate to 1e-3, the accuracy inexplicably dips.
It’s as if the model resets itself when the learning rate changes.
But, as I mentioned, if I do the same thing manually, I get the expected behavior.
Any ideas would be greatly appreciated.
(P.S. I’m using Keras 2.0.8 with TF 1.3.0 backend.)