I am following this tutorial:
and I am rewriting some of the code on Google Colab.
They are using the following for a learning rate decay:
initial_lr = 0.096505
learning_decay_rate = 0.7
lr_schedule = tf.compat.v1.train.exponential_decay(
learning_rate = initial_lr,
global_step = tf.compat.v1.train.get_global_step(),
decay_steps = checkpoint_steps,
decay_rate = learning_decay_rate,
staircase = True)
…and the following modell is which I need to rebuild:
estimator = tf.estimator.DNNRegressor(
feature_columns = dnn_features,
hidden_units = [128, 64, 32, 16],
config = tf.estimator.RunConfig(
save_checkpoints_steps = checkpoint_steps),
model_dir = model_dir,
batch_norm = True,
dropout = 0.843251,
optimizer = tfa.optimizers.ProximalAdagrad(
learning_rate = lr_schedule,
l1_regularization_strength = 0.0026019,
l2_regularization_strength = 0.0107146))
tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
I cannot run the model like this, because I get a
ValueError: None values not supported.
…and the reason is the function get_global_step. My results are pretty bad compared to theirs, when I use i.e.:
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(...)
My questions are:
- What exactly is the global_step?
- Is it crucial for the model to get better?
- In case that I need it: How can I make it work like this?