Question of tensorflow : How could I turn is_training of batchnorm to False

Deep Learning
1 
net = tf.layers.conv2d(inputs = features, filters = 64, kernel_size = [3, 3], strides = (2, 2), padding = 'same')
net = tf.contrib.layers.batch_norm(net, is_training = True)
net = tf.nn.relu(net)
net = tf.reshape(net, [-1, 64 * 7 * 7]) #
net = tf.layers.dense(inputs = net, units = class_num, kernel_initializer = tf.contrib.layers.xavier_initializer(), name = 'regression_output')

#......
#after training

saver = tf.train.Saver()
saver.save(sess, 'reshape_final.ckpt')
tf.train.write_graph(sess.graph.as_graph_def(), "", 'graph_final.pb')

How could I turn the is_training of the batchnorm to False after I save it?
I try the keywords like “tensorflow batchnorm turn off training”, “tensorflow change state”, but cannot find out how to do it.

ps : dnn module of opencv cannot load the model if is_training is True, I have to change it back to False and save the model again.

2

You can use a placeholder for the is_training parameter and then you use the feed_dict mechanism to turn training on or off
training = tf.placeholder(tf.bool, name='training')
_ = session.run(my_op, feed_dict={training: True})

1 Like
3

Thanks, I set the “training” and “is_training” to false in for loop(for tensor in tf.trainable_variables()), how could I validate the training flag is turn off already?

with tf.Session() as sess:
        saver = tf.train.import_meta_graph('reshape_final.ckpt.meta')
        inference_graph = tf.get_default_graph()
        saver.restore(sess, 'reshape_final.ckpt')
    
        training = tf.placeholder(tf.bool, name = 'training')
        is_training = tf.placeholder(tf.bool, name = 'is_training')
        for tensor in tf.trainable_variables():
            if tensor.name.find('BatchNorm') == 0:
                print("1,", tensor.name, tensor.name.find('BatchNorm'))    
                sess.run(tensor.name, feed_dict={training:False})
                sess.run(tensor.name, feed_dict={is_training:False})        
        
        #batch norm layer still print out
        for tensor in tf.trainable_variables():
            print("2,", tensor)
4

I’m not sure what you’re doing there. The common way to set training on or off is as follows.

When you are training you do
sess.run(train_op, feed_dict={training:True})

When you do inference call
sess.run(prediction, feed_dict={training:False})

Have a look at https://www.tensorflow.org/get_started/mnist/pros they set dropout rate differently when training and just doing inference. That’s the same idea.

5

According to your answer, I guess I should feed place holder to the batchnorm layer instead of pure bool value?

training = tf.placeholder(tf.bool, name='training')
net = tf.contrib.layers.batch_norm(net, is_training = training)

What if I did not feed the placeholder into batch_norm layer before(I set it with bool value directly)? Could I change the training state to false after the model trained without placeholder(The model is trained and save without the training placeholder)

6

If you did not use the place holder before, setting the value directly should work. Have a look at https://www.tensorflow.org/programmers_guide/debugger to check the runtime value of is_training.

1 Like
7

Thanks, I do a small experiment with the solution of placeholder

    input_shape = [None,128,128,2]
    features = tf.placeholder(tf.float32, input_shape, name = 'input')
    net = tf.layers.conv2d(inputs = features, filters = 64, kernel_size = [3, 3], strides = (2, 2), padding = 'same')
    training = tf.placeholder(tf.bool, name='training')
    net = tf.contrib.layers.batch_norm(net, is_training = training)
    net = tf.nn.relu(net)
    net = tf.reshape(net, [-1, 64 * 64 * 64])
    net = tf.layers.dense(inputs = net, units = 8, kernel_initializer = tf.contrib.layers.xavier_initializer(), name = 'regression_output')

    x = tf.placeholder(tf.float32, shape=[None,8])
    #apply sqrt to avoid the loss value bloated
    loss = tf.sqrt((tf.nn.l2_loss(net - x)))

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        imgs = np.zeros([1,128,128,2])
        labels = [[1,2,3,4,5,6,7,8]]
        sess.run(loss, feed_dict = {features : imgs, training : False, x : labels})
        
        saver = tf.train.Saver()
        saver.save(sess, 'reshape_final.ckpt')        
        tf.train.write_graph(sess.graph.as_graph_def(), "", 'graph_final.pb')

After that, I freeze->optimize->transform the model by following commands

python3 ~/.keras2/lib/python3.5/site-packages/tensorflow/python/tools/freeze_graph.py --input_graph=graph_final.pb --input_checkpoint=reshape_final.ckpt --output_graph=frozen_graph.pb --output_node_names=regression_output/BiasAdd

python3 ~/.keras2/lib/python3.5/site-packages/tensorflow/python/tools/optimize_for_inference.py --input frozen_graph.pb --output opt_graph.pb --frozen_graph True --input_names input --output_names regression_output/BiasAdd

~/Qt/3rdLibs/tensorflow/bazel-bin/tensorflow/tools/graph_transforms/transform_graph --in_graph=opt_graph.pb --out_graph=fused_graph.pb --inputs=input --outputs=regression_output/BiasAdd --transforms="fold_constants sort_by_execution_order"

Last command give me error:

“You must feed a value for placeholder tensor ‘training’ with dtype bool”

I already feed bool value at this line

sess.run(loss, feed_dict = {features : imgs, training : False, x : labels})

What should I do?