ResNet50 NaN predictions

Part 1 (2017)
1

Hello,
I’ve been trying to finetune the resnet50 model and have issues when using it to predict.
Only thing i changed in the model is remove the Flatten() layer and replace it with a GlobalAveragePolling() one.
I then trained the new model on some sample data (10k images) for 60 epochs and got to 0.92 training acc and 0.70 val acc ( i know it’s overfitting and I’ll deal with this at some later point).

Problem is when I’m trying to use the model to predict, i get NaN for ALL predictions. At first i though it’s some issue with what I’ve done when finetuning it… but the original ResNet50 with the weights from ImageNet still gives NaN predictions.

Is this something anyone else encountered? Help much appreciated.

Thanks!

2

Sounds like your prediction code has a problem. The first thing I would check is if you’re preprocessing the images in the same way as you’re doing for training.

3 
img_input = Input(shape=input_shape)
x = Lambda(self.vgg_preprocess)(img_input)

This is the first layer in the model… I am instantiating the model from the same class when using it for predict, so I’m pretty sure I am preprocessing the images in the exact same way.

calling predict code:
img = image.img_to_array(image.load_img(path_img, target_size=[195, 283])).reshape(1, 3, 195, 283)
preds = self.resnet.model.predict(img)
logging.critical(str(preds))

I am doing the exact same thing with a vgg16 model and it works… Only difference is that in vgg i use the predict_proba() function which is only available in the keras.models.Sequential class.

4

So… the exact same code runs fine on a GPU machine with tensorflow-gpu as backend and not in a docker env with tensorflow-cpu…

5

Just a small thing, but:

reshape(1, 3, 195, 283)

Shouldn’t that be:

reshape(1, 195, 283, 3)

if you’re using TensorFlow as the backend (as opposed to Theano)?

6

@machinethink I think this is handled by setting image_dim_ordering to ‘th’.
Tried setting dim ordering to ‘tf’ but more things broke, like loading weights. Anyway… things looks like they’re working on the GPU instance i have… I just need to figure out why is it not working in my docker env (ubuntu16, python3, keras==1.2.2 and tensorflow (1.3.0) .
All libraries have the exact same version in my docker env as in the gpu instance…
I’ve been able to use it with a vgg16 model though…