Unable to save model checkpoints when using Lambda in model

Part 1 (2017)
1

I made minor modification to basic usage of VGG model. The changes are to save model checkpoints. However, I am having issues with saving the checkpoints.

After first epoch, it crashes with
UnicodeDecodeError: ‘rawunicodeescape’ codec can’t decode bytes in position 106-107: truncated \uXXXX

Has anyone tried similar things? I am trying to rule out whether its my setup? Using keras 1.2.0 with Theano 0.9 dev on windows.

I have used model checkpoint before with code from keras examples.

–
Main code:

from keras.callbacks import ModelCheckpoint

callbacks = [
ModelCheckpoint(filepath=checkpoint_file, verbose=1, save_best_only=True),
]

Import our class, and instantiate

from vgg16 import Vgg16# Import our class, and instantiate
from vgg16 import Vgg16
vgg = Vgg16()

Grab a few images at a time for training and validation.

NB: They must be in subdirectories named based on their category

batches = vgg.get_batches(path+‘train’, batch_size=batch_size)
val_batches = vgg.get_batches(path+‘valid’, batch_size=batch_size*2)
vgg.finetune(batches)

vgg.fit(batches, val_batches, nb_epoch=1, callbacks=callbacks)

–
vgg.py changes to include callbacks

def fit(self, batches, val_batches, nb_epoch=1, callbacks=None):
self.model.fit_generator(batches, samples_per_epoch=batches.nb_sample, nb_epoch=nb_epoch,
validation_data=val_batches, nb_val_samples=val_batches.nb_sample, callbacks=callbacks)

–
full error

File “c:\anaconda3\lib\site-packages\keras\keras\models.py”, line 934, in fit_generator
initial_epoch=initial_epoch)
File “c:\anaconda3\lib\site-packages\keras\keras\engine\training.py”, line 1555, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
File “c:\anaconda3\lib\site-packages\keras\keras\callbacks.py”, line 43, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File “c:\anaconda3\lib\site-packages\keras\keras\callbacks.py”, line 305, in on_epoch_end
self.model.save(filepath, overwrite=True)
File “c:\anaconda3\lib\site-packages\keras\keras\engine\topology.py”, line 2624, in save
save_model(self, filepath, overwrite)
File “c:\anaconda3\lib\site-packages\keras\keras\models.py”, line 52, in save_model
’config’: model.get_config()
File “c:\anaconda3\lib\site-packages\keras\keras\models.py”, line 1030, in get_config
’config’: self.layers[0].get_config()})
File “c:\anaconda3\lib\site-packages\keras\keras\layers\core.py”, line 604, in get_config
function = func_dump(self.function)
File “c:\anaconda3\lib\site-packages\keras\keras\utils\generic_utils.py”, line 40, in func_dump
code = marshal.dumps(func.code).decode(‘raw_unicode_escape’)
UnicodeDecodeError: ‘rawunicodeescape’ codec can’t decode bytes in position 106-107: truncated \uXXXX

2

What if you tried it with the latest stable release of Theano (0.8.2) and/or TensorFlow?

3

Thanks, Jeff. I tried lot of tool version changes including Theano 0.8.2, python 2.7 (was using 3.4), …

Finally found the line with issue. Seems like Lamda() in model is not happy with checkpoints.

If I change the code as below - remove Lambda, it can save checkpoints.

def create(self, size, include_top):

    model = self.model = Sequential()
    # model.add(Lambda(vgg_preprocess, input_shape=(3,)+size))

    model.add(ZeroPadding2D((1, 1), input_shape=(3,)+size))
    model.add(Convolution2D(64, 3, 3, activation='relu'))
    model.add(ZeroPadding2D((1, 1)))
    model.add(Convolution2D(64, 3, 3, activation='relu'))
    model.add(MaxPooling2D((2, 2), strides=(2, 2)))

    #self.ConvBlock(2, 64)
    self.ConvBlock(2, 128)

…

4

It would be great if someone else can repro what I originally posted. Then it can rule out my environment, and point to keras bug.

5

You should save weights, not the whole model, to avoid this problem, with save_weights_only=False

6

Thanks, jeremy and jeff. That workaround works.

7

I know this post is a few months old, but it is currently one of the top results when searching for this error. It looks like it related to a Keras bug, either https://github.com/fchollet/keras/issues/4135 or the related bugs listed there.

The issue can be reproduced with the following two scripts. testme.py will run correctly in C:\ but not in C:\Users…

# library.py
from keras.layers import Lambda
def external_lambda():
    return Lambda(lambda x: x, input_shape=(1, 1))


# testme.py
from keras.models import Sequential
from keras.layers import Lambda
from library import external_lambda

def local_lambda():
    return Lambda(lambda x: x, input_shape=(1, 1))

model1 = Sequential()
model2 = Sequential()

model1.add(local_lambda())
model2.add(external_lambda())

model1.save("tmpfile")
model2.save("tmpfile2")