Differance Between Saved Models; learn.save() vs SaveModelCallback()

fastai
1

Hi everyone,

I am training an unet segmentation model on my data, by using this CAMVID one as an example.

I included EarlyStoppingCallback() together with SaveModelCallback() to the original notebook. In addition, I kept learn.save() too. The code looks like this:

callbacks=[EarlyStoppingCallback(monitor='valid_loss', min_delta=0.1, patience=3),
           SaveModelCallback(monitor='valid_loss', comp=np.less, min_delta=0.1, fname=fname, every_epoch=False)]
learn.fit_flat_cos(8, slice(lr),cbs=callbacks)
learn.save('stage-1')

The training was early stopped after 5th epoch and there are two saved models available now.

models
├── myModel.pth    165MB       #SaveModelCallback()
└── stage-1.pth    492MB       #learn.save()

I am surprised with this striking difference between two models in terms of file size. I was expecting to get identical files, since SaveModelCallback is using also using learn.save() as we can see here.

It seems like learn.load() function works for the both of them while I only receive this warning message for the second one:
UserWarning: Saved filed doesn't contain an optimizer state.

My questions are:

  1. What is the reason of this huge difference between file sizes?
  2. Are they really the same model?
  3. If not, which of them should I use for further training and inference?

Thank you in advance
caner

edit: fastai version: 2.4.1

2

Learner.save saves the optimizer by default while SaveModelCallback does not (via the with_opt argument).

1 Like