Logging training of ULMFiT

(Akim) #1

I am training my ULMFiT model on a server, but my computer tends to disconnect often and I miss training info - training & validation loss, accuracy etc. Is it somehow possible to log this into a file? I have already changed SAVE_PATH in fastprogress.py so that the function print_and_maybe_save saves the data into this file when the sample is small, but when it performs each step for 30-40 minutes (as when training on wikipedia), by an unknown to me reason it does not produce any changes in the file. Will be grateful for any help.

0 Likes

#2

Hi Aktsvigun hope all is well!
Have a look at my posts in this thread Platform: Colab ✅
maybe there is ometing there, that you could use to help solve your problem.

Hope this helps

mrfabulous1 :smiley::smiley:

1 Like

(Stefano Giomo) #3

Try Converting the notebook to script and run it.
Fastprogress works fine on the console!

Btw saving stats is a good approach.

1 Like

(Akim) #4

It is amazing. Thank you!

1 Like