I am training my ULMFiT model on a server, but my computer tends to disconnect often and I miss training info - training & validation loss, accuracy etc. Is it somehow possible to log this into a file? I have already changed SAVE_PATH in fastprogress.py so that the function print_and_maybe_save saves the data into this file when the sample is small, but when it performs each step for 30-40 minutes (as when training on wikipedia), by an unknown to me reason it does not produce any changes in the file. Will be grateful for any help.
Aktsvigun (Akim) #1
Hope this helps
ste (Stefano Giomo) #3
Try Converting the notebook to script and run it.
Fastprogress works fine on the console!
Btw saving stats is a good approach.
Aktsvigun (Akim) #4
It is amazing. Thank you!