I’m finetuning a xresnet50 with audio data by converting the audio files into spectrograms first and then training the network with the spectrograms. I am doing a 5-fold cross validation. Every time I let the program run, at some point it just stops training, always at the first epoch of the fine tuning process, usually at the very beginning, sometimes at 1,3%, sometimes at 4,5%, sometimes at 0%, but always at the first epoch. However, the problem sometimes occurs at the very first fold of the validation, sometimes only at the last fold, sometimes at one of the other folds.
Did anyone ever have a problem like that? There is no error message, so I don’t know what to look for to fix the problem, the progress bar just stops progressing. I’m not using Jupyter-notebooks, because I have to run the program at the server of my University, instead I run it in the Terminal with an ssh-session.
Any help is much appreciated!