Hello,
I’m currently trying to reproduce the segmentation results from lesson3-camvid with my own dataset. I’m using 2D .png slices of 3D CT scans, and trying to segment the various lobes of the lung. Training progresses well at first, with training and validation loss getting down to about 0.1, but it then skyrockets, increasing exponentially (think loss of 10^10 and higher after a few epochs.) I’ve observed this across multiple datasets (all .png slices of CT scans), some as large as 16,000 images. Decreasing the learning rate does help to some extent, but this skyrocketing always happens eventually. Some visuals that may help:
lr_find output:
learningrate|386x262
fit_one_cycle output (lr=1e-5). Also note that acc_camvid returns nan:
plot_losses output (lr=1e-5):
show_results output (note how segmentation results in gridlike pattern):
Does anyone have any idea what might be causing this?