Unet / ResNet for image segmentation.
I’m doing hyperparameter optimization with Optuna. One of the hyperparameters is the gamma factor for FocalLossFlat. I’ve set a very low minimum value for it: 1e-10. Based on a few trials so far, when gamma is very low, I get NaN in the validation loss.
What could be considered a “safe” minimum value for gamma, that would not waste time generating NaN in validation?
Ideally I would like to explore low values for this parameter, since gamma=0 is equivalent to CrossEntropyLoss. But at the same time I need to make sure I’m not wasting time.
So, any suggestions for the lowest safe value of gamma?