Dynamic SSD implementation for fastai v1

I’m having a consistent problem with both an SSD implementation I built on my own as well as this one. All of my classes get assigned to background (whether I use focal loss or not). This appears to be a result of something the loss function is doing though it is identical to the one defined here. Could this instead be a problem with my data or the way classes are defined in my data? Has anyone seen anything similar? Basically achieving low loss numbers after some epochs of training but actual results are not valid at all using my own data set or PASCAL VOC.