Random Bounding Boxes During Inference Time

I have trained a Focal Loss based multi-object detector. If I try the model on validation images I see a random behavior. Every time that I run the detection step on a batch of validation images I see different bounding boxes for the same image. I use the code below and show_nmf is the same as in the pascal_multi notebook.
x,y = next(iter(md.val_dl))
y = V(y)
batch = learn.model(V(x))
b_clas,b_bb = batch
x = to_np(x)
for i in range(8): show_nmf(i, thres=0.2)

I have attached a couple of images to compare.
Have you encountered such a problem?