I had the same difficulty understanding and had to “unpack” the functions within ssd_1_loss function with comments and print-outs almost line-by-line to understand it. Here are my notes if it’s helpful to compare:
# original function
def ssd_1_loss(b_c,b_bb,bbox,clas,print_it=False):
bbox,clas = get_y(bbox,clas)
a_ic = actn_to_bb(b_bb, anchors)
overlaps = jaccard(bbox.data, anchor_cnr.data)
gt_overlap,gt_idx = map_to_ground_truth(overlaps,print_it)
gt_clas = clas[gt_idx]
pos = gt_overlap > 0.4
pos_idx = torch.nonzero(pos)[:,0]
gt_clas[1-pos] = len(id2cat)
gt_bbox = bbox[gt_idx]
loc_loss = ((a_ic[pos_idx] - gt_bbox[pos_idx]).abs()).mean()
clas_loss = loss_f(b_c, gt_clas)
return loc_loss, clas_loss
Reshape labels (bbox and clas) into usable format:
bbox,clas = get_y(bbox,clas) # reshape bbox and clas into usable form
Get the predictions (activations) for bounding boxes and rescale them to anchor box sizes (relative to center points and height/width). Note that this section also defines the “limits” of how much an anchor box prediction can move or transfrom from its original position and shape: center x,y can move 1/2*grid_size in any direction, height and width can be x0.5 to x1.5 of its original h or w:
# a_ic = actn_to_bb(b_bb, anchors) unpacked:
actn_bbs = torch.tanh(b_bb) # tanh(activation bb) into range [-1, 1]
actn_centers = (actn_bbs[:,:2]/2 * grid_sizes) + anchors[:,:2] # activation bb's center (x,y) can be up to 1/2*grid_sizes offset from original anchor box center
actn_hw = (actn_bbs[:,2:]/2+1) * anchors[:,2:] # activation bb's height and width can be between 0.5-1.5x the original anchor box h&w
a_ic = hw2corners(actn_centers, actn_hw) # convert activation bb x,y,h,w to bb corner coordinates
print(a_ic) # shape should be n activations (== # of anchor boxes) x 4 corner coordinates
Variable containing:
0.0990 0.0915 0.2647 0.2559
0.1293 0.3460 0.2624 0.4835
0.1314 0.5262 0.3009 0.7321
0.0968 0.8347 0.2352 0.9731
0.3896 0.1076 0.5187 0.2329
0.4016 0.3421 0.5279 0.4671
0.4036 0.5651 0.5344 0.6904
0.3719 0.8358 0.5008 0.9613
0.5301 0.1139 0.8497 0.2577
0.5093 0.3696 0.8775 0.5001
0.5085 0.5798 0.8768 0.7096
0.5369 0.7888 0.7736 0.9160
0.6248 0.0229 0.9921 0.3468
0.5901 0.2130 0.9647 0.5712
0.5938 0.4490 0.9673 0.7858
0.6393 0.7419 0.9741 0.9378
[torch.FloatTensor of size 16x4]
Calculate jaccard index (Intersection over Union value) for every ground truth bbox and anchor box
overlaps = jaccard(bbox.data, anchor_cnr.data) # calc jaccard scores between every anchor box and bbox ground truth
print(overlaps) # shape should be number of gt_objects (from bbox.data) x number of anchor boxes (from anchor_cnr.data)
Columns 0 to 9
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0091
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0356 0.0549
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
Columns 10 to 15
0.0922 0.0000 0.0000 0.0315 0.3985 0.0000
0.0103 0.0000 0.2598 0.4538 0.0653 0.0000
0.0000 0.1897 0.0000 0.0000 0.0000 0.0000
[torch.FloatTensor of size 3x16]
Matching round 1: find the highest IoU anchor boxes for each gt object and redefine their overlap scores as 1.99. This essentially “matches” those anchor boxes 1-to-1 to each gt object.
#gt_overlap,gt_idx = map_to_ground_truth(overlaps,print_it) unpacked:
prior_overlap, prior_idx = overlaps.max(1) # for each ground truth object, find anchor box that has max overlap (highest jaccard index) with it
gt_overlap, gt_idx = overlaps.max(0) # for each anchor box, find max jaccard index and which obj category this corresponds to
gt_overlap[prior_idx] = 1.99 # set jaccard index of matched anchor boxes to 1.99
for i,o in enumerate(prior_idx): gt_idx[o] = i # redefine the 'matched' indices in gt_idx to prior_idx indices. take a look below at gt_overlap and gt_idx values where gt_overlap = 1.99
print(gt_overlap, gt_idx)
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0356
0.0549
0.0922
1.9900 <---- matched anchor box 11! set the 'overlap' value = 1.99
0.2598
1.9900 <---- matched anchor box 13! set the 'overlap' value = 1.99
1.9900 <---- matched anchor box 14! set the 'overlap' value = 1.99
0.0000
[torch.FloatTensor of size 16]
0
0
0
0
0
0
0
0
1
1
0
2 <---- matched anchor box 11! set gt_idx value to index ([2]) of this anchor box in prior_idx
1
1 <---- matched anchor box 13! set gt_idx value to index ([1]) of this anchor box in prior_idx
0 <---- matched anchor box 14! set gt_idx value to index ([0]) of this anchor box in prior_idx
0
[torch.LongTensor of size 16]
Matching round 2: find the highest IoU scores for every remaining unmatched anchor box. If their IoU > threshold (0.2 in my case), match these as well to the gt object which it has the positive IoU. What results is that each gt object should have AT LEAST 1 anchor box matched to it, but it MAY have >1 anchor boxes matched if there are more anchor boxes with IoU > threshold (stored in ‘pos_idx’) for that gt object.
gt_clas = clas[gt_idx] # define the gt category for each matched anchor box
pos = gt_overlap > 0.2 # bool array for anchor box jaccards > threshold value, i've changed from 0.3 to 0.2 to illustrate how gt_overlap[12] (==0.2598) is counted as 'pos'
pos_idx = torch.nonzero(pos)[:,0]
gt_clas[1-pos] = len(id2cat) # set class cat of non-matches to the null class (20 aka 'bg' in this case)
gt_bbox = bbox[gt_idx] # define the gt_bboxes for each matched anchor box
print(gt_clas, gt_bbox)
Variable containing:
20 <---- no match! gt cat 20 = 'bg'
20
20
20
20
20
20
20
20
20
20
17 <---- matched anchor box 11! gt cat 17 = 'sofa'
10 <---- matched anchor box 12! gt cat 10 = 'diningtable' additional match to the 'diningtable' gt object because IoU of this anchor box is 0.2598 > threshold of 0.2
10 <---- matched anchor box 13! gt cat 10 = 'diningtable'
8 <---- matched anchor box 14! gt cat 8 = 'chair'
20
[torch.LongTensor of size 16]
Variable containing:
0.6786 0.4866 0.9911 0.6250 <---- no match so these gt bboxes are ignored
0.6786 0.4866 0.9911 0.6250
0.6786 0.4866 0.9911 0.6250
0.6786 0.4866 0.9911 0.6250
0.6786 0.4866 0.9911 0.6250
0.6786 0.4866 0.9911 0.6250
0.6786 0.4866 0.9911 0.6250
0.6786 0.4866 0.9911 0.6250
0.7098 0.0848 0.9911 0.5491
0.7098 0.0848 0.9911 0.5491
0.6786 0.4866 0.9911 0.6250
0.5134 0.8304 0.6696 0.9062 <---- matched! gt bbox for 'sofa' object
0.7098 0.0848 0.9911 0.5491 <---- matched! gt bbox for 'diningtable' object
0.7098 0.0848 0.9911 0.5491 <---- matched! gt bbox for 'diningtable' object
0.6786 0.4866 0.9911 0.6250 <---- matched! gt bbox for 'chair' object
0.6786 0.4866 0.9911 0.6250
[torch.FloatTensor of size 16x4]
Calculate the two loss functions (L1 for regression of bboxes, BCE or FocalLoss for classification of categories):
loc_loss = ((a_ic[pos_idx] - gt_bbox[pos_idx]).abs()).mean() # calc regression loss (L1 = absolute mean distance between each predicted bbox value and gt value)
clas_loss = loss_f(b_c, gt_clas) # calc classification loss (binary cross entropy or focal loss)