we’re not exposing the model to the labeled test set.
Makes sense! But the situation that we are in, where we are using the model that we’ve created using (in part) the test set to further evaluate that same test set, feels intuitively shaky (or maybe it’s just me?). The same images are being seen by the model twice – once to build the model, and once to get a probability for those same images 
. At the very least, it’s interesting that there is not a potential problem here.