In lesson 6, Jeremy is doing pseudo-labeling using the test data. It seems there is an implicit assumption that your test data is balanced in terms of different classes(I can be wrong here). My question is will pseudo-labeling work if your test data is imbalanced? For example, if there are 10000 instances of cats and dogs in the test set, will this work if only 1000 are cat images and the rest of the 9000 are dog images?
I think you can apply the same rules to pseudo labeling imbalances as to other cases of imbalance. I would try to augment your 1000 images to maybe 4000 and take 4000 from the other predictions by random.
A little imbalance is usually not a big problem, but a lot can be troubling.