There’s been some theoretical work in this area under the name of open set recognition and open world recognition.
Open-set recognition(introduced formally in here) is a topic in Computer Vision whose goal is to train a model which is able to recognize a set of known classes with high accuracy and is also able to recognize when an unknown class is shown to the model.
Open-world recognition(introduced formally here) is an extension to open-set recognition and aims to recognize unknown classes, as well as help label them and then retrain the original model to recognize the new classes.
A recent survey on open-set recognition lists common approaches and discusses future directions for the topic.
One basic way to do it is to train a one-class SVM which recognizes each individual class and the ensemble of SVMs predicts whether the class is known or unknown