Many Out-Of-Vocabulary words while fine tuning a language model

I want to fine tune a pre-trained language model on my specific corpus. The problem is that most of the important words from my corpus are not available in the vocab of the pre-trained model and if I replace them with UNK, then I wont get a good result.
Any idea on how to deal with this problem?