How would I turn CodeGen into an ULMFit-like classifier?

I found a causal language model named CodeGen on HuggingFace pre-trained to generate code, and I want to turn it into a classification model that would identify malware. However, I am not able to run it with AutoModelForSequenceClassification.

and nowhere in the docs does it provide how to convert it from a causal LM to a classifier. (something like this)
What pre-processing to I have to do to turn this into a classifier?

Just guessing by looking at HF documentation:

I would probably create a new pytorch module using CodeGenModel as encoder and adding a classification head to classify the “hidden states” (aka “embeddings” generated by your input code).

Be sure only the CodeGenModel is frozen while you’re training initially; run for few epochs like this then unfreeze the CodeGenModel.

Good luck! It seems an interesting and pretty challenging task for classification.

Another approach, at least to have a strong baseline, is to use GPT (aka see how good is one of the best general purpose LLM to spot vulnerabilities)

For this I suggest to take a look at this great video FSDL where he review Jeremy’s “LLM Science Exam” kaggle competition starter notebook on

Here it shows you how to organize your experiment, how to pre-prompt the model in order to “reason before decide”, and how to force it to return what you need (in your case it will be malware/not-malware).