I found a causal language model named CodeGen on HuggingFace pre-trained to generate code, and I want to turn it into a classification model that would identify malware. However, I am not able to run it with AutoModelForSequenceClassification.
and nowhere in the docs does it provide how to convert it from a causal LM to a classifier. (something like this)
What pre-processing to I have to do to turn this into a classifier?
I would probably create a new pytorch module using CodeGenModel as encoder and adding a classification head to classify the “hidden states” (aka “embeddings” generated by your input code).
Be sure only the CodeGenModel is frozen while you’re training initially; run for few epochs like this then unfreeze the CodeGenModel.
Good luck! It seems an interesting and pretty challenging task for classification.
Another approach, at least to have a strong baseline, is to use GPT (aka see how good is one of the best general purpose LLM to spot vulnerabilities)
For this I suggest to take a look at this great video FSDL where he review Jeremy’s “LLM Science Exam” kaggle competition starter notebook on
Here it shows you how to organize your experiment, how to pre-prompt the model in order to “reason before decide”, and how to force it to return what you need (in your case it will be malware/not-malware).