Finetuning a model (unfreezing) provides low accuracy


I was successful at using resnet18 to solve a multilabel classification problem and getting a high accuracy (using stratisfied 5 folds Cross-Validation):

Yet, when I unfreeze the model and train based the chosen lr value from the new lr curve and the old one, the model seems to be getting worse in partial accuracy, even with different values of new lr (slice(1e-05, lr/10); slice(1e-04)) :

Do you think I should choose another value of lr, or maybe I should not unfreeze the model and just leave it as it and maybe then try bigger sizes of images?