Unfreezing model leads to higher loss and lesser accuracy

Part 1 (2019)
1

Hi, I am trying to build an image classifier. Here are the steps being followed by me.

  1. Create databunch
  2. Create a learner object
  3. Call fit_one_cycle with 4 epochs
  4. Call unfreeze
  5. Find the discriminative learning rates
  6. Call fit_one_cycle again with discriminative learning rates

I tried retraining after unfreezing but accuracy goes down, always. Can anybody tell me where am I going wrong?