Replicate Fit one cycle in Keras?

Part 1 (2019)
1

Hello everybody!
I tried the fastai transfer learning technique of Lesson 1 on a dataset i’ve been working on with Keras and it gives amazing results! Way more than i’ve been able to achieve with Keras also using transfer learning.

As I want to learn more about the field, I need to understand what is really going when you create a cnn learner and use fit one cycle.

The model converge realllly fast with fastai, and it takes way more epochs in keras to start to converge.
I tried really hard to replicate the fastai results with keras but I’m not getting good results… :confused:

So my question is, how can i know what are the default hyperparameters used in fit one cycle? And how could I try to replicate the fastai technique in keras.

Thanks a lot!

1 Like
2

I’ve implemented a while ago the original one_cycle: https://github.com/nathanhubens/KerasOneCycle

Similar to what Sylvain explained his blog post

The only difference is that I didn’t use cosinusoidal steps but that should be easy to change :slight_smile:

If you want to see the default hyperparams of fit_one_cycle in fastai, you should take a look at the source code here

5 Likes
3

Thanks a lot, I will check this out!

4

Hi, is there any way to unfit a model?
Like if I have over fitted and I want to reverse that, so is it possible to unfit or anything like that?
@NathanHub

5

@AjayStark You either have made a checkpoint prior to the overfitting, and can now reload the model back to that checkpoint; or you have just went pass a point of no return, and have to retrain it from scratch.

1 Like
6

ohhhh ok.
Thank you :slight_smile:

7

You can try more agressive data augmentation if you see that you are overfitting but, as stated by James, it can sometimes be difficult to reverse. So you should restart the training with maybe more regularization (dropout, weight decay, data augmentation, batch norm, …) :slight_smile:

1 Like
8

Okay, Thankyou :slight_smile: