Agree with this. The callback feature in Keras is quite powerful for which metrics have to be returned as dict. Helps with checkpointing, visualizations and various other nifty little things. Adding https://keras.io/callbacks/ for reference.
Wow, hub is pretty amazing! Created my first pull request using hub (albeit a simple one).
Had a bit of a snuffle separating my work account from my personal account. But once I got that sorted out, it was straightforward from there.
2 Likes
You can do this with cycle_save_name.
2 Likes
We have callbacks in fastai too, much inspired by keras in fact! Have a look at sgdr.py to see an example of their use.
2 Likes
These both sound like nice additions.
Added a new feature request to the Wiki Above for Unfreeze of Layers:
Currently unfreeze automatically unfreezes to 0. This causes long runs when working with larger sized images see (Lesson 3 In-Class Discussion). Would be good to have options on how may layers / sub-layers to unfreeze.
Jeremy suggested - you can use freeze_to() for that
But most pre-trained networks have only two layers above the finetune layer. They were both huge. Is it possible to freeze_to a sub layer or breakdown to more layers in the pre-trained network?
The caveat is we have to give more learning Rates. Might be better to give an option to specify a dictionary of layer names we want to unfreeze and learning rates for them? Thoughts / suggestions?
One idea I’ve had is if you have a cycle_save_name=’ ’ in a fit, it would be nice if it noted where the models were saved so something like this:
[ 0. 0.29109 0.22709 0.9116 ]
[ 1. 0.27596 0.21606 0.91596]
[ 2. 0.25896 0.21201 0.91738]
[ 3. 0.23578 0.21034 0.91812]<-----PtH_cyc_0.h5
[ 4. 0.25659 0.20687 0.92041]
[ 5. 0.2449 0.1977 0.9226]
[ 6. 0.23827 0.1925 0.92457]
[ 7. 0.22647 0.19253 0.92468]<-----PtH_cyc_1.h5
1 Like
You can keep three learning rates while adding more possibilities on how many layers to unfreeze. You can make a function. We can write something called:
freeze_to_layer()
I’m considering changing freeze_to to make it take a layer number, rather than a layer group number. I do agree the current behavior isn’t necessarily what we want…
1 Like
Aren’t the models always saved in the same location though? Do you mean it would be nice to be able to choose where they are saved?
Currently they are saved inside tmp\sz\models?
Hi,
In the dataset.py file, we define get_cv_idxs() as:
def get_cv_idxs(n, cv_idx=0, val_pct=0.2, seed=42):
np.random.seed(seed)
n_val = int(val_pct*n)
idx_start = cv_idx*n_val
idxs = np.random.permutation(n)
return idxs[idx_start:idx_start+n_val]
We have set seed=42. Shouldn’t we create truly random idxs everytime we run the code? If the validation loss (or any other metric) is consistent regardless, then, it would mean our model generalizes well.
Many times we care about reproducibility, so it’s useful to have the option. If you want a random value, you could just pass in a value of None for the seed. np.random.seed(None) is Random.
But you bring an important point, may be by default seed should have been None instead of 42. I checked scikit-learn docs and looks like they use None as default for seed. We should also consider changing default value to None.
2 Likes
Yes, this is exactly what I wanted to say 
It should be None by default and the definition should contain the argument seed if the user still wants to set it.
But but 42 gives the answer to everything! 
1 Like
I would like to add the following to my fastai Wish List:
- print out the names of the metrics
Currently this is the output
A Jupyter Widget
[ 0. 0.03597 0.01879 0.99365]
[ 1. 0.02605 0.01836 0.99365]
[ 2. 0.02189 0.0196 0.99316]
Would be nice if it included the following:
A Jupyter Widget
[epoch training loss validation loss accuracy]
[ 0. 0.03597 0.01879 0.99365]
[ 1. 0.02605 0.01836 0.99365]
[ 2. 0.02189 0.0196 0.99316]
2 Likes
Generally like the approach of having titles to these outputs. The issue is, the last metric could be anything - not always accuracy. So might be better to say it as Val. Metric instead.
2 Likes
@ramesh I just started watching the ML videos, and I’m on Lesson 2, random forests. Do you know what the first two columns represent?
[xxx, xxx, training R^2, validation R^2]
[0.1026724559118164, 0.33553753413792303, 0.9786895444439101, 0.79893791069374753]
Its RMSE for Train and Valiation if this is coming from print_score(m). print_score is defined near the top of the notebook.
