Bayesian Optimisation with architecures?

fastai
1

I’ve been using Bayesian Optimisation with layer counts + sizes. I presume you could use the same idea but with model architectures.

The basic idea being

  • For each layer have whatever layer types you want in a list e.g. nn.Conv2d
  • Build your fit function with appropriate inputs being e.g. {‘layer_1’: (0, 1)}
  • In the fit function use list indexing to pull out the layer type (e.g. layer_types = [nn.Conv2d, nn.Conv3d], layer[1])
  • Run the training + evaluate
  • Repeat using BO

This obviously doesn’t take into every property/input but would this idea work + help discover new architectures that work for your problem?

2

I use this for tabular all the time :slight_smile: I’m not sure on a “new” architecture, seems really computationally expensive to do so, but is also reminiscent of automl too

1 Like
3

Yes I’ve been working with BO with tabular also.

Granted computationally expensive but could lead to some improvements no? Ah yes didn’t consider the automl.

I’m just trying to think of ways to improve my metric/losses besides feature engineering/tweaking tabular_learner.

Besides the tabular_learner/fine_tune inputs what other things have you used BO on regarding tabular?

4

That’s mostly it (including all the hps like wd, emb_drop, etc) However another bit to try is combining all of that with a Permutation Importance algorithm and remove any negative/non helpful variables

5

Yes I’ve been looking into permutation importance as well, very handy.

Are there any other popular optimisation methods besides BO?

I understand a lot of gain comes from feature engineering, just curious what else is out there (preferably something I can shove stuff into so I don’t have to repetitively type in new params).

6

Training with Weights & Biases (Callback) can help you see what’s happening in your arch as it’s training, but no not to my knowledge

7

I’ve yet to learn what callbacks are for/how they are used. I’ve seen them in the docs but haven’t dived in.

Could you sum up what they are used for?

8

Callbacks allow for modification to anything in the training loop you wish to do, at every single step: https://docs.fast.ai/callback.core#Callback

1 Like
9

Great thanks Zach!