Tweaking on ImageNet

Part 1 (2019)
1

Hi Fastians! (Fastaians?)

I have been thinking about tweaking some large pretrained models, then retraining them on their original ImageNet task to see if my tweaks improved anything.

Has someone tried to do this as well?

NB: this is different from the usual ‘finetuning’ approach in this course, where we throw out the final parts of the model and replace them with our own layers and apply to our own data. This is really about fiddling with the (architecture of the) entire model (without breaking the current accuracy) and then resuming the training on the original data.

2

The latest commits in this repo https://github.com/mgrankin/SimpleSelfAttention inject an extra parallel layer to a resnet, then retrains (“transfer” learning) on imagewoof/imagenette (no improvement detected, but not any worse either). Note that imagewoof/imagenette are subsets of imagenet.

I’d say it’s close to your idea; also it might save you some time/cost to first test on those smaller datasets.

1 Like