Start by reading about it in the book, then do some experiments in a notebook, and tell us what you find out – if you have any questions along the way, let us know!
(Yes, I could just tell you directly, but you’ll learn way more if you experiment yourself… 
)
This walkthrough was so useful. To make sure I was understanding it I re-created, but due to issues with kernels dying in my wsl I ended up running this on my Apple M1 (cpu). So used tiny and just 3 epochs, and 3 runs across images 32,64 and 128 - then averaged them (weighting the larger images) and ended up above 94% - which surprised me. That would have been about 120 on the leaderboard :). My best is at 50 right now - time to try paperspace again (then might take a look at the Metal options with Pytorch if I get brave).
Perhaps I’m mistaken, but in walkthru 13, Jeremey mentions that we can indicate the number of independent inputs via the ImageBlock function. I will try this out, but I assume we could add the variety as input and change the n_inp to 2. Thanks for running the code without explicitly stating n_inp = 1 so we could gain a deeper understanding of the DataBlock function.
In case others run into this - I was getting an error on Paperspace - not a CUDA memory issue but “Could not do one pass in your dataloader, there is something wrong in it. Please see the stack trace below” and the bottom of the trace was “cusolver error: CUSOLVER_STATUS_EXECUTION_FAILED, when calling `cusolverDnSgetrf( handle, m, n, dA, ldda, static_cast<float*>(dataPtr.get()), ipiv, info)”. I found that reducing the image size in the resize property (not the size) avoided this. Despite dropping down from 480 to 240 I still got a good result on a large swinv2.
Finally solved the problem. It turns out it is not about the walk-thru code/data or anything else. It is all about a PyTorch bug that appears on certain Nvidia drivers.
Had the same problem, intermittently… Reducing from 480 to 360 also addressed it.
Although I just found that increasing could fix it too. I think just different values hit the PyTorch bug - so you have to see what works.
Walkthru 10 detailed note in the form of questions
The best vision models for fine-tuning notebook
00:00 - Questions on tabular data and the fastbook has the answer
paddy dataset is similar to ImageNet in terms of shape and size but have no paddy labels
Is the dataset (e.g., PETS dataset) very similar to the pre-trained model’s dataset (e.g., ImageNet)?
The more similar, the better the dataset can fine tune the model by making use much of the pretrained weights
How large is the dataset, especially when the dataset is not similar e.g., the planet dataset to the Imagenet?
When datasets are very different, most of weights from the pretrained model will be useless, so the larger of the dataset, the more weights can be trained, the better the model can learn
If we can find the best model from PETS dataset and Planet dataset, then it may be applied to other similar senarios
Explore the fine_tune.py from fastai_timm repo
Explore the sweep_planets_lr.yaml from the repo
Weights and biases API can enable us to see our experiment results inside Jupyter notebook
StringIO is the key to make sure pd.to_csv to save dataframe into a string rather than a file
import ghapi.core as gh
g = gh.GhAPI()
gist = g.create_gist('description of the gist', content_as_string, filename='', public=True)
gist.html_url
What does Jeremy use gist for here and generally?
How to calculate the score for all models based on their error_rate, fit_time, and GPU_mem?
How does Jeremy come up with the score design?
How to sort all the models based on their score and display the top 15 models?
#question How much does fit_time and GPU_mem matter more and when?
What is gpu mem and when does it matter?
Because small datasets won’t help large models to learn much.
Can we apply the findings (models, model families, good parameters) to all computer vision classifications? Yes
How many GPUs and for how long does Jeremy run the experiment? 3 GPUs for 12 hours
Why we don’t need to try every possibility on every level?
How did Jeremy pick the range of values of parameters for experiments?
to google
model zoo
paper with code
hugging face
Jeremy wants to share knowledge more freely and openly whereas academic journals generally make it difficult.
convnext small in22k
squish, but they both use square images for augmentation`Resize(480, method=‘squish’), batch=aug_transforms(size=224, min_scale=0.75)
Resize((480, 640)) do? to reverse only 3-4 images which have opposite aspect ratio, and do nothing to the rest of imagesvit_small_patch16_224 model
squish, and as Jeremy said generally squish version works better`Resize(480, method=‘squish’), batch=aug_transforms(size=224, min_scale=0.75)
#question why still use Resize(480) rather than Resize(640)?
Resize(640, method=ResizeMethod.Pad, pad_mode=PadMode.Zeros)?What is the logic behind it?
The first time error rate down to <2%
all models must use augmentation image size 192 and Resize(480)
build two models with or without squish using `Resize(480, method=‘squish’), batch=aug_transforms(size=224, min_scale=0.75)
build a third models on Resize(640, method=ResizeMethod.Pad, pad_mode=PadMode.Zeros)
Jeremy found it very interesting that this swin large and slow model family works better than previous small and fast model families on even smaller resized images.
The first two are with or without squish
Resize(480, method='squish'), batch=aug_transforms(size=224, min_scale=0.75)
The third one is on Resize(640, method=ResizeMethod.Pad, pad_mode=PadMode.Zeros)
convnext_large_in22k
Why to use a different seed number or different set of batches for doing experiments in this group of models?
How does Gradient accumulation prevent out of memory problem?
How much time Jeremy spent on all these work
53:56
Just to make it more confusing, I found that if I rerun the cell a second time, it works fine without dropping the size.
How do you resume training on a saved model after the kernel is shutdown.
I found the following link but I think start_epoch is depricated.
I loaded my model using load_leaner. I created a DataLoaders in the usual way, and attached it to the leaner using learner.dls = dls.
When I call fit_one_cycle it works, but if I train for one epoch, the accuracy rate is much worse than on the saved model. (I confirmed that inference on the saved model before additional training is fine).
You’ll need to use a much lower LR when you continue training a fine-tuned model.
Would lr_finder be the method to use for that?
Possibly. Although I normally just divide my previous LR by 5 and it works OK.
Yes, I have seen the same. Looking at the bug mentioned in the other thread, it looks maybe like a rounding/precision error - so if things get loaded into memory differently maybe things go ok.
For anyone who finds this thread with a search:
One other thing that took me some time to work out is that if you are loading a model to resume training (or even do inference on large amounts of data), the load_learner function loads to the CPU by default, so everything is very slow. If you plan to do additional training, set a flag in the load_learner like this:
load_learner('path/to/file', cpu=False)
This puts it on the GPU. Took me some time to figure out what was going on and how to fix it.
tags for searches: learner is slow, put learner on GPU
FYI, I tried it both ways and your “divide original LR by 5” way worked better.
So I did my homework and I think I understand tta and its rationale
I presume that if you use tta on your test set as the validation of your model, then you would also use tta for inference if the model is put into production?
Yeah otherwise you wouldn’t get similar results in production.
I copied and ran Jeremy’s code twice using the convnext_small_in22k model and factory method from_folder. The overall result on the validation set using tta was the same, but each epoch (and the final accuracy and validation loss) were slightly different from their corresponding epochs in the other experiment. I used the same seed for each experiment. (And obviously the same hyperparameters.)
There must be some randomness, but I thought that using the same seed was supposed to eliminate this. What is the source of randomness? Does this have any implications for creating models (eg. you may get slightly different results if you rerun so run more than once)?
(I also ran using the DataBlock api and got a slightly different answer).
I encountered a problem to run this notebook:
“course22/10-scaling-up-road-to-the-top-part-3.ipynb at master · fastai/course22 · GitHub”
To fix this, I modified the following, which is in the pull request right now. Please take a look at the following pull request. Thanks