Index
- Understanding fastai import system
- Save Databunch when using fastai.text
- Leslie’s papers use accuracy vs lr, but fastai uses loss vs lr graphs
- Should I use the techniques I learned in fastai like cyclic learning, momentum, adamW in normal ML tasks also
- Random Links
- Exploring Neural Networks with Activation Atlases
- Tip on how you can get source code of a paper
- Drone Simulation Resources + DL
- PyTorch Resources
- Maths Resources (how to understand backprop)
- Einsum example using Transformer
- Implementing the ‘empirical’ initialization from All you need is a good init
- Paper submission advice?
Understanding fastai import system
from fastai import *
from fastai.vision import *
Why we have to do these imports or why not have a single import? The short answer is due to the __init__.py files. So when we make __init__.py file in a folder it tells python to treat the py scripts as modules which we can then import in other files. Now if you see the fastai github repo and then look for the __init__ files you can see what is being imported.
Further reading python import system docs. Forum link
Save Databunch when using fastai.text
When using Text classifiers and you have created a Databunch, you should specify a fixed vocab. This means if you created your databunch once and then again try to create a databunch your vocabulary would change, in case you are doing random splits. So save your data and then load it when needed.
data = (TextList.from_csv(...)
.split_by_rand_pct()
# Now if I run the above code twice I would
# get a different vocab, so save your data
# and then load it again
Forum link
Leslie’s papers use accuracy vs lr, but fastai uses loss vs lr graphs
The report by Leslie N. Smith used accuracy vs some parameter, to decide the value of that parameter but fastai uses loss vs learning rate. The reason being accuracy is only a useful metric for certain tasks (such as classification but not for regression) while you always have a loss regardless of the task. Also, while training your model you are interested in optimizing your loss function, not the accuracy.
Forum link
Should I use the techniques I learned in fastai like cyclic learning, momentum, adamW in normal ML tasks also
Depends on the models you are using. Like some models have a convex loss surface that do not need advanced learning scheduling.
Random Links
- Exploring Neural Networks with Activation Atlases by OpenAI and Google. It allows you to visualize activations from an image classification network.
- Tip:- If you are implementing some paper and the source code is not available then you can try emailing the authors of the paper to get the code and maybe some extra info too. They are happy to share with people who like their work. (I did not make any claim on how long the reply would take in the first place)
- For those interested in learning about drone simulations, applying deep learning to it and things related to it, refer to this forum.
- PyTorch Resources PegasusWithoutWinds wrote a really nice summary of resources that you can use to learn PyTorch. link
- Maths Resources A quick summary where James Thompson gives the path he took to learn maths and improve his understanding of backpropagation. link
- Fun matrix math example: the Transformer’s attention mechanism. Post by cqfd, ideal practice for your einsum skills.