How long are the sequences used to train translation models? Sentences?
There are two cool papers from Oct of last year that show how to do neural machine translation w/o parallel sentences!
Was an attention layer tried in the language model? Do you think it would be a good idea to try to add one and see what happens?
How would we start with pre-trained models of French and English and then fine-tune in this case (I.e., use the “Jeremey special” method)?
Google’s neural machine translation system.
Here is the link to my notes from lesson 6 if anyone is interested in a refresher.
Just heard him mention that we divide num_cpus
by 2 because with hyperthreading we don’t get a speedup using all the hyperthreaded cores. Is that just based on practical experience, or is there some underlying reason why we wouldn’t get additional speedup from hyperthreading?
Why are we not starting with language model of English and French then starting the translation from scratch.
How should we tokenize audio or video files.
why he didn’t add bos token?
How would you keep the pip- installed git version of fastai up-to-date? Would you just rerun the pip install command, or would you do some git pull command?
Do dimensions of both language embeddings have to be equal?
Perhaps pip install [library name here] --upgrade
?
pytorch official examples are not as good as the Keras official examples. Because most users of pytorch are researchers, I think they did not put too much importance on the best practices there assuming people know these.
So install would be ! pip install git+https://github.com/fastai/fastai.git
and would update be: ! pip install git+https://github.com/fastai/fastai.git --upgrade
I’m not sure – I was just guessing, but that may work.
why bs=bs*1.6?
you’re not doing the backprop, so you can run a larger batchsize.
For non-editable installs, the project is built locally in a temp dir and then installed normally. Note that if a satisfactory version of the package is already installed, the VCS source will not overwrite it without an --upgrade flag. VCS requirements pin the package version (specified in the setup.py file) of the target commit, not necessarily the commit itself.
https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support
Nothing prevents us from using different embedding sizes for encoder and decoder. It’s not a constraint of the architecture.