Part 2 lesson 11 wiki

How long are the sequences used to train translation models? Sentences?

3 Likes

There are two cool papers from Oct of last year that show how to do neural machine translation w/o parallel sentences!

13 Likes

Was an attention layer tried in the language model? Do you think it would be a good idea to try to add one and see what happens?

6 Likes

How would we start with pre-trained models of French and English and then fine-tune in this case (I.e., use the “Jeremey special” method)?

4 Likes

Google’s neural machine translation system.

Here is the link to my notes from lesson 6 if anyone is interested in a refresher.

11 Likes

Just heard him mention that we divide num_cpus by 2 because with hyperthreading we don’t get a speedup using all the hyperthreaded cores. Is that just based on practical experience, or is there some underlying reason why we wouldn’t get additional speedup from hyperthreading?

7 Likes

Why are we not starting with language model of English and French then starting the translation from scratch.

3 Likes

How should we tokenize audio or video files.

5 Likes

why he didn’t add bos token?

4 Likes

How would you keep the pip- installed git version of fastai up-to-date? Would you just rerun the pip install command, or would you do some git pull command?

3 Likes

Do dimensions of both language embeddings have to be equal?

9 Likes

Perhaps pip install [library name here] --upgrade?

1 Like

pytorch official examples are not as good as the Keras official examples. Because most users of pytorch are researchers, I think they did not put too much importance on the best practices there assuming people know these.

2 Likes

So install would be ! pip install git+https://github.com/fastai/fastai.git
and would update be: ! pip install git+https://github.com/fastai/fastai.git --upgrade

1 Like

I’m not sure – I was just guessing, but that may work.

why bs=bs*1.6?

1 Like

you’re not doing the backprop, so you can run a larger batchsize.

2 Likes

For non-editable installs, the project is built locally in a temp dir and then installed normally. Note that if a satisfactory version of the package is already installed, the VCS source will not overwrite it without an --upgrade flag. VCS requirements pin the package version (specified in the setup.py file) of the target commit, not necessarily the commit itself.

https://pip.pypa.io/en/stable/reference/pip_install/#vcs-support

Nothing prevents us from using different embedding sizes for encoder and decoder. It’s not a constraint of the architecture.

4 Likes