Reproducibility Challenge 2020 - fastai folks interested

[stefan-ai (Stefan Josef)]

Hi everybody,

We had a great meeting yesterday with @arampacha, @Dean-DAGs and @PriyanK7n.

@arampacha made a lot of progress, getting a Reformer language model to train successfully on a subset of Wikitext 103. See his post and notebook above.

@arampacha

Trying it on 64k tokens seq_len causal lm: https://colab.research.google.com/gist/arampacha/9cc2fd7b5818c91ce64013b83bcfa567/reformer_wikitext_clm.ipynb

A couple of other points we discussed:

• Training speed could become an issue when training Reformer (could you please share the training stats that you mentioned yesterday, @arampacha?)

• Relatively soon - maybe in the next meeting - we should create separate tasks so that we don’t end up all working on the same issues.

• Among the first tasks should be to re-create and share the enwiki dataset to make sure everyone is working with the same data and we can save pre-processing time

• @arampacha reported an issue when trying to load hugginface’s google/reformer-enwik8 so decided to train from scratch.

• The other pre-trained model on huggingface model hub, google/reformer-crime-and-punishment, uses a different tokenization approach than the enwiki model. Due to conflicting sequence lengths, I didn’t manage to successfully fine-tune the model on downstream tasks.

• Since the Reformer paper is very brief and leaves out some important details, we might have to reach out to the authors for clarification. However, let’s first collect our issues before doing so.

• @Dean-DAGs potentially has a contact at huggingface and kindly offered to reach out if needed.

• Additionally, we could try to replicate these results from training Reformer on SQuAD 2.0

@arampacha

While searching for related stuff I stumbled on this report from Stanford cs224n project. Reporters failed to achieve good results with Reformer on SQuAD 2.0 and complain about slow training:

If I missed anything, please add it guys!

@hallvagi: Thanks for sharing your ideas. We agreed on some of these points in our meetings already. This list is a great starting point for formulating specific tasks that team members or smaller groups can start working on. Nice to hear that you had a good experience implementing basic LSH.

@hallvagi

• test if the authors repo code works out of the box
• consider lucidrains reformer implementation
• getting the datasets (from hf/datasets I guess?)
• setting up a wandb project (wandb is easy to use with the fastai callback)
• einops transformer implementation (have to look into einops a bit…)
• implement revnet, lsh-attention etc. separately
• run ablations and experiments

I think we’re on a good way. Let’s keep the momentum going and meet again soon to define concrete next steps. Have a nice weekend everybody

PS: I agree that a separate slack/discord channel would be helpful. Could someone set it up?

@hallvagi

Also, should we continue discussion in the forum or set up slack/discord etc.?