Lesson 12 (2019) discussion and wiki

Part 2 (2019)
102

I guess it is more about using some digital notebook, i.e. OneNote or Evernote maybe. To easily dump the files, attach logs, diagrams, etc.

2 Likes
103

I think itā€™s just a traditional text file with the resutls of the experiments copy-pasted.

3 Likes
104

Not really. I just see combining .3 of the Feb 2nd and .7 of the Feb 15th for Feb 11thā€¦ its probably just the 2 combined images that is confusing me.

105

Any sort of version control system is essential, even for teams of 1. That way you can go back to any point in the history of your project. And sometimes you need to.

106

I use Evernote for this exact purpose. Every research project has a notebook. I keep records of changes, plans for future changes, and try to record results along the way.

3 Likes
107

Notion is also a fabulous tool for this.

5 Likes
108

What about just cloning notebooks after each experiment and keeping .ipynb files in version control system? I am going with this strategy and it seems to be working so far. Is there a reason why copying results into a text file would be better?

109

ā€œThat weā€™ll transfer in a future lessonā€ could become a meme of this course :wink:

110

As I understand it, itā€™s a linear combination of things in vector space. Images are already tensors so thatā€™s easy. For categorical data you would need to convert all categoricals to embeddings and whatnot.

So if you have day_of_month and month categoricals you would need to pass them through embeddings first.

2 Likes
111

transformer xl has state

112

Regarding applying mixup to NLP or other fields where you have categorical inputs, is it necessary to have pretrained embeddings first to do the mixups on? It doesnā€™t seem obvious to me how to do mixup when youā€™re dynamically learning the embedding weights as well. And you certainly donā€™t wanna do mixup on one-hot encoded inputs before embedding them.

2 Likes
113

Side note: much respect for both Jeremy and Rachel for tonight - both look visibly unwell.

5 Likes
114

I tried doing that. I need some dropout with my notes.

So I use a physical notebook for ā€œstrategic non-code notesā€ then move onto one-note for coding. Separated by project. As I finish my project, I publish my findings in Github and an article to myself, my associated one-note gets deleted.

If it was important enough to remember, it is important enough to publish.

1 Like
115

What is required to use ULMFiT with SentencePiece?

Do we have to re-train the full model using SP for tokenization?

2 Likes
116

Jeremy is speaking about parameters of a script, so they are easy to copy and paste. A notebook (as long as you save the commit/date of the corresponding libraries), it should work.

1 Like
117

We are using ULMFit in prod, and it performs great even with small data set.

6 Likes
118

Yes, you need a pretrained model with the same tokenization.

119

Any examples of ULMFiT being applied to multi-task learning problems (esp. seq2seq tasks)?

3 Likes
120

thanks! that helps, but then wouldnā€™t embedding drop out be similar?

121

Iā€™m assuming Jeremy is about to discuss this, but if there are folks who have done it with SentencePiece, Iā€™d be interested to know how they did it.