Lesson 7 In-Class Discussion

Part 1 (2018)
21

Weren’t we taking the transpose of these chunks? (bptt x bs, versus bs x bptt)

22

So, we should try to increase bptt as much as possible?

23

Just keep things below the explosion point.

2 Likes
24

@KevinB, you make it sound so simple :slight_smile:

1 Like
25

Since the loop is dynamic, can we keep changing BPTT value as we loop?

26

“In theory there is no difference between theory and practice. In practice there is. - Yogi Berra” - Kevin Bird

8 Likes
27

For each mini-batch it automatically happens in pytorch!

28

What else can I include on that tokenize?

29

Look at get_tokenizer:

2 Likes
30

Are you sure BPTT is not a constant?

31

For words it can be quite complicated
https://nlp.stanford.edu/IR-book/html/htmledition/tokenization-1.html

32

Jeremy explaining now.

1 Like
33

you’re right…last time we saw something approx 70…not exactly 70. So, if we did the Python RNN from scratch,we could approzimately ramdomize it. cool.

34

If my Language model isn’t predicting well, what could be the reason?
all we’ve is bs, bptt and some dropout to play with?

35

Do you have any details about what we email for part 2? Do we just say “yes, I’d like to do part 2” to the same email address? @yinterian / @jeremy?

1 Like
36

How much training are you doing? I was a bit surprised when I kicked off the imdb code and found it did 70+ epochs of 4000+ batches.

37

I’ve less data than IMDB. :frowning:

38

International you have to contact fast.ai.
In person would be through the data institute at USF.

39

How many words are you working with?

40

bs=64; bptt=70

image
image.png814×71 2.24 KB