Lesson 9 Discussion & Wiki (2019)

Part 2 (2019)
37

is there an advantage of paying close attention to initialisation vs using something like batch norm?

2 Likes
38

Jeremy said that if you forgot something to go back to the part1 notebooks, but where are they? Is this https://github.com/fastai/fastai_docs/tree/master/dev_nb correct?

39

Oh yes, it will change the way your model train by a lot.

2 Likes
40

The notebooks from part 1 are here: https://github.com/fastai/course-v3/tree/master/nbs/dl1

3 Likes
41

in what way? faster convergence over batch norm?

1 Like
42

Why do we want to keep variance =1 all the time? Is it because of the gradient explosion/vanishing problem?

2 Likes
43

Itā€™s only a definition

44

Thanks!

What is the link I found thenā€¦

45

how does your computer explode when taking std deviation and mean again and again?

46

Jeremy is explaining it now.

47

I believe it could even be like convergence vs. not convergence at all. (If you have networks with many layers, like it is discussed in the papers shared during previous lesson).

1 Like
48

Yup, that is so

49

No itā€™s multiplying a vector by the same matrix again and again.

50

this feels important to the discussion but I think I missed what jeremy was saying here.

51

Is there any reason not to initialize each layer and directly scale the weights/biases to achieve 0 mean/1 stdev based on the observed statistics, rather than pre-solving for particular activations/structures?

52

This isnā€™t really a ā€˜little differenceā€™. Activations/Initializations are crucial parts of the network and probably a detail the biggest % of realtively new users never adjusts default values.

Good job!

53

is it sqrt(5) or sqrt(3) ?

2 Likes
54

Thatā€™s what BatchNorm does.

5 Likes
55

Link to the Twitter thread that Jeremy had mentioned

9 Likes
56

BatchNorm isnā€™t about init, though, itā€™s applied during net operation, right?