Lesson 13 in-class

@prabu Yes. Google used attention for picking out house numbers from Google Street Images, and then used digit recognition on the numbers to determine addresses.

1 Like

Cool. Thanks @rachel.

Attention models are relatively new. Are neural networks used by apps like Google assistant (for speech recognition) or is it some other system?

Do you know if the current work strictly uses the hidden states of the final RNN layer, or have people tried using attention sums over different indices over multiple layers (the way we did with using perceptual loss from multiple layers)?

1 Like

@harveyslash Yes, Google uses deep learning for speech recognition. Vincent Vanhoucke, former tech lead of Google’s speech recognition and now at Google Brain, even created the Udacity TensorFlow/Deep learning course.

1 Like

I have taken that course, but I don’t think they covered exotic systems like what we do here. They mostly glossed over simpler concepts.

@harveyslash Agreed, it’s a fairly short, high-level course.

I did not get the ‘getting it to the right shape again’ part.Would it be possible to explain it again?

1 Like

@jeremy You showed how you figured out/tested the tensor shapes? How did you debug the Attention class itself as a whole? Dont know if this a right question to ask in this thread

1 Like

Are RNNs easier/cleaner in PyTorch? For. e.g would the Attention class have been relatively cleaner in PyTorch?

1 Like

I guess TimeDistributed doesn’t do that? Cause in that cause, it would have made sense to do TimeDistributed(Lambda()) right before the RNN to apply those calculations to the input, no?

Any specific reason why we used tanh non-linearity as opposed to sigmoid?

You still have the audio popping/crackling problem from time to time

Do you know if there are any training methods that capture the fact that “What is the population of Canada” and “What is Canada’s population”, are very nearly the same, and arguably the same/correct in meaning? My first instinct would be “use a vector representation instead of a binary to get your loss”, but it doesn’t occur to me that there’s an obviously sensible way to do that for sentences.

1 Like

could we translate between Chinese and English using method shown in class?

1 Like

Can we do transfer learning on Densenet?

3 Likes

Can you share the link to the densenet notebook we were going over at the end of the class? I can’t find it among all the class resources.

I was super impressed and intrigued by the results Vincent got out of his style transfer implementation. Can’t wait to see more!

2 Likes

Found it: https://github.com/fastai/courses/tree/master/deeplearning1/nbs

Edit: Oops, that is part 1.

Edit: Found this as well, but it doesn’t include lesson 13: http://files.fast.ai/part2/

Wait a bit @dennisobrien, it will be done during the week.