@prabu Yes. Google used attention for picking out house numbers from Google Street Images, and then used digit recognition on the numbers to determine addresses.
Attention models are relatively new. Are neural networks used by apps like Google assistant (for speech recognition) or is it some other system?
Do you know if the current work strictly uses the hidden states of the final RNN layer, or have people tried using attention sums over different indices over multiple layers (the way we did with using perceptual loss from multiple layers)?
@harveyslash Yes, Google uses deep learning for speech recognition. Vincent Vanhoucke, former tech lead of Googleâs speech recognition and now at Google Brain, even created the Udacity TensorFlow/Deep learning course.
I have taken that course, but I donât think they covered exotic systems like what we do here. They mostly glossed over simpler concepts.
I did not get the âgetting it to the right shape againâ part.Would it be possible to explain it again?
@jeremy You showed how you figured out/tested the tensor shapes? How did you debug the Attention class itself as a whole? Dont know if this a right question to ask in this thread
Are RNNs easier/cleaner in PyTorch? For. e.g would the Attention class have been relatively cleaner in PyTorch?
I guess TimeDistributed doesnât do that? Cause in that cause, it would have made sense to do TimeDistributed(Lambda()) right before the RNN to apply those calculations to the input, no?
Any specific reason why we used tanh non-linearity as opposed to sigmoid?
You still have the audio popping/crackling problem from time to time
Do you know if there are any training methods that capture the fact that âWhat is the population of Canadaâ and âWhat is Canadaâs populationâ, are very nearly the same, and arguably the same/correct in meaning? My first instinct would be âuse a vector representation instead of a binary to get your lossâ, but it doesnât occur to me that thereâs an obviously sensible way to do that for sentences.
could we translate between Chinese and English using method shown in class?
Can we do transfer learning on Densenet?
Can you share the link to the densenet notebook we were going over at the end of the class? I canât find it among all the class resources.
I was super impressed and intrigued by the results Vincent got out of his style transfer implementation. Canât wait to see more!
Found it: https://github.com/fastai/courses/tree/master/deeplearning1/nbs
Edit: Oops, that is part 1.
Edit: Found this as well, but it doesnât include lesson 13: http://files.fast.ai/part2/