Lesson 11 wiki

[jeremy (Jeremy Howard)]

Resources

• Video
• Lesson 11 video timeline
• Memory Networks Notebook
• Powerpoint
• .py files and notebooks in github (git pull)

Papers

CNN improvements

• Systematic evaluation of CNN advances on the ImageNet
• Cyclical Learning Rates for Training Neural Networks
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• An Analysis of Deep Neural Network Models for Practical Applications
• Deep Convolutional Neural Network Design Patterns

(Deepening Lesson 10) Multi-modal image and text space

• Zero-Shot Learning by Convex Combination of
  Semantic Embeddings
• Attention for Fine-Grained Categorization

Data science bowl (lung CT)

• Shape-based computer-aided detection of lung nodules in thoracic CT images
• Lecture: k-means and mean-shift clustering
• Practical approach to mean-shift clustering
• V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation

Memory networks

• End-To-End Memory Networks
• Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
• End-to-End Prediction of Buffer Overruns from Raw Source Code via Neural Memory Networks
• Tracking the World State with Recurrent Entity Networks

[brendan (Brendan Fortuner)]

I have to admit there is so much new material in this class I feel increasingly behind every lesson. I think it will take me the rest of the year to catch up.

[jeremy (Jeremy Howard)]

@brendan

I have to admit there is so much new material in this class I feel increasingly behind every lesson. I think it will take me the rest of the year to catch up.

I try to make sure there’s enough material that those interested can keep themselves pretty busy for quite a few months. That doesn’t make you “behind”, just committed and curious

[jeremy (Jeremy Howard)]

The notebooks and .py files are there now.

[iNLyze (Constantin)]

Yes, @brendan, same here. Like I can see every single bit developing into a mid-size to large project. Thanks for sharing your feelings on this. I must say this class is so demanding and so incredibly rewarding. For lack of a better metaphor it feels like surfing the event horizon of a nuclear detonation. Make that a supernova. A live science fiction.
This is the birth of the 21st century - what it is going to be known for in the future. You don’t get the chance to be part of something like this many times in life.

[kelvin]

I’d rather Jeremy went faster and provide material for self-study. i.e. it’s better to have too much to do, rather than too little. Personally the best part of the course was pointers to how to keep up w/ the community at large. (wildml, import ai, arxiv-sanity, twitter, r/ML)

[jeremy (Jeremy Howard)]

@kelvin

I’d rather Jeremy went faster and provide material for self-study. i.e. it’s better to have too much to do, rather than too little

That’s the idea - you can always pick a subset to study, if I cover more stuff than you have the time to look at. And there’s no need (or expectation) to understand everything within the brief time period of the course; the forums and resources stick around afterwards and hopefully people will continue their own studies (perhaps along with some of their team).

[jeremy (Jeremy Howard)]

@iNLyze

For lack of a better metaphor it feels like surfing the event horizon of a nuclear detonation. Make that a supernova.

That’s what teaching it feels like too!

[sravya8]

Same here It is a fantastic learning experience. But one side effect of awesome content and eagerness to do it all is that it can feel like you are doing so little as compared to what you want to do, even if you are already doing a lot. Uff, I guess we can benefit from prioritization 101

[brendan (Brendan Fortuner)]

Regarding prioritization, each day I try to ask myself: what is the highest leverage activity I can complete today? With the flurry of new material and so many possibilities I’m finding it difficult to answer this question.

I want to get up to speed as fast as possible, but I also imagine there is a logical order in which to tackle this material? I’ve also been thinking a lot about how to balance different types of activities: side projects, Kaggle, theory, blog posts, papers, networking. How should we apportion our time?

Example activities:

• Learning pytorch or Keras
• Reimplementing all of Jeremy’s notebooks
• Revisiting Kaggle competitions from part I
• Studying CNN, GAN, RNN theory
• Deep diving into neural style

I don’t mean any of this as a critique. Just some musings. This course continues to amaze me and by providing so much material it does allow each student to focus on areas of personal interest.

[jeremy (Jeremy Howard)]

@brendan

How should we apportion our time?

I’m not sure it matters too much how you prioritize - you have the rest of your life to fill up with things you’re interested in! So just keep doing what you’re doing - follow up on what seems interesting and relevant to you today.

Whatever you work on, I very very strongly recommend (as you all know!) that you spend most of your time writing code to solve some problem. Whether it’s winning a kaggle comp, reimplementing a notebook, implementing a paper, working on your personal or work project, etc, is entirely up to you.

As you try to implement something, you’ll discover specific areas where you need to understand things better in order to solve your problem. So then you can do as deep a dive as necessary to figure that out. Using these “pull” approach means that you’re always studying things that you know are useful.

[analytics20132333 (Kicho)]

Hi @jeremy!
I think the last three links on CNN improvements are the same. I think the last two would be

• An Analysis of Deep Neural Network Models for Practical Applications
• Deep Convolutional Neural Network Design Patterns
  Thanks!

[aifish (Kent)]

Hi @jeremy, the Powerpoint link http://www.platform.ai/part2/lesson11/lesson11.pptx returns a 403 Forbidden error “You don’t have permission to access /part2/lesson11/lesson11.pptx on this server”

[aifish (Kent)]

Is the babi-memnn.ipynb also going to be posted? I couldn’t find it in the lesson11 download folder. Thanks!

[jeremy (Jeremy Howard)]

Thanks @aifish - both those issues should be fixed now.

[jeremy (Jeremy Howard)]

Thanks @analytics20132333 - since this is a wiki thread, you can fix problems like this yourself! (I see in fact someone has already fixed it )

[mariya]

I found this beautiful, interactive journal called Distill which publishes machine learning content with interactive visuals. Here’s an article on Attention & Augmented RNNs that clarify how the “attention” mechanism really works:

Attention and Augmented Recurrent Neural Networks

[Surya501 (Suresh)]

@brendan

I have to admit there is so much new material in this class I feel increasingly behind every lesson.

You spoke the words out of my mouth. I’d rather have more material than less as long as I can understand. As Jeremy pointed out, you have a long time to play with this. One of the most interesting things I picked up in this course is reading/scanning papers and linking it back to what we already know. For example, There was a paper from adobe on adding photo style (not painting style) from one photo to another. It was very similar to our style transfer exercise.

Thank you very very much for teaching this Jeremy. I feel like a kid in a candy store with too many choices.

[Surya501 (Suresh)]

Embedding #1

Embedding #2

Embedding #3

I used tensorboard to visualize the embeddings using PCA for the babi-memnn notebook. I could not get this to work with t-SNE and can’t understand why.

As you can see from the pictures, you can clearly see that for embedding #1, the names are far apart, but everything else clustered close to each other with numbers being spread out.

For the question embedding, you can see that “where is ?” is clustered together and far from others and the names are as far apart as possible. You can see the reverse in the question #3 where bathroom, bedroom, etc are far apart.

The code export the embedding to tensorboard is simple, but took me a few tries.

import tensorflow as tf
saver = tf.train.Saver()

LOG_DIR = ‘/home/surya/tensorboard_log’
os.makedirs(LOG_DIR, exist_ok=True)
saver.save(keras.backend.get_session(), os.path.join(LOG_DIR, “model.ckpt”))

metadata_file = open(os.path.join(LOG_DIR, ‘metadata.tsv’), ‘w’)
metadata_file.write(‘Name\tClass\n’)
for key, value in word_idx.items():
metadata_file.write(‘%06d\t%s\n’ % (value, key))
metadata_file.close()

In your log directory, create a file called projector_config.pbtxt with the following content.

embeddings {
tensor_name: “embedding_1_W”
metadata_path: “/home/surya/tensorboard_log/metadata.tsv”
}
embeddings {
tensor_name: “embedding_2_W”
metadata_path: “/home/surya/tensorboard_log/metadata.tsv”
}
embeddings {
tensor_name: “embedding_3_W”
metadata_path: “/home/surya/tensorboard_log/metadata.tsv”
}

start the tensorboard with tensorboard --logdir=~/tensorboard_log and explore the embedding tab (look for embedding_*)

Question: If someone can play with this, can you please help me understand why the t-SNE is not as stable/reliable for this short data (32 words, size-20 embedding) when PCA does just fine?

[lin.crampton]

transcribed the video, it is in https://drive.google.com/open?id=0BxXRvbqKucuNaFN2V09LcUozSU0

let me know if there should be changes