About the Part 2 & Alumni (2018) category

Use this category for discussions of Cutting Edge Deep Learning for Coders (2018), Part 2. Also use this category if you’ve finished the course and want to discuss topics that may be of interest to other part 2 students and alumni.

For topics that aren’t mentioned in the course, like RL, tensorflow, etc, and for general interest topics such as AI news, please use the #deep-learning forum.

Here’s is a link to all the lesson resources for this part:

I’ve made this a wiki thread so feel free to add any resources, FAQs, etc that you think new students may find helpful.

92 Likes

HTML Code to review Part 1 Lessons

I’ve been reviewing the Part 1 Lessons to prepare for the class and wanted a re-sizable browser window with nothing but the class video. So I’ve written this handy HTML file, just save the code below to a file called “class.html” or something else and double click on it to run it. You can select any class from the buttons tab on top of the video.

<html>
<style>
.video-container {
   position: relative;
   padding-bottom: 56.25%;
   padding-top: 30px; height: 0; overflow: hidden;
}

.video-container iframe,
.video-container object,
.video-container embed {
   position: absolute;
   top: 0;
   left: 0;
   width: 100%;
   height: 100%;
}
</style>
<div>
<button onclick='changeIFrame("https://www.youtube.com/embed/IPBSB1HLNLo")'>Lesson 1</button>&nbsp;
<button onclick='changeIFrame("https://www.youtube.com/embed/JNxcznsrRb8")'>Lesson 2</button>&nbsp;
<button onclick='changeIFrame("https://www.youtube.com/embed/9C06ZPF8Uuc")'>Lesson 3</button>&nbsp;
<button onclick='changeIFrame("https://www.youtube.com/embed/gbceqO8PpBg")'>Lesson 4</button>&nbsp;
<button onclick='changeIFrame("https://www.youtube.com/embed/J99NV9Cr75I")'>Lesson 5</button>&nbsp;
<button onclick='changeIFrame("https://www.youtube.com/embed/sHcLkfRrgoQ")'>Lesson 6</button>&nbsp;
<button onclick='changeIFrame("https://www.youtube.com/embed/H3g26EVADgY")'>Lesson 7</button>&nbsp;
<div class="video-container">
   <iframe id="classVideo" width="1280" height="720" src="https://www.youtube.com/embed/IPBSB1HLNLo" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> 
   <!-- List of videos in order -->
   <!--<iframe width="1280" height="720" src="https://www.youtube.com/embed/IPBSB1HLNLo" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> -->
   <!--<iframe width="1280" height="720" src="https://www.youtube.com/embed/JNxcznsrRb8" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> -->
   <!--<iframe width="1280" height="720" src="https://www.youtube.com/embed/9C06ZPF8Uuc" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> -->
   <!--<iframe width="1280" height="720" src="https://www.youtube.com/embed/gbceqO8PpBg" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> -->
   <!--<iframe width="1280" height="720" src="https://www.youtube.com/embed/J99NV9Cr75I" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> --> 
   <!--<iframe width="1280" height="720" src="https://www.youtube.com/embed/sHcLkfRrgoQ" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> -->
   <!--<iframe width="1280" height="720" src="https://www.youtube.com/embed/H3g26EVADgY" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe> -->
</div>
<div>
<script>
function changeIFrame(newURL)
{
    console.log("Changing URL to "+newURL)
    document.getElementById("classVideo").src=newURL;
}
</script>
</html>
11 Likes

@SMjameel,
If you want to build your own DL box from scratch, there are a number of blogs which explains it in detail. Two of them are the following:

  1. https://blog.slavv.com/the-1700-great-deep-learning-box-assembly-setup-and-benchmarks-148c5ebe6415

  2. https://towardsdatascience.com/build-and-setup-your-own-deep-learning-server-from-scratch-e771dacaa252

6 Likes

For the Part 2, I have made a resolution to try to advance the state of the art in deep learning by involving myself in the active research areas. I have a general direction in mind and a project that I’m working on right now.

I’m open to pursuing other interesting directions as well, and ready to expose myself to diverse ideas during brainstorming sessions/discussions with everyone in the community. :smile:

Anyone else with similar goals?

2 Likes

Probably @sermakarevich, that’s the impression I get.

Hey question - (not just for you but for other students who have taken previous versions of this course).

For a given week - What is the definition of “Done” ?

I find myself overwhelmed with the extensive discussions on the forums and a laundry list of ideas that I want to try for a given week. When should one move on from a given week to the next?

Is this definition of done - “Repro the Jupyter Notebooks on the given data set by understanding concepts” sufficient?

Thoughts?

Yup that’s about right - we say something similar here: http://wiki.fast.ai/index.php/How_to_use_the_Provided_Notebooks . Although the last sentence (repeat on a different dataset you have) is a great next step.

Everyone has different things they can add if they wish, such as writing a blog post about something they learnt, looking into the fastai source (and maybe adding docs), etc.

9 Likes

I ended up doing code-alongs. I’ll watch the lecture one day, then come back to it on another with a fine-toothed comb and build the notebook Jeremy uses, with lots of notes, and making sure I can get the same results so I have a baseline.

The wiki is usually set up by then, and that provides a good list of relevant readings/links.

After that, the goal is to implement the week’s material in a kaggle competition or some other project – and that leads to coming back a third time to the lecture for specific parts, and that’s when the why’s get answered.


That’s an ideal workflow, and it usually takes more than a week especially with new material.

7 Likes

As long as you have your Nvidia GPU configured right with CUDA and Cudnn and test it with Pytorch you should be good to go. I have a second-hand HP Elite minitower 8300 (About $150) with GTX 1070 8GB and it feels faster than cloud solutions. I used to have a cheap GTX 760 but Pytorch no longer supports it so after wasting lots of time I ended up buying the 1070. At this point time is money :wink:

3 Likes

Get a laptop with 1080ti. If you can’t afford that try 1070 or 1060 with atleast 6gb of vram. In the lecture jeremy has mentioned about it. He also recommended the new surface book.

Check out this out for more info…

For a desktop,

1 Like

My suggestion is to get a desktop, not a laptop, since you’ll be able to get a lot more powerful GPU for less money that way. Whilst I do like my Surface Book for training models when not connected to the internet, 99% of the time I’m training on a desktop, and it’s much faster.

Perhaps the most important thing to look for is plenty of GPU RAM.

5 Likes

To be honest and truthful, Actually I don’t know how to ask this, but I am actually quite overwhelmed by the content of part2v2(it’s like 10x tougher for me than the part1v2 and ML course). To be honest It’s making very little sense to me because of my little Knowledge .(Jeremy , Rachel, Yinnet and fast.ai folks are awesome no doubt at All)

How can I improve my understanding about what is happening and what will happen?

Sorry for being spammy but couldn’t find a suitable thread.

(Any suggestions will be helpful and Thanks for being helpful)

Thanks all for your inputs (see below comments)

13 Likes

Maybe just slow down ? No rush, just step by step get into every formula and every concept used. Take your own pace which is comfortable for you. There might be people around who work for years in this domain and some who just started a year ago.

6 Likes

Like @sermakarevich mentioned, slow down to a comfortable pace and make sure you understand key ideas before moving on to more complex ideas. Most of these ideas have taken years, even decades to come together. So, don’t sweat it, if you don’t get all of it right away within a 3 hour lecture. (I for one, surely don’t.)

The lectures, notebooks, papers etc. are available for free, so take your time and read the papers, rewatch the lectures, search for more info online, hand-on practice/experiments etc. until the ‘big picture understanding’ emerges.

And, don’t forget to enjoy the ride !

6 Likes

4 tips:
(1) Write down notes while re-watching lectures
(2) reproduce notebooks in the style described by @radek - Jeremy mentioned it in class
(3) learn topics on demand - videos, MOOCs, blog posts. Each fast.ai lecture is to be treated as a springboard into all the ideas/topics covered. Don’t be afraid to scour the web.
(4) try to teach others - this is very important as it solidifies your understanding

Don’t give up. It’s hard…but that’s what cutting edge is meant to be.

12 Likes

When a true master of a discipline performs an action, it seems effortless. Consider great pianists - oh how easy it seems to just sit in front of a piano, close your eyes, sway a bit and the music just flows. But then we go home and it doesn’t seem to be that easy.

Same with watching Jeremy. We see those outstanding notebooks incorporating research findings from many, many years and we think: ‘oh, this is cool and Jeremy makes it seem simple, nice!’.

What we don’t see is how much effort Jeremy - being an absolutely out of this world practitioner and teacher - put into creating the notebooks. I can only guess there have been a lot of set_trace calls involved and other development practices that we don’t see. And it is not like Jeremy has been doing this for a couple of months nor that fastai is the first library that he authored.

We see notebooks shared on the forums by students but we don’t see how much has been copied over nor how long it took us to piece things together.

Just to put things in perspective, I don’t get much of this either and I have given all this my best since last October. I’ve put countless hours into the fastai courses. This is from my last coding session:

  • I had to read how broadcasting works in numpy since I didn’t quite get it. BTW I think this comes from the book you shared on Twitter
  • I noticed in a notebook I worked on some time ago that I didn’t include a relu activation after every second layer! And I completely missed it. I spent good 20 minutes trying to figure out if it really wasn’t there and whether it should be there as the model is somewhat performing in line with expectations. Doah.
  • Reading (wrong version on top of that) of PyTorch docs to confirm if I understood what was it that they refer to as logits.
  • For the trillionth time in my life, googling for how to disable the axes when plotting images.

My other recent claims to glory involve:

  • Having to write an article so that I could semi-comfortable figure out how to use Twitter.
  • Having to write an article to convince myself to start using samples instead of training on the whole dataset and starting like a zombie at the computer screen.

There are a couple of things that could be happening. Maybe I am just not smart enough. Could be. Maybe I spend to much time on the forums instead of doing actual work. Could be.

Or maybe part of the reason why it is so tough at time is what @sermakarevich mentioned:

and more than that. We see the results of work of other people but we don’t see how much they struggle. And if they struggle even 1/4th as much as I struggle, than they struggle a lot. And I don’t think even people who perform - in my eyes - on a superhuman level place such emphasis on debugging because all of this comes easy. Don’t think it does even to them.

I think other people have given great advice here. All I wanted to say is that if you find this overwhelming you are not alone and to list some of the reasons why it all might seem like we are doing worse than others where we actually are not.

I find that for that what works best for me is playing with toy problems. If I can figure out how to train a fully connected network consisting of a layer or two on MNIST, maybe I can figure out how to train a conv net. And from that maybe I can figure out how to create a resnet.

If I can plot images of objects and bounding boxes (and I can now, because nearly most of the time I spent on lesson 8 went into this), maybe I can get the model to output 4 numbers and see if it learns anything. And if I can do this maybe over this week I will be able to implement something from what Jeremy covered in lecture 9. And if I don’t - oh well, maybe I will need another two months after this course to finish the material (I still haven’t finished part 1 cause I got sick). As long as I have not given up and I am moving forward even if at what seems to be as a snail’s pace, I think I am doing good. (BTW, I am not only writing these words to convince you only, but to also convince myself that all is well :slight_smile: )

27 Likes

That’s a great motivation.

Thanks a lot and i surely understand the pain Jeremy would have gone through to create a notebook which runs without complaining because I am self taught in this field completly.

I understood this pain when for the 1st time in my life I had participated in a hackathon at Analytics-vidhya and won it…(it took me 4 days to build a model i.e a nbs which ran completely).

But it would be great if Jeremy reveals his scratch notebook…

Thank you all again, I muster some of my lost confidence back …

Enjoy Deep Learning…

2 Likes

That is quite an achievement, congrats! :slight_smile:

1 Like

Follow 3 steps of eternal learning approach. Here are the steps

  1. Listening - Why - To address “I don’t know” - Get all inputs without any filters
  2. Reflection - Why - To address “I don’t understand” - At this step try to get all your doubts cleared / understand from all aspects
  3. Contemplation / Mediation - Why - To address “I understand but I don’t have any experience” - To clear opposite understanding and you have become one with knowledge.
    If it in not your understanding / experience - Repeat above steps till you and knowledge have become one.
2 Likes

This is a great discussion. It took me months to create this notebook. Not months of progress, but months of continuous failure. The difference between those who succeed and those who don’t is the ones that succeed didn’t give up! :slight_smile:

Also, I spend at least 50% of every day learning and/or practicing new things, and have done so since I made that commitment at the age of 18. (Nowadays it’s about 90% of my day). I don’t watch TV or play computer games or get lost in social media so I maximize the time I spend on things I care about. So over time I’ve gotten faster at doing stuff, since I’ve been practicing and learning lots.

I don’t have a “from scratch” notebook to share, since I’m continuously refactoring and experimenting within one notebook, and what you see is the result. But @radek has done a good job in earlier posts documenting some of the kinds of tools and approaches I use.

One of our MOOC students, Louis Monier, is known as the “father of internet search” - he was the CTO of the first big web search company (Alta Vista). He’s a pretty smart guy! He told me he had watched the part 1 (2017) videos so many times that he knew much of them off by heart. He also practiced on a home deep learning project whilst watching the videos, and IIRC he spent something like 6 months working on that. If Louis needs to study this much, then the rest of us should probably expect to work even harder if we want to master the material.

Today, Louis is the head of the AI Lab at AirBnB. So I guess the hard work can pay off… :wink:

71 Likes

Since we are on the topic of learning in this thread, I wanted to share this with everyone here - without creating an all new thread for it. @ecdrid I think this will also provide some good source of motivation for you!

This is an interview just from yesterday with Pytorch creator Soumith Chintala - he is also a Facebook AI Researcher. If anyone here has the time, I definitely highly recommend watching it as he had some really good insights. BTW the two who interviewed before and after him (Eric Schmidt and Marc Andreessen) also had some pretty awesome things to say.

Some key takeaways:

"

How did he get started in Deep Learning & Neural Networks?

  • Soumith went to study at NYU because he found out about Yann Lecun (after doing a google search). Ended up meeting Yann (simply by emailing him and asking! :slight_smile: ). Soumith admitted to Yann that he knew nothing about neural networks so Yann offered help with one of his Phd students. When Soumith graduated from NYU he couldn’t find any jobs in DL since it wasn’t really a “thing” at the time and positions for DL engineers didn’t exist. Later he realized that Yann had co-founded a startup and was using neural networks so Soumith ended up initially working there. Eventually, Yann was recruited to head up the FB AI Research Lab and he invited Soumith to join him (once again). - KEY TAKEAWAY: Don’t be afraid to admit when you don’t know something (be honest with yourself and others - don’t pretend to know) and most importantly don’t be afraid to ask for help! All it took was Soumith taking that first step in emailing to Yann that kickstarted his whole career path and eventually creating Pytorch. Of course, this is not to say that he has most definitely earned everything that he has gained today and must have worked extremely hard to get to the level he is currently at. :sweat_smile:

What does he recommend for people who want to start a career in this field?
You don’t need a 4 year education to learn about and become good at deep learning and neural networks (high school math level (as jeremy loves to say :slight_smile:) . You definitely don’t need a Phd. No-one on the Pytorch team had a Phd and they still turned out fine. However, doing a course is not enough, you have to apply what you learn. That means creating Github projects, writing blog posts, doing Kaggle competitions, reading & implementing research papers. Don’t just read papers or consume information, you have to put it in practice and also you have to keep your skills sharp. (Oh wait that sounds a lot like what we are doing here at Fastai :slight_smile:) , etc. As a result of doing all this, you are sending signals to others that you have the skills they need. Other people won’t know what you know unless you show them.

What projects is he most interested in?
He likes generative models a lot - GANs

What do you think is the future of this field?
The problem right now is that we still need a lot of data for the neural networks to do well in most situations. One thing we really have to figure out is how are humans able to generalize like we do and pick up on things so quickly with so little data. Is it transfer learning, that humans can transfer all of their skills to generalize to new tasks with very little data? Is the answer that we all should just be doing unsupervised learning where nothing is labelled and neural net should just be able to figure things out on its own? This is all research that is being worked on.

tfw you just realized you wrote a @radek post :slight_smile: (btw your posts are legendary:+1:)

30 Likes