How to Read Research Papers (Andrew Ng)

Part 2 (2019)
1

During FastAI part 2 we have to read many research papers. I’ve personally found this a bit of a chore, and I have lacked motivation. I googled around to see if there were ways to do this better. I found several blog posts and youtube videos. The best I found was a great lecture from Andrew Ng (Stanford, deeplearning.ai):
Career Advice / Reading Research Papers [YouTube]

I made a summary of it, which I will share here:

How to Read Research Papers

  1. Take multiple passes through the paper
    • Worst strategy: reading from the first word until the last word!
  2. Read the Title/Abstract/Figures
    • Especially in deep learning, there are a lot of papers where the entire paper is summarized in one or two figures.
    • You can often get a good understanding about what the whole paper is about without reading much of the text.
  3. Read the Introduction/Conclusions/Figures (again)/Skim Rest
    • Part of the process of writing papers is convincing the reviewers that your paper is worthy of acceptance, so you find that the abstract, intro, and conclusion are where the authors summarize their work really carefully. These are therefore the most useful parts to read.
    • Neural network architectures are often written up in a table.
    • Maybe also skim Related work section, for context or see if there is something you have read before.
  4. Read the Paper, but skip the maths
  5. Read the Paper, but skip parts that don’t make sense
    • In cutting edge papers we don’t always know what is really important and what isn’t important.
    • Some great, highly cited papers have some parts which are groundbreaking and other parts which later turn out to be unimportant, but at the time the paper was written the authors could not know.
    • Maybe what was the key part of the algorithm wasn’t what the authors thought.

Questions to Keep in Mind

  1. What did the authors try to accomplish?
  2. What were the key elements of the approach?
  3. What can you use yourself?
  4. What other references do you want to follow?

Deeper Understanding

  1. Maths
    • If you want to make sure you understand the maths of a paper, read through it and make some notes then try to re-derive from scratch on a blank piece of paper.
    • As you get good at doing this you will gain the ability to derive novel algorithms yourself.
    • Learn from the masters, not from their students.
  2. Code
    • Lightweight: download and run their open-source code (assuming they have it).
    • Deeper: reimplement their code from scratch.

General Advice: Steady reading, not short bursts. Better off reading 2-3 papers a week for the next year, than cramming everything in over Christmas.

Where to Find Papers

To keep up with the state of the art:

  1. Twitter
  2. ML Subreddit: r/MachineLearning
  3. Top ML Conferences: NeurIPS, ICML, ICLR
  4. ArXiv Sanity
  5. Friends / Online Community (e.g. fastai forums).

Other Stuff

  • This older thread from fastai has great info too: Reading deep learning papers
  • I’ve been using Mendeley for organizing and reading papers on my computer and iPad (it syncs across devices).
29 Likes
2

This is awesome summary. Thank you.

1 Like
3

Thanks for sharing, very helpful summary!

1 Like