An overview of gradient descent optimization algorithms

(Matthew Walker) #1

FYI I’ve just found what appears to be a pretty good survey of gradient descent optimizers. It’s written in fairly plain language and does a descent job of discussing differences and suitability for various applications. I have not finished reading it, but I was thrilled to find it and thought I’d share it ASAP.

1 Like

#2

I also liked that page! Note that it is linked from the wiki’s SGD page already: http://wiki.fast.ai/index.php/Gradient_Descent#References

0 Likes

(Matthew Walker) #3

Excellent, glad it’s on the radar. I had the presence of mind to search the forum for mention of it, but didn’t think to search the wiki. Oh well, best laid plans… :slight_smile: Cheers.

0 Likes

(Raimi bin Karim) #4

Hi all, I’ve written a similar (hopefully more digestible) article on 10 Gradient Descent Optimisation Algorithms, and compiled them into a cheat sheet. Hopefully you will find it useful too. Let me know if you have any feedback!

0 Likes