An overview of gradient descent optimization algorithms

FYI I’ve just found what appears to be a pretty good survey of gradient descent optimizers. It’s written in fairly plain language and does a descent job of discussing differences and suitability for various applications. I have not finished reading it, but I was thrilled to find it and thought I’d share it ASAP.

1 Like

I also liked that page! Note that it is linked from the wiki’s SGD page already:

Excellent, glad it’s on the radar. I had the presence of mind to search the forum for mention of it, but didn’t think to search the wiki. Oh well, best laid plans… :slight_smile: Cheers.

Hi all, I’ve written a similar (hopefully more digestible) article on 10 Gradient Descent Optimisation Algorithms, and compiled them into a cheat sheet. Hopefully you will find it useful too. Let me know if you have any feedback!