What is Limited Memory Broyden Fletcher Goldfarb Shannon in Vowpal Wabbit?

#1

"Note that L-BFGS was empirically observed to be superior to SGD in many cases, in particular in deep learning settings"from here

I just discovered Vowpal Wabbit and I am looking into its underlying parts and they talk about the L-BFGS optimization method.

Do any of you smart people have resources or an explanation on this for us lesser mortals?

My feeble understanding thus far is:

!= Newtonian approximation but QuasiNewton ???

The L-BFGS algorithm, named for limited BFGS, simply truncates the BFGSMultiplyBFGSMultiply update to use the last m input differences and gradient differences.

What? Anyone?

0 Likes