FitLaM: What I've been working on recently


(Jeremy Howard (Admin)) #1

Figured you all might be interested in the paper I just put up on Arxiv - it’s basically the NLP classification approach I showed in lesson 4 of the new course, which Sebastian Ruder was kind enough to write up into a paper. :slight_smile: I’ve run the classifier on quite a few different datasets and got the state of the art on every one!

https://arxiv.org/abs/1801.06146


(helena s) #2

Sebastian and Jeremy are such no-nonsense guys, enjoyed the clarity of writing/presenting from both of you - looking forward to read the paper :star_struck:


(Jeremy Howard (Admin)) #3

Thank you! The paper really came out of me contacting him to tell him about lesson 4, and he was gracious enough to take the time to watch it…


(ecdrid) #4

It would be the first paper devoid of Mathematics at Large or uses Simple Maths

Thanks a lot !!!

Congratulations

Quite Reasonably Readable even by a beginner like me…


(sergii makarevych) #5

Congrats Jeremy.


(Vishal Pandey) #6

Congratulations… @jeremy .


(Sanyam Bhutani) #7

To the international fellows, It’s like watching an Award winning movie preview before the world release of it.

We got to know about it before anyone read the paper :smile:

Congratulations!


(WG) #8

This is the first arxiv paper I’ve read in it’s entirety AND completely understood (thanks of course to part 1v2 of the course).

Hope to see it make its way into part 2 v2!


(Vishal Pandey) #9

That’s true… I have had the same feeling…:grinning:


(Renato Hermoza) #10

Awesome :D. It would be interesting to put attention on top of a pre-trained model and check if seq2seq scenarios also benefit.


(Jeremy Howard (Admin)) #11

I’m so glad folks are liking the paper - Sebastian Ruder wrote nearly every word so that’s all thanks to him. Here’s a terrific technical writer.


(Kouassi Konan Jean-Claude) #13

Congratulations @jeremy !
I stay connected to test the code when you will make it available at a future time, according to the paper.
Thank you.


(Hafidz Zulkifli) #14

Was doing some reading on this subject when the FitLAM paper came out. Turns out it was exactly the stuff that I needed to be able to connect the dots in my narrative.

Below is a link to the end result. Hope you guys can a have a look and comment if there are gaps in my understanding.

“Understanding Learning Rates and How It Improves Performance in Deep Learning” @ikanez https://medium.com/@hafidz/understanding-learning-rates-and-how-it-improves-performance-in-deep-learning-d0d4059c1c10


(Sebastian Ruder) #15

Hey guys,

Appreciate the positive feedback and glad you like the paper. :slight_smile: fast.ai is doing an amazing job educating at the forefront of such a fast-moving field; it’s a pleasure to contribute to that. We hope you’ll find the ideas in the paper (those you didn’t already know about :wink: ) useful and hope they’ll inspire you to try out your own ideas. There’s so much untapped potential in transfer learning in general and transfer learning in NLP in particular!


#16

Thank you @sebastianruder :slight_smile: It is amazing to have you here on the forums and the paper is a wonderful read :slight_smile:

I am a big fan of your writing and two bookmarks on my chrome just below the URL bar lead to posts from your blog :slight_smile:

Quite amazing to have you stop by here :slight_smile:


(Himanshu) #17

Wow, @sebastianruder is also on this forum. I really like your blog and NLP newsletter. Thanks for making the information accessible.
Thanks for the great paper Jeremy.


(Jeremy Howard (Admin)) #18

This is quite exceptional. Congrats!


(Hafidz Zulkifli) #19

Thanks @jeremy. Wouldn’t have been able to compile all those without your work and this vibrant community.


(Even Oldridge) #20

@hafidz That was a very well written blog post. Thanks for sharing :slight_smile:


(maddogS) #21

It’s great! Finally we have comparable tools for NLP as we do with images.

@jeremy When would you expect to have the code avail for us to play with? I am working on a actual/real-life problem for a startup and would love to take FitLam for a spin.

If it’ll be a while I might hazard a try to implement it myself. (I’ll learn a bunch even it my code don’t end up working.)