Fastai2 vs pytorch-lightening ... pros and cons? integration of the two?

I keep seeing pytorch-lightening coming up more and more in articles I read, demos, and tweets from folks I follow. I don’t know much about it yet or how it compares to fastai2 (the only thing I’ve been able to find comparing the frameworks is over a year old).

So I’m curious for those experienced with the later:

  1. What are its pros and cons when compared to v2?
  2. Has there been any integration with the two that have proved helpful? For example, can fastai2 be used to prepare DataLoaders objects that can be fed into the pytorch-lightening training pipeline?
  3. What has your experience been in terms of training and deployment with the two?

Thanks - wg

3 Likes

Note that the article you mention is written by the creator of PyTorch lighting, so it’s, by definition, a bit partial.

1 Like

Ha … I didn’t notice. Makes sense now seeing how fastai only gets 1 checkmark on most tasks :slight_smile:

How does v2 compare with it?

Still would like to hear from folks using it and v2 to get their thoughts.

As @sgugger mentioned, the linked comparison biased. More importantly, the comparison is terrible and actually doesn’t understand what fastai is. Even when this was published (Aug. 2019), there are a lot of features that fastai has that weren’t “checked”.

Under High-performance Computing:
He missed:

  • Accumulated Gradients
  • Multi-GPU training
  • Gradient clipping
  • Tensorboard
  • Print param summary, inputs, outputs summary

Under Debugging tools:
He missed:

  • Built-in visualization
  • Log GPU usage
  • Overfit on small amount of data
  • Validate on subset of data

Under Usability:
He missed:

  • Checkpoint callback
  • Early stopping Callback

Honestly, the rest of the issues are either already solved in fastai2, or were already very easy to do with a little bit of extra code (except for multi-cluster training).

PyTorch Lightning looks like a great library and it’s very interesting, but I am disappointed by the very unfair comparison made in that post. In fact, if this comparison is done fairly today, there probably isn’t much additional features in PyTorch Lightning than fastai2. Maybe maybe someone who has actually used PyTorch Lightning could share their thoughts as well.

4 Likes

Will love examples of all that :heart:!

Check out the notebooks the library was built on in here: https://github.com/fastai/fastai2/tree/master/nbs

The summary, check learn.summary(), dblock.summary(). The rest are in callbacks (gradient accumulation, checkpoint, early stopping), logger is in there too in the other callback notebooks. And if you prefer an HTML, look at https://dev.fast.ai (the docs) and each notebook has been generated into a documentation webpage

2 Likes

Its been interesting to see the push Lightening has made recently, I think its partially related to people’s reluctance in spending time learning a whole new api vs dabbling with a more light weight one.

I haven’t used it, but from my (shallow) understanding, TPU support is the main thing it has that fastai v2 lacks (for now), for the rest I believe fastai has it more or less covered

Another very good framework is Nemo from Nvidia which includes very good free models for ASR and TTS.

Recently I have used fast v2 and pytorch-lightening in some small project. I find it that fastai2 is like a useful tools that you need to take time to learn it. And Pytorch-lightening is more like a way to organize your PyTorch code. If you want to start a model from scratch, I prefer fastai2 because it offers great tools to deal with dataset. If you want to modify others project for experiment, I prefer pytorch-lightening because I could have a better understanding about the project after move the project to pytorch-lightening.

The only thing that makes intermediate users prefer pytorch-lightning over fastai2 is pytorch-lightning is just plain python + pytorch, fastai2 comes with it’s own api design, and even after looking at the source code fastai2 still looks mysterious, fastbook has 1 chapter explaining the midlevel api, it would have been great if there are more chapters like it or similar resources explaining the api in detail.

1 Like

This from one of their comments on this intro video

  • Lightning is built for professional researchers using PyTorch.
  • fastAI is built for students for the fastAI course (with PyTorch).

Seems to trivialize fastai as a library fine for learning if you are taking the fastai course … but that’s about it. If you want to do real research, I guess you use PyTorch-Lightening or write everything out by hand??

Sounds a bit snarky imo.

2 Likes

I’d agree. I’ve done plenty of research with fastai (literally the last year and a half), and I also converted over a few professors at my university to use fastai instead of Keras for their research with their students

Yah … it seems like a shitty oversimplification that isn’t even factual. Kinda like a White House Press briefing.

1 Like

You can’t get impartial comparison from the person who built one of those libraries, which is why I’m personally refraining from commenting.
I’d be curious to have the feedback of @lgvaz since I think I saw on Twitter he used PyTorch Lightening for a project on object detection.

1 Like

Concerning TPU support and distributed training (across multiple machines and multiple GPUS) … where should I look in the fastai2 docs?

I don’t think the TPU support is there but I could be wrong.

TPU isn’t currently. Distributed: https://dev.fast.ai/distributed

1 Like

Ow booy, that is a tough spot you put me in @sgugger :rofl: :rofl:

Let me get used to lightning a bit more and I’ll share my thoughts here :smile:

3 Likes

I am a core developer of the unofficial fastai2 audio extension and also have recently experimented with pytorch-lightning in one project, so I’ll try to do a unbiased comparation of the two libraries.

First of all, note that they have different purposes. Lightning is a lightweight library focused on the training loop, and tries to make the engineering aspects (like logging or distributed/TPU training) trivial while giving full flexibility in the way you write your research code. On the other hand, fastai is a powerful framework that tries to integrate best pratices on all of the aspects of deep learning training, from data loading to architectures and also the training loop.

Pros about fastai2:

  • Modern best pratices are constantly implemented

  • Datablock API is wonderful to load data

  • Huge variety of applications ready to be used

Cons about fastai2:

  • The library is strongly opinated on how things should behave, down to the level of changing how python works. This introduces a huge friction when you want to do something new or different.

  • It’s hard to integrate with other libraries in the pytorch ecosystem. More than once I’ve seen people reimplement code in fastai2 because it’s easier.

  • Error messages in fastai2 have really degraded from fastai1. Often they are a couple of pages long and hard to tell where the problem actually is.

Pros about pytorch-lightning:

  • No friction at all to use with other libraries. Just import them and do your thing, no need to use wrappers or rewrite code.

  • Automatic experiment management is great, you just run the code a bunch of times with different hyperparameters and fire tensorboard to easily compare results.

  • Larger community of core and active contributors

  • Plain pytorch, code is simple to understand and reason about

Cons about pytorch-lightning:

  • Mixed precision training is currently tied to NVIDIA apex (waiting on official torch.amp to stabilize)

  • Could have better integration with hydra

4 Likes

Thanks for the detailed comment!

So in your PL project … how did you assemble/create your Datasets/Dataloaders? How did you export everything needed for inference, and how did you actually code things up for inference?

fastai2 takes care of most of those bits for ya, whereas PL seems almost purely focused on one thing = the training/eval loop; leaving the rest up to you.

To add to this, I remember someone mentioning his company was trying to decide to go pytorch or fastai v1 route. They ended up going with pytorch for easier maintainability (converting to fastai v2 down the road was an unknown). As said, each has pros and cons, and the user should figure out his/her needs.