Have you ever wanted to use nbdev on your existing libraries? Normally you need to rebuild from the ground up, copy all the source code to notebooks, add tags, etc etc. lib2nbdev aims to fix that!
What does it do?
Through a simple CLI, you will generate nbdev’s settings.ini file, and then your entire library will be converted into an nbdev one, with the proper tags, notebooks generated into a nbs folder (or any you specify), and optionally you can include the GH CI that nbdev utilizes as well.
And you are done! No headache, no needing to go and modify any files, everything is automatically done for you. Even private and public functions are given the right declarations:
The last and final step you need to do is go and make titles and descriptions on your notebooks (as of course I can’t figure that out for you )
I hope people find this tool valuable for trying to bring nbdev into their organizations or existing projects.
Feel free to use this thread for questions or debugging help
The documentation is available here:
And the source code as well:
VERY big thank you to @jeremy, @sgugger, and @hamelsmu for not only helping me with answering all my various nbdev questions, but also for releasing nbdev in the first place. It’s a magical software
I converted over the entirety of our AdaptNLP project
Most of my time was writing more tests, as since Jupyter is all right there, it forced me to not be stingy. Didn’t feel right adding in documentation when there weren’t any tests wrapping around with that.
(And also don’t have the capability to automatically shove pytest tests into jupyter, etc)
Otherwise in regards to productionizing, absolutely nothing. We’ve fully converted over the library and are about to push a new release, but nbdev has little (per se) to do with said release, I just added some things as I was going through it.
@KevinB I believe also used it recently. I have a few issues open of where I (and others) have found some issue with the processes.
I’m not through the productionalization piece for my project. It definitely saved me time though. I had a bunch of functions that I would have had to copy and paste manually and instead lib2nbdev handled all of that automated work for me. I’m currently working my way through the output of lib2nbdev to refactor the code and write tests/documentation, but I would definitely use it again to get me to this point. It did create a few extra nbs that I had to clean up, but that was just because of how messy the folder was before. My code wasn’t technically a library but just a .py file with a bunch of functions so I would imagine it would work even better if it was going from a true library to nbdev environment.
I hadn’t considered the overhead of migrating existing tests. It does sound fairly straightforward to though. And I love the idea of taking the opportunity to rework at least some of them into working examples in the notebook.
Almost all of the use-cases I’m considering are actually not libraries either. Mostly deployables: a dozen microservices, a monolith, and one large spark-scala repository.
The spark-scala repository is the one that nbdev has helped with the most, even though it’s not actually in Python. Basically, what I’ve found is that leveraging sparkmagic and nbdev enables “exploratory programming” in Spark, and that’s been a much better development paradigm.
There are some hiccups to working this way. Since the final artifact is not a Python library, I need to copy-paste the final code out of the notebook and into the Scala repository. But that’s not very difficult. Then there is the overhead of getting started with each Spark job that’s not already ported in a notebook. That’s actually the main source of friction of working this way. lib2nbdev should make it easier to get the whole codebase bootstrapped and avoid that overhead each time.
Hi @muellerzr, I am new to nbdev and I feel at home to write codes and docs in my obsidian notes, but I also like to use nbdev eventually. So, I wonder could nbdev convert my MD files into ipynb files, so that I can take full advantages of nbdev some point later?
Thanks
)