Incoporating databricks notebooks into nbdev

nbdev
1

I’m working within databricks as a part of my company’s policy, but want to try to use nbdev as a platform. Is there someway to convert cells in databricks that look like this:

# Databricks notebook source
# MAGIC %run ./machine_learning_utils

# COMMAND ----------



# COMMAND ----------

column_names =['pmid', 'journal', 'title', 'abstract', 'keywords', 
               'label','pub_type','authors','date1','doi','date2',
               'label_category']
text_columns = ['title', 'abstract']
categories = ['Case Report', 'Diagnosis', 'Epidemic Forecasting', 
              'General Info', 'Mechanism', 'Prevention', 
              'Transmission', 'Treatment']
ds = load_dataset('csv', skiprows=1,
                  column_names=column_names,
                  data_files={'train': 'train.csv',
                              'valid': 'valid.csv',
                              'test': 'test.csv'})

# COMMAND ----------

i.e. cells separated by # COMMAND ---------- lines into standard jupyter notebook cells that can be processed easily by nbdev?

6 Likes
2

Looking for support NBDEV with databricks notebooks too!

1 Like
3

Just FYI, you can choose the ipynb file format in Databricks. I think you need to do something like File → Export or Format or something and select the ipynb format. This exports the file like a Jupyter notebook indeed.

However I haven’t figured out how to get nbdev really to work with this. It appears to me that Databricks Notebooks have their own format (in fact I think the notebooks don’t have a .ipynb file extension, even if you do the above steps) and the only thing that you can control, is the format of the exported file. I haven’t been able to find a way for nbdev_clean and all those nice git hooks for git-friendly notebooks to work on (Azure) Databricks. Very frustrating.

But thanks for the question. Perhaps someone else figures it out and let’s us know how to do it.

4

I haven’t used databricks myself. Would it be possible to write a little tool that converts between these file formats?