MLFlow tracking support

fastai fastai dev
1

Hi,

I have added support for MLFlow Tracking tracking experiments run, parameters as well storing the notebook used to run given experiment. This helps in comparing different runs with different configuration with easier GUI and FastAI callback functions make it easy to use it.

This does require adding MLflow as dependecy which user needs to have to use it. So let me know if this support is of interest to community in general. If yes, then how we should proceed, as creating PR template requires we agree on feature here before creating a PR.

12 Likes
2

@sgugger Here is an example notebook adapted from lesson 1 which shows as how to use mlflow for tracking. Remember to install mlflow and run mlflow server before running this notebook.

This will create view in mlflow ui available at http://localhost:5000 . Let me know if you have any comments/questions.

Experiment view

Screenshot_2019-02-22%20MLflow(2)
Screenshot_2019-02-22 MLflow(2).png2062×1083 126 KB

Compare runs
Screenshot_2019-02-22%20MLflow
Screenshot_2019-02-22 MLflow.png2536×1033 87.8 KB

Error rate in two runs
Screenshot_2019-02-22%20MLflow(1)
Screenshot_2019-02-22 MLflow(1).png2880×1630 99.7 KB

4 Likes
3

Hi Gurvinder,
Love how you’re using MLFlow with this project. We would love to help out and get your feedback on MLFlow. If you shoot me an email at web@databricks.com I can connect you with our team here to assist with any questions you may have. We can even let you trial run this on https://databricks.com/mlflow

Thanks and keep up the good work!

Sam
Databricks - The Unified Analytics Platform for AI and ML.

1 Like
4

@sgugger I have updated the code and created a separate file like Tensorboard integration and use the similar method to resolve optional dependency. Also updated the behavior to use this functionality. Now to use MLFllow tracking, user need to add this partial functional call

create_cnn(data, models.resnet34, metrics=error_rate,
                   callback_fns=[partial(MLFlowTracker, exp_name="lesson1-pets", params=params, nb_path="/data/course-v3/nbs/dl1/lesson1-pets-Copy1.ipynb")])

and recording of all parameters, metrics as well notebook on training end is taken care by new code.

3 Likes
5

Thanks Sam for the offer. But i am using a local installation of MLFlow and it serve my needs well.

2 Likes
6

Great job @gurvinder

If you want to go one step further and share your MLflow experiment runs with others you can have it hosted on neptune.ml (free tool unless you are a for-profit organization).

All you need to do is change mlflow ui to neptune mlflow.
For example I can share this experiment run link with all of you and let you access my model weights.

You can read more about in this blog post.

1 Like
7

Hi, I love this callback. however, I’m having some problems - all left site of run log (user, code version, source, run name stays empty… do you have any idea what could be the reason?
Selection_651

8

Hi, what is the state of this? Will it be merged? I’m really interested in having a callback for logging metrics on MLflow.

5 Likes