Yet another experiment tracker

I know there are a number of experiment tracking designs and services out there ā€” I use neptune.ml for example. But I wanted something a little bit different.

TrainTracker is light weight. It only writes one line of CSV per experiment. (It can write it anywhere you want; I have it set up to add it to a Google spreadsheet.) It does not save files, data or other artifacts. One unique thing it does do is watermark model weights, so you can tell which experiment created the weights, without relying on filename. And it tracks the watermark an experiment started with, so you can reconstruct the sequence of experiments that lead to a particular result.

TrainTracker isnā€™t entirely self-sufficient. It needs to be integrated with Learner, and Learner (or something) needs to set some of the values that will be written out. In my code, I have a subclass of Learner that does these things.

You can find TrainTracker in the infra directory of my project here. Iā€™d love to hear any feedback on it.

denise

1 Like

Thank you very much for posting it. Will TrainTracker work with the new, upcoming version of fast.ai? Also is there an example or a notebook on how to use it when running experiments with some simple lessons like cats/dogs or dog breed classifications?