I know there are a number of experiment tracking designs and services out there — I use neptune.ml for example. But I wanted something a little bit different.
TrainTracker is light weight. It only writes one line of CSV per experiment. (It can write it anywhere you want; I have it set up to add it to a Google spreadsheet.) It does not save files, data or other artifacts. One unique thing it does do is watermark model weights, so you can tell which experiment created the weights, without relying on filename. And it tracks the watermark an experiment started with, so you can reconstruct the sequence of experiments that lead to a particular result.
TrainTracker isn’t entirely self-sufficient. It needs to be integrated with
Learner (or something) needs to set some of the values that will be written out. In my code, I have a subclass of Learner that does these things.
You can find
TrainTracker in the infra directory of my project here. I’d love to hear any feedback on it.