I am Thomas, I work at Weights and Biases. We wrote with @jeremy the hyperparameter search for fine-tuning the image models that he showed on the walkthrus. Sweeps are great to explore different training recipes and configurations, so…
I think it’s due time for an introduction to experiment tracking with W&B
I am planning on doing an intro to Weights and Biases for fastai users. The idea is to give students a simple walkthru on how to use W&B with fastai. If you want to start reading about W&B and fastai you can start with:
I wonder if Jeremy changed his opinion about experiments tracking; as (if I am not mistaken) it was mentioned during one of the lectures that sometimes it is too distracting and making you watching training curves instead of preparing next experiments. (Though I like W&B and used it quite a lot.)
Though I guess it depends on use-case. Running multiple experiments in parallel, i.e., like you did for picking the best fine-tuning architecture, is definitely a good idea.
Yes I still think that’s a danger. I don’t look at the runs while they’re happening or use any of the GUI features. I just used it for the same purpose as fastgpu, basically (i.e managing a list of jobs to run and putting them onto free GPUs as they’re ready). For doing sweeps over a cartesian product of configuration options it’s a bit easier to do in wandb than fastgpu at this stage.
I know a lot of folks really like the full experiment tracking system and it seems to be very well designed in wandb. It’s just not something that I have need of myself.
I’ll join if I can (from Canberra). A colleague introduced me to wandb a few months ago, but I have yet to use it fully. I am pretty sure some form of sweeps are on the card. I can be choosy about the level of logging/provenance I need depending on the task, but in this case I’ll need something like wand. Having written in-house metaheuristics frameworks logging system back a decade ago, I certainly am keen to use existing offerings this time around…
I would not say that Sweeps is our most powerful feature. it’s cool, but we have other more important features. As Jeremy said, we could have done this cartesian product of parameters with a couple of for loops. The async of the sweep controller (enabling you to run agents in parallel) can be very practical.
Most of the value I saw from W&B was in my previous company, when I needed to present analysis to managers/stakeholders that did not know anything about programming but understood the problem we were solving. Being able to share results embedded in a WYSWYG format was life-changing. I could stay on notebooks, and they could edit and present the data on a Report.
Hi Thomas,
Could you go through what you did for Jeremy and the setup in wand?
I published a paper previously. It is a standard question for reviewers to ask benchmarking our results against multiple architectures. Different journal/conference has a different preference for architecture. It was a time-consuming task. Also, is any logging system to list out the final output together with all the hyperparameters?
I am a beginner and would like to join in person. I am in Perth, Western Australia (GMT+8).
This is an important point – I don’t have this use-case myself, and it’s a critical foundation of what wandb is all about, so I’m not really a great person to comment on whether it does it well or not!
I have not fully explored this myself yet, but this sounds like a nice way of causing some mayhem on social media Presentation of results goes a long way!
Yeah, that’s a good point. Interesting enough that at my work, we mostly use DIY reporting solutions. I guess that some years ago W&B / Streamlit / etc. didn’t exist or weren’t yet that advanced. But I like the reporting features and dashboards provided by W&B. Would be great to see if this keeps evolving into even more flexible visualization toolkit.
I am planning on doing this intro on Friday, June 24, at 10am CET (18h Brisbane Time). I may do a second one for our US folks, later on.
It will be a chilled-out zoom call with time for questions and demos. My idea is not to overwhelm people with W&B features, just give some value straightaway for fastai users.
Presentation of results goes a long way!