In part two of the course, Jeremy mentioned the importance of having a logbook to track your experiments & results over time. I ran across a link on twitter to Facebook Research’s lab notebook for training their recently released 175B parameter language model and it’s a fascinating look behind the scenes of how research is done.
114 pages chronicling months of around the clock work for 9 people is here: metaseq/OPT175B_Logbook.pdf at main · facebookresearch/metaseq · GitHub