Learn.fit_one_cycle makes me run out of memory on CPU while I train on GPU

Weird finding, when setting num_workers to 0, the RAM usage stays constant during the early batch (up to around batch 70), and then goes up like before.