I’m attempting to use Lesson 4 principles on an old (2012) Kaggle competition. Specifically, I’m training a model to determine whether or not a user would have clicked on a given ad. The training file has over 140million rows, and I’m able to load all of them into a data-frame. However, I run out of memory when attempting to run learning rate finder (i.e., data loads fine in regular memory, but too much for GPU)
So I loaded only 10 million rows at once and was able to go through training end-end. Thinking batches of 10m rows at a time would be a safe bet for training my model, I loaded 20 million rows at once and attempted to train the model on the first 10m rows. However, I get an out of memory error (I also tried adjusting the batch down to only 1million rows, but still get an out of memory error). It’s as if the model attempts to load all 20m rows into the GPU’s memory anyway, instead of only the selected rows that I pass into it.
Would greatly appreciate any help (notebook attached).
kddCup2012.pdf (56.9 KB)