Is there a way to save persistently intermediate results in Kaggle kernels?

Deep Learning
1

When working inside of a Kaggle kernel is there a way to save intermediate results somewhere in persistent way so that it will be not be destroyed between sessions?
In Google colab you can connect your Google drive and save/load the data there.
IS there something similar in Kaggle?

1 Like
2

Out of curiosity, for results do you mean cell outputs or you mean files generated by your program? Because I cannot find a way to save the cell outputs in google colab

3

I meant intermediate results that I wanted to save in a file. Since experiments may take a long time, and it is not fun if you need every time to repeat them from the beginning.

4

I have the same question. I was training some language model and I would like to save that language model in a pkl file to save time next time I am running the kernel.
I tried savinf on /kaggle/working, but that too is lost once the session is reset.

5

Probably you can download it locally and then insert as Private dataset to a new Kaggle kernel.

6

@darshak you could use the FileLink command to download intermediate results without committing.

7

Thanks @vladgets & @ilovescience.
I decided to pre-train my language model on colab since it is much easier to store the weights on Google Drive from there, and then imported that as a private dataset to kaggle.

Thanks!

8

@darshak By the way, output files are not lost in Kaggle kernel. If you commit a version then it runs all the code and stores results as output files which you can access later, by choosing this kernel version. And then you can download these files to your local machine and add to another kernel as a private data set.
I successfully did it in the last Kaggle NLP competition.
Also when you save your model, possibly you do not need to save everything. For example, if I do not save word embeddings and recreate them in the inference part from the test set it saves me a lot of space.

1 Like
9

Sure, I’ll try that.
Thank you.

10

yes when you commit it saves the output files with that commit. so you have input data, code and output files saves as one commit revision.
you don’t need to download the file you want to use in another kernel. from that kernel you can click on add data button and point to your kernels output and select the file.