Platform: Colab ✅

This works well. Where do you store all this data from sources when you are running out?

Can this be used to store the kaggle data sets on google. How do you direct them to the drive?

Yes this is possible. First you download the kaggle dataset to your local machine. From there you upload it to your Google drive.

If you mount the drive as described in my post, you can access any folder that is part of the drive. Since your training data is part of the drive you just have to figure out the path. After you mounted your drive You can open the tab on the left (in Colab) to browse your files in the filetree. Let me know if you need help setting this up :slight_smile:

1 Like

My 2 cents, one thing I’ve found is if you instead copy the download link when you download the dataset and wget it in colab, it will typically download much faster for me (since it uses google’s servers) and then mv or cp bash command the folder to your google drive If you want or need a visualization of this I can post that later on today.

You’re right. Loading directly from the drive is really slow. I’ve been working with univariate timeseries lately. They’re so small in size that I kind of forgot about that problem…

Copying the data from drive to the notebook (basically what you just described) works though.

Problem - Colab session timing out after 12 hours model requires 20 hours what is the solution?

Hi everyone hope all is well!
In the snippet below takes 1 hour 55 minutes to complete. (with GPU enabled)

The snippet learn.fit_one_cycle(10, 1e-3, moms=(0.8,0.7)) below takes approximately 20 hours. Google Colab resource limit is set at 12 hours, which means the session is terminated before its finished. (frustrating :frowning_face:)

Any Idea’s how to resolve this problem?

Reading the forum possible solutions could be:

  1. checkpoints
  2. callbacks
  3. run learn.fit_one_cycle(10, 1e-3, moms=(0.8,0.7)

    saving and reloading after each epoch. I haven’t got my code to work so far.

Any solutions, tips or ideas greatly appreciated.

mrfabulous1 :smiley::smiley:

Hey @mrfabulous1! I’d do checkpoints and model saves every x epochs. And save that save into your google drive. Else usually for LM’s I go to paperspace for a few hours…

1 Like

Cheers muellerzr I will learn checkpoints tomorrow its 4:30am here. Zzz
Many Thanks mrfabulous1 :smiley::smiley:

1 Like

Problem - Colab session timing out after 12 hours model requires 20 hours what is the solution?

To solve this problem I have created command using callbacks To test it works I have used the multilabel example from

learn = cnn_learner(data, models.resnet18, callback_fns=[CSVLogger])
learn.fit_one_cycle(30,1e-2, callbacks=[ShowGraph(learn), SaveModelCallback(learn, monitor=‘train_loss’, mode=‘min’, name=‘mini_train_30_best_model’)])

The output of the command above is shown below, the command saves a list of the epoch results to a csv file and a model is also saved to a file after every epoch.

How can I change the command above to stop at epoch 28 when the training loss is less than the validation loss ?

I have tried using other values such as error_rate but I get the following error, and am not sure how to change the command to achieve the result I require.

/anaconda/envs/fastai_uvicorn_0_7_1/lib/python3.6/site-packages/fastai/callbacks/ UserWarning: <class ‘fastai.callbacks.tracker.SaveModelCallback’> conditioned on metric error_rate which is not available. Available metrics are: train_loss, valid_loss warn(f’{self.class} conditioned on metric {self.monitor} which is not available. Available metrics are: {", ".join(map(str, self.learn.recorder.names[1:-1]))}’)

Thanks in advance mrfabulous1 :smiley::smiley:

Typically it is better to save the model with the best valid_loss. Then, with early stopping, it seems like you would be done much earlier.

1 Like

Cheers ilovescience I changed the command to monitor valid loss and removed min and it now stops at epoch 13.

Thank you very much!

mrfabulous1 :smiley::smiley:

1 Like

Need some help with lr_find function on Colab. I am trying to do lesson 2 of the fast ai course -
This involves picking up urls of images from the internet, and running a cnn to categorize it.

I picked 3 items - forks ladles and spoons (attached). I ran the download images, stored and verified them. This is all fine.

Ran fit one cycle cnn through it, which gave some numbers.

The next steps ask you to unfreeze the model, and run a learning rate finder through it. Here is where I get stuck. Instead of giving me some numbers and a graph it gives me #na#.

I’ve tried below with start and end lr.

I’ve also removed those parameters, and tried just lr_find()

Please help :slight_smile:

1 Like

It’s working fine. Answer here:

1 Like

You should do learn.recorder.plot() as well to see the graph

1 Like

Thanks a bunch @ilovescience! I’ll try again.

So I finished setup for colab successfully but the Practical Deep Learning for Coders, v3 says that after setting up I should see a Jupyter notebook like this

But I don’t. Am I missing something?

Hi maverick891 hope your having lots of fun today!

Your notebook will look like this:

You then have to do file > open notebook and you will get the following screen.

As far as I am aware I have I only see the screen you have shown when I run fastai on my desktop.

This is the link I have used for all the lessons in fastai part 1 V3.

mrfabulous1 :smiley::smiley:

Hi everyone

I am a Data Science intern from a non Computer Science background so apologies for the noonish questions.

Does anyone have a detailed tutorial on how to follow the download.ipynb notebook using Google Colab??

I’m trying to follow the notebook and after I’ve entered the given JavaScript code snippet in the first step, the console returns “null” not too sure if that’s supposed to happen?

And even if I am on the right track I’m not sure where the images are stored with the regards to using/accessing them in Colab since it’s a different server.

Any help would be greatly appreciated.


There is an easier way to download google images using the google-images-download python module in your jupyter notebook.
Here are the steps to download the different bears images

First install the python module. In a cell, enter the following command
!pip install google_images_download
Then in the subsequent cells, start downloadig the grizzly bears (-k grizzly argument below)
!googleimagesdownload -k grizzly -l 100
A new folder called downloads/grizzly will be created with 100 images in it (that’s the -100 argument n the command above)
Do the same thing for the other categories
!googleimagesdownload -k ‘black bear’ -l 100
!googleimagesdownload -k ‘teddy bear’ -l 100

Run the ls command
!ls downloads/
and you should have the 3 following folders
‘black bear’ grizzly ‘teddy bear’

If you are using the lesson2-download, you can update your path like this (assuming you are using Colab)
classes = [‘grizzly’, ‘black bear’, ‘teddy bear’]
path = Path(’/content/downloads’)

The cell output should look like this
PosixPath(’/content/downloads/black bear’),
PosixPath(’/content/downloads/teddy bear’)]

From there, you can just continue executing the rest of the cells.

If you want to keep the original folder names (used in lesson2-download.ipynb) you can rename the new folder names listed here above to match those of lesson2-download.ipynb notebook.


1 Like

Dude Thank You so much! The help is much appreciated man :pray:t4::pray:t4::pray:t4: