I’ve been working through all the lessons in Colab, and haven’t set up Clouderizer.
Here’s my notebook explaining how to use Colab for Fastai (including importing data from various places, which is where I’ve had the most trouble so far!). I’ll keep adding cells for each lesson as I work through it.
The basics:
Always add a cell to the top of the notebook like this:
!pip3 install fastai
How to import data from fastai URLs (lessons 1, 3, 4):
For lesson 1:
# Get the Dogs & Cats data, unzip it, and put it in the 'data' directory:
!wget http://files.fast.ai/data/dogscats.zip && unzip dogscats.zip -d data/
# Check to make sure the folders all unzipped properly:
!ls data/dogscats
For lesson 3:
# Get the Rossmann data and make a directory to put it in:
!wget http://files.fast.ai/part2/lesson14/rossmann.tgz && mkdir -p ~/data/rossmann
# Unzip the .tgz file, and put it in the right directory:
# x for extract
# -v for verbose # NOTE: I usually turn this off; it prints a lot...
# -z for gnuzip
# -f for file (should come at last just before file name)
# -C to extract the zipped contents to a different directory
!tar -xzf rossmann.tgz -C ~/data/rossmann/
# Make sure the data is where we think it is:
!ls ~/data/rossmann
For lesson 4:
# Get the IMDB data:
!wget http://files.fast.ai/data/aclImdb.tgz
# Unzip the tgz file, and put it in the right directory:
# x for extract
# -v for verbose # NOTE: I usually turn this off; it prints a lot...
# -z for gnuzip
# -f for file (should come at last just before file name)
# -C to extract the zipped contents to a different directory
!tar -xvzf aclImdb.tgz -C data/
# Make sure the data is where we think it is:
!ls data/aclImdb
How to import data from Kaggle using the Kaggle CLI (lesson 2):
I found this forum post very useful.
# Install the Kaggle API
!pip3 install kaggle
# Import kaggle.json from Google Drive
# This snippet will output a link which needs authentication from any Google account
from googleapiclient.discovery import build
import io, os
from googleapiclient.http import MediaIoBaseDownload
from google.colab import auth
auth.authenticate_user()
drive_service = build('drive', 'v3')
results = drive_service.files().list(
q="name = 'kaggle.json'", fields="files(id)").execute()
kaggle_api_key = results.get('files', [])
filename = "/content/.kaggle/kaggle.json"
os.makedirs(os.path.dirname(filename), exist_ok=True)
request = drive_service.files().get_media(fileId=kaggle_api_key[0]['id'])
fh = io.FileIO(filename, 'wb')
downloader = MediaIoBaseDownload(fh, request)
done = False
while done is False:
status, done = downloader.next_chunk()
print("Download %d%%." % int(status.progress() * 100))
os.chmod(filename, 600)
# List the files for the Planet data
!kaggle competitions files -c planet-understanding-the-amazon-from-space
# Download the data from Kaggle
# -c: competition name
# -f: which file you want to download
# -p: path to where the file should be saved
!kaggle competitions download -c planet-understanding-the-amazon-from-space -f train-jpg.tar.7z -p ~/data/planet/
!kaggle competitions download -c planet-understanding-the-amazon-from-space -f test-jpg.tar.7z -p ~/data/planet/
!kaggle competitions download -c planet-understanding-the-amazon-from-space -f train_v2.csv.zip -p ~/data/planet/
# In order to unzip the 7z files, need to install p7zip
# This was helpful: http://forums.fast.ai/t/unzipping-tar-7z-files-in-google-collab-notebook/14857/4
!apt-get install p7zip-full
# Unzip the 7zip files
# -d: which file to un7zip
!p7zip -d ~/data/planet/test-jpg.tar.7z #-oc:/data/planet
!p7zip -d ~/data/planet/train-jpg.tar.7z #-oc:/data/planet
# Unzip the .tar files
!tar -xvf ~/data/planet/test-jpg.tar
!tar -xvf ~/data/planet/train-jpg.tar
# Move the unzipped folders into data/planet/
!mv test-jpg ~/data/planet/ && mv train-jpg ~/data/planet/
# Unzip the regular file
!unzip ~/data/planet/train_v2.csv.zip -d ~/data/planet/
# Make sure everything looks as it should:
!ls ~/data/planet/
Finally, if you’re worried about how much of the GPU is available, there’s a cell you can run that checks the % utilization of your current GPU. See the Stack Overflow link that Sebastian posted earlier on in this thread.
Hope this helps some of you get started more quickly!