Lesson 3 In-Class Discussion ✅

Archaeologist · January 9, 2020, 9:16pm

Hi Peter I am using LIDAR images which sounds similar. Height as greyscale works just fine. Just 40 training images sounds challenging though. I use 13,000 and got 92 percent. Consider getting more samples.

peter.ahlstrom · January 10, 2020, 5:43am

Thank you!

olgias · January 17, 2020, 12:16pm

Hi William,

is the second part of the unet the reverse of the CNN of the first part?

Cheers.

wdhorton · January 18, 2020, 1:22am

Functionally it is meant to “reverse” the effect of the first part, which reduces the image dimensions, but the architecture may not be the exact reverse. Like if you adapt resnet34 for the first part, your second part will not be the exact reverse of resnet34.

tarun98601 · February 18, 2020, 4:25pm

Apologies if there is a similar question out there, can someone point me to a planet like dataset ? Want to try out my knowledge on another dataset.

andrew77 · March 17, 2020, 9:10am

Hi all,

Houston we got a problem. The train files for the kaggle planet dataset are non downloadable already, we can torrent if ourselves or you can follow the steps.

I hope this is useful for colab users it should work for the rest of the peeps ( minus those colab stuff )

https://pastebin.com/K9q2c7gY

sja · March 21, 2020, 5:03am

I had the same issue with the CamVid dataset as well.
This worked like a charm! Thank you for posting!!

beck · April 10, 2020, 7:53am

Hi @tarun98601 . There is a similar dataset
here

Its a movie genre classification based on the poster.

jin_and_tonic · April 14, 2020, 9:47am

Hi guys, I’ve been trying to torrent the planet datasets for two days - but seems like there aren’t any seeders available!

If anyone has any alternative download locations, I’d be grateful. Cheers!

avenio · April 24, 2020, 3:05pm

Can someone please explain me that in the head-pose lesson, while we are creating the data bunch, we are writing something like ‘split_by_valid_func(lambda o: o.parent.name==‘13’)’ , so how the lambda function is working here to create a validation dataset for the model?
Thanks in advance.

satyartha · April 27, 2020, 8:22pm

Hi @jin_and_tonic, Were able to download the file? I have the same problem, no seeders available.

satyartha · April 29, 2020, 11:01pm

Hi Folks, Hope y’all are doing well.

I have this following metrics from training a learner on the BIWI head pose dataset. My training loss is higher that validation loss. I tried to increase the learning rate
(from 2e-2 to 8e-2 ), still behaves the same. Can someone comment on this behavior.

Thanks,

saurabh_wadhawan · May 3, 2020, 7:22am

/bin/bash: move: command not found

I am getting this error when I try to execute the following command from Planet competition code in lesson 3:

! mkdir %userprofile%.kaggle
! move kaggle.json %userprofile%.kaggle

What is the issue? Could anyone please help to resolve?

mrfabulous1 · May 3, 2020, 12:29pm

Hi saurabh_wadhawan I hope you are having a beautiful day!

Have you tried mv instead of move

Cheers mrfabulous1

saurabh_wadhawan · May 15, 2020, 5:47pm

Thanks. This worked!

But facing another error now.

! kaggle competitions download -c planet-understanding-the-amazon-from-space -f train-jpg.tar.7z -p {path}
! kaggle competitions download -c planet-understanding-the-amazon-from-space -f train_v2.csv -p {path}
! unzip -q -n {path}/train_v2.csv.zip -d {path}

When I execute these, I get this error:-
Could not find kaggle.json. Make sure it’s located in /root/.kaggle. Or use the environment method… But I have already uploaded kaggle.json

Also, this command doesn’t work:
! conda install --yes --prefix {sys.prefix} -c haasad eidl7zip
Error: /bin/bash: conda: command not found

Kindly help to resolve. Thanks.

yannklein · May 20, 2020, 5:46am

Can you try to run only the non-windows commands:

! mkdir -p ~/.kaggle/
! mv kaggle.json ~/.kaggle/
# For Windows, uncomment these two commands
# ! mkdir %userprofile%\.kaggle
# ! move kaggle.json %userprofile%\.kaggle

I believe it should solve your problem.

AdityaG · May 22, 2020, 5:07pm

Hi, I’m having trouble creating a databunch for image classification. The dataset is called MIT indoor scenes.
The images are under folders which contain the labels. But the problem is that the file names for which should be taken as train and test set is in a text file. How do I create the data bunch? Please help.

mainak · May 24, 2020, 1:32pm

Hi All,

I have tried to do multi-label image classification on the dataset below:

While training for I am getting negative training and validation loss. What is could be going wrong here? I am running on collab.

Thanks
Mainak

The code is available here:

github.com

mac-ban12/ai-ml/blob/master/style_color_multi_label.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"colab":{"name":"style_color_multi_label.ipynb","provenance":[],"private_outputs":true,"collapsed_sections":[]},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"ys8hywu84ENf","colab_type":"text"},"source":["## Prediction with Style-color-size dataset - Multi label classification "]},{"cell_type":"markdown","metadata":{"id":"HdzC5Uod5Lri","colab_type":"text"},"source":["I have made the database of photos sorted by products and brands. Screenshots were performed only on official brand websites.\n","\n","The main dataset (style.zip) is 2184 color images (150x150x3) with 7 brands and 10 products, and the file with labels style.csv.\n","Photo files are in the .png format and the labels are integers and values.\n","\n","Above para from - https://www.kaggle.com/olgabelitskaya/style-color-images  \n","\n","kaggle datasets download -d olgabelitskaya/style-color-images \n","\n","https://github.com/Kaggle/kaggle-api - reference for kaggle cli"]},{"cell_type":"code","metadata":{"id":"A_K3SqTd4ENh","colab_type":"code","colab":{}},"source":["%reload_ext autoreload\n","%autoreload 2\n","%matplotlib inline"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Bt52J1ny5CPr","colab_type":"code","colab":{}},"source":["# Set up fastai for collab \n","!curl -s https://course.fast.ai/setup/colab | bash\n","!pip uninstall torch torchvision -y\n","!pip install torch==1.4.0 torchvision==0.5.0"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"PvZjiimq5SVg","colab_type":"code","colab":{}},"source":["#from google.colab import drive\n","#drive.mount('/content/gdrive', force_remount=True)\n","#root_dir = \"/content/gdrive/My Drive/\"\n","#base_dir = root_dir + 'fastai-v3/'"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Iu2HXaoj4ENn","colab_type":"code","colab":{}},"source":["from fastai.vision import *"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"yveC6Fs14ENr","colab_type":"text"},"source":["## Getting the data"]},{"cell_type":"markdown","metadata":{"id":"VrgxRSdJ4ENs","colab_type":"text"},"source":["First, install the Kaggle API by uncommenting the following line and executing it."]},{"cell_type":"code","metadata":{"id":"YW2AqX7u4ENt","colab_type":"code","colab":{}},"source":["! pip install -q kaggle  "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"fXDY4aFD4ENw","colab_type":"text"},"source":["Then you need to upload your credentials from Kaggle on your instance. Login to kaggle and click on your profile picture on the top left corner, then 'My account'. Scroll down until you find a button named 'Create New API Token' and click on it. This will trigger the download of a file named 'kaggle.json'.\n","\n","Upload this file to the directory this notebook is running in, by clicking \"Upload\" on your main Jupyter page, then uncomment and execute the next two commands (or run them in a terminal). For Windows, uncomment the last two commands."]},{"cell_type":"code","metadata":{"id":"7FJURRkgEGci","colab_type":"code","colab":{}},"source":["from google.colab import files\n","\n","files.upload()"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"JIz-NFFJ4ENx","colab_type":"code","colab":{}},"source":["! mkdir -p ~/.kaggle/\n","! mv kaggle.json ~/.kaggle/\n","\n","# For Windows, uncomment these two commands\n","# ! mkdir %userprofile%\\.kaggle\n","# ! move kaggle.json %userprofile%\\.kaggle"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"2tbgKWB3EfQa","colab_type":"code","colab":{}},"source":["! chmod 600 ~/.kaggle/kaggle.json\n","# ! kaggle datasets list"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"KLWn_qn94EN2","colab_type":"code","colab":{}},"source":["path = Config.data_path()/'style-color'\n","path.mkdir(parents=True, exist_ok=True)\n","path"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"j3zb62nO4EN_","colab_type":"code","colab":{}},"source":["! kaggle datasets download -d olgabelitskaya/style-color-images --force -p {path} --unzip "],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"5VxaxIaSIwPy","colab_type":"code","colab":{}},"source":["path.ls()\n","# Training data \n","path_style = path/'style'\n","# path_style.ls()"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"9-5srVFS4EOQ","colab_type":"text"},"source":["## Image multi label classification"]},{"cell_type":"code","metadata":{"id":"D77XZAvJyZo0","colab_type":"code","colab":{}},"source":["temp = pd.read_csv(path_style/'style.csv')\n","temp['file'] = 'style/' + temp['file'].astype(str)\n","# temp.head() \n","df = temp[['file', 'brand_label', 'product_label']]\n","df.head()"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"AnY3mfdR4EO

This file has been truncated. show original

Vlad_Tagunkov · May 27, 2020, 12:17pm

I try to dig deep about mixed training precision - how new it is.
Found at Nvidia docs:

|Model|Speedup| - with Pytorch
|—|---|
|NVIDIA Sentiment Analysis| 4.5X speedup|
|FAIRSeq| 3.5X speedup|
|GNMT| 2X speedup|
|ResNet-50| 2X speedup|

Q: Is Automatic Mixed Precision (AMP) dependent on a PyTorch version or can any PyTorch version enable AMP?

A: AMP with CUDA and CPP extensions requires PyTorch 1.0 or later. The Python-only build might be able to work with PyTorch 0.4, however, 1.0+ is strongly recommended.

https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html#faq-pytorch

montraz2016 · May 27, 2020, 3:46pm

My question is concerning the image segmentation lecture in lesson 3. I have created my own datasets. How are you able to determine the number of neurons that are in your input layer of the Neural Network? Saying that the train data:

ImageDataBunch;

Train: LabelList (160 items)
x: SegmentationItemList
Image (3, 128, 128),Image (3, 128, 128),Image (3, 128, 128),Image (3, 128, 128),Image (3, 128, 128)
y: SegmentationLabelList
ImageSegment (1, 128, 128),ImageSegment (1, 128, 128),ImageSegment (1, 128, 128),ImageSegment (1, 128, 128),ImageSegment (1, 128, 128)