A walk with fastai2 - Vision - Study Group and Online Lectures Megathread

kofi · January 23, 2020, 9:02am

Please I want to watch the lesson 2 video as I was not able to join the live stream. But it seems that the video is no longer available.

kofi · January 23, 2020, 9:03am

@muellerzr

muellerzr · January 23, 2020, 11:12am

@kofi thanks! I put the right one up there

muellerzr · January 23, 2020, 11:13am

What is the error? I ran it that day so that should all work

kofi · January 23, 2020, 2:02pm

Thanks…
I also have some question from the first lesson:

In the first video, when we were using ImageDataBunch.from_name_re(), we didn’t specify how the data was going to be split but we did that when we were using the DataBlock api.
Also when i try to use shuffle =True i get this errror
Screenshot from 2020-01-23 13-56-59.png1214×417 51.7 KB

muellerzr · January 23, 2020, 2:06pm

Always check the source code (part of why I’m doing this study group is to get you guys more comfortable with it, hence why the end of the last lesson we went over it)

Take a look here and your questions will be answered:

github.com

fastai/fastai2/blob/master/fastai2/vision/data.py#L21


# Cell
def _using_attr(f, attr, x):
    return f(getattr(x,attr))


# Cell
def using_attr(f, attr):
    "Change function `f` to operate on `attr`"
    return partial(_using_attr, f, attr)


# Cell
class ImageDataBunch(DataBunch):
    @classmethod
    @delegates(DataBunch.from_dblock)
    def from_folder(cls, path, train='train', valid='valid', valid_pct=None, seed=None, vocab=None, **kwargs):
        "Create from imagenet style dataset in `path` with `train`,`valid`,`test` subfolders (or provide `valid_pct`)."
        splitter = GrandparentSplitter(train_name=train, valid_name=valid) if valid_pct is None else RandomSplitter(valid_pct, seed=seed)
        dblock = DataBlock(blocks=(ImageBlock, CategoryBlock(vocab=vocab)),
                           get_items=get_image_files,
                           splitter=splitter,
                           get_y=parent_label)
        return cls.from_dblock(dblock, path, path=path, **kwargs)

A few hints if you’re struggling a bit following that:

cls.myFunc() means we’re running another function that is of that type. In this case, it goes from from_name_re -> from_name_func -> from_path_func

Let me know if you still have questions afterwards @kofi and report back what you find!

kofi · January 23, 2020, 2:19pm

Oh I get it now, it was using it by default. thanks @muellerzr .
What of the second question?

muellerzr · January 23, 2020, 2:20pm

The second comes from the fact it’s not an available parameter, and has a default set in place for it instead that is always used. If you want to get that specific you’d be better using the mid-level or low-level DataBlock api

barnacl · January 23, 2020, 2:48pm

So weird ran the exact same notebook now and no error.
I was getting the following error - NameError: name 'Cuda' is not defined
All works now. Thank you

muellerzr · January 23, 2020, 2:49pm

One possibility is I believe that in a future update they are getting rid of it, and you may have installed the dev version or something (once it’s live I’ll update the notebooks with this change)

kofi · January 23, 2020, 2:50pm

Well so how do you do that. I am looking at the prototype for the DataBlock api and I can’t see how you would specify that.

kofi · January 23, 2020, 2:52pm

ok… thanks

muellerzr · January 23, 2020, 2:52pm

If you check further it can be seen in a call to .databunch (not in that particular notebook/file)

github.com

fastai/fastai2/blob/master/nbs/06_data.block.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#default_exp data.block"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "from fastai2.torch_basics import *\n",
    "from fastai2.data.core import *\n",

This file has been truncated. show original

So after you make your block, when you do .databunch you can pass in shuffle=False. This is really more of a lower level API deal, where we create two data loaders and then pass them to DataBunch

vijayabhaskar · January 23, 2020, 5:37pm

Has anyone tried fastapi instead of Starlette? It is built on top of Starlette, and therefore you can do everything that Starlette supports quickly. It even generates docs automatically.

lukew · January 24, 2020, 9:03am

FastAPI is a great library IMO, deploying an API has gone from difficult to too-easy.
(this is a good resource if anyone is interested https://www.youtube.com/watch?v=mkDxuRvKUL8)

mrfabulous1 · January 24, 2020, 9:58am

Hi vijayabhaskar

Yes its great, it also uses Pydantic, so you can specify your own data types, includes testing tools as well. https://fastapi.tiangolo.com/

cheers mrfabulous1

mgloria · January 24, 2020, 11:01am

Thanks! I took a look at it and this helped me a lot. I will post my solution since I believe it is a common problems that would help others.

imgs = L()
path_train = path/'train'
imgs = verify_images(path_train.ls())
hashed_ids = [os.path.splitext(os.path.basename(im))[0] for im in imgs]

This returns from a full path in an L() the desired hashed.

[‘a2b380442e’, ‘fff4c5a0f8’, ‘9e8dc3116f’, ‘0ddbf46844’, ‘edd73511dc’, ‘4045344683’, ‘fc03d4a895’, ‘4b00fd9217’, ‘41c740d0cc’, ‘c60d3022aa’]

These hashes can be then used to filter in the csv as follows:

df = pd.read_csv(path/'train_labels.csv')
df_filtered = df[df['hashed_id'].map(lambda x: x not in hashed_ids)]

This df_filtered can then be safely used to create the datablock.

mgloria · January 24, 2020, 11:08am

I am running into some weird behavior. I am resizing my images to be 224*224 but somehow this is not having any effect
Anyone knows what is up here?

muellerzr · January 24, 2020, 12:04pm

@mgloria remember item transforms are happening first, then batch transforms are performed each time a batch is called (they’re done on the fly). Try calling dbunch.one_batch() and you should get an error

(You should resize all your images in the item_tfm first if they are different sizes)

leaf · January 24, 2020, 2:01pm

Has anyone actually managed to go through the 01_Custom notebook to train a model? I am struggling a bit with something else and would love to use the approach @muellerzr took. But the fact that the livestream only shows what he would do rather than it acually working is not helpful.

Btw, great work @muellerzr but I would advise that showing your work would be greatly appreciated by many