Lesson 2: further discussion ✅


For your first question, have you tried plotting the entire (not truncated) graph?

(Benedikt Brandt) #44

Well he could certainly do that and maybe he would see an upward slope. But that would mean that the optimal learning rate is smaller than 10E-6. This seems suspiciously small given the learning rates we usually use.

(Jeremy Howard (Admin)) #45

It suggests you don’t need to unfreeze and fine-tune the lower layers. Is that what you found when you tried?

(Lankinen) #46

I wonder why we download and use only .jpg images? Could we use .png or other types of images mixed together or even only .png type images? What is the difference and couldn’t we just somehow convert type from .png to .jpg?

(Aakash N S) #47

That’s an interesting idea. But we might end up with too many categories (28 * 4).

(Arunoda Susiripala) #48

I’ve no experience in running with that many categories. But I have seen some good results even with such an amount in some other threads.

Give it a try :smiley:

(Mahesh Khatri) #49


Lot of work already done in this area.

One example. [Project] Fully convolutional watermark removal

There are more resources that make it easy to remove watermarks.

(Mahesh Khatri) #50

You could. I believe Resnet works with both image types. This article may help


(Andrea de Luca) #51

Nothing changes, actually. It just happens that the loss surface is a N-dimensional hypersurface embedded in a N+1-dimensional euclidean space.
It still got maxima and minima (but see below), and it’s generally quite bumpy. For, we make good use of cyclical learning rates and restarts: they’ll get you out of bad local minima.
Another important consideration is that as the dimensionality increases, almost every stationary point is a saddle point (= ability to escape).
This is not difficult to believe: intuitively, imagine a stationary point on a thousand-dimensional surface. For it to be a saddle point, it just suffices that it’s not a maximum/minimum with respect to one coordinate axis.

Yes. The research community is starting to think about the loss surface not as embedded in a euclidean space (and thus inheriting its metric), but as a standalone non-euclidean manifold with its own intrinsic metric.
It would be great if @rachel could write an article about it.

(Andrea de Luca) #52

I think that as long as the image can be turned in a torch tensor, you’ll be fine. Maybe the focus is on .jpgs since the huge vision datasets like Imagenet are a bit old, and back then .jpg was the most common compressed format.

(Sam) #53

jpg are compressed images (a lossy compression at that) which makes their size on servers smaller. they have 3 channels namely RGB
pngs have 4 channels namely RGBA. where the last layer is alpha which defines transparency.
if you open a 600 X 400 png image using PIL like:


and then see it as numpy array, its shape will be seen as:
np.array(image).shape>>>>(960, 1440, 4)

Depending upon how the image was constructed you could get a surprise as described here:

(Daniele Galiffa) #54

Thanks for you reply.
I’m not yet so into the math behind deep-learning, but your words are a good starting point for me to start operating deeper study :slight_smile:

(Jonathan Aghachi) #55

Here is a simple ‘for loop’ for helping make image folders more automatically.

All you need to do is create a “data/urls” folder where your’re running your notebook and upload all the link files into the urls folder.

Folder names will be named whatever you name the url txt files.

Note: This isn't to take away from highlighting the fact that you can run things out of order in notebooks, I just got a bit lazy in changing lots of code blocks, so this 'for loop' was born.

EDIT: variable path to images is now "pathI" instead of "path" just fyi.

folderNames = []

pathU = 'data/urls/'
files = os.listdir(pathU)
for name in files:

pathI = Path('data/images')

for title in folderNames:
    dest = pathI/title
    filename = title + '.txt'
    dest.mkdir(parents=True, exist_ok=True)
    download_images('data/urls/' + filename, dest, max_pics=200,max_workers=0)

EDIT: oops! the code above didnt show the line for “folderNames = []”
fixed it.

(Allan Jackson) #56

Yes I would say stop at epoch 2. But look at the accuracy…you can do the same flipping a coin. GL

(Vitaliy Bondarenko) #57

Guys, is there an SGD with restarts available in fastai v.1?

(Swagato Chatterjee) #58

Yes it is,I guess, in learn.fit(); however I have no clue how to use it.

(Andrea de Luca) #59


It should be cyc_len

(Joseph Russavage) #60

Hello! I am going through the Lesson 2 training and thinking about the time required to complete each epoch. I would love to see the amount of time required to complete each epoch included in the table showing learning rate, accuracy and loss by epoch. What could I add to this function to see this additional data?

(Joseph Russavage) #61

How could we incorporate a “none of the above” option for classification? We regularly see mis-classified images where the image does not really belong to any of the classification options. But since “none of the above” is not really an option the image gets mis-classified. Any ideas on this? It seems like it should be a “one minus” type function…

(Zach) #62

Hello all,

If you’ve been using the FileDeleter widget to clean your data, some changes came out today that may interest you. source

FileDeleter was removed in favor of two new widgets: ImageDeleter and ImageRelabeler.

ImageDeleter is essentially the same as the old FileDeleter.

ImageRelabeler renders the files you pass in and lets you re-label them. This is to find mis-categorized images in your data directory. NOTE: ImageRelabeler currently only works with files where labels were created from the names of their parent directory (i.e. with .from_folder()). The widget moves mislabeled photos from the incorrect parent directory to the properly-labeled parent directory.

To relabel an image, just click the proper label in the widget dropdown.

Both widgets take a formatted dataset from DatasetFormatter. You can easily specify which dataset you’d like to render by passing in DatasetType.Valid, DatasetType.Train or DatasetType.Test to the ds_type keyword arg in DatasetFormatter.from_toplosses.

All three classes are available in fastai.widgets.

Example use:

ds, idxs = DatasetFormatter().from_toplosses(learn, ds_type=DatasetType.Train)
ImageRelabeler(ds, idxs)
# or
ImageDeleter(ds, idxs)

Real docs for these are coming soon!

FileDeleter Troubles
Lesson 3 In-Class Discussion ✅
Lesson 2 Errors: FileDeleter and 'defaults" attribute
Lesson 2 In-Class Discussion ✅
Lesson 3 In-Class Discussion ✅
Jupyter plugin to fix mislabeled data