Developer chat

jeremy · November 2, 2018, 6:35pm

We agree. Stand by…

sgugger · November 2, 2018, 6:59pm

Done. image_opener is now an attribute of every image dataset, and you can pass any function that will take a file name and return an Image object.
I did the same for segmentation masks and remove the kwargs that were stored in SegmentationDataset, people can modify the default value by changing the mask_opener attribute.

To change the attributes on all the datasets at once, I strongly suggest using the data block API. Here is a simple use case:

data = (InputList.from_folder(path)
        .label_from_folder()
        .split_by_folder()
        .datasets(ImageClassificationDataset)
        .set_attr(image_opener=my_custom_opener)
        .transform(tfms, size=224)
        .databunch(bs=64))

Kaspar · November 2, 2018, 8:23pm

hurray that makes my life easier - no more patching after git pull:)

jeremy · November 2, 2018, 9:00pm

In order to make it more convenient to submit a PR with docs and lib changes together (amongst other things) I’ve now merged the fastai_docs and fastai repos. The downside is that the fastai repo is pretty big now - sorry about that, but I think it’ll be worth it.

jeremy · November 3, 2018, 11:38pm

Just released 1.0.19.

Biggest changes are:

add an argument resize_method that tells apply_tfms how to resize the image to the desired size (crop, pad, squish or no).
jupyter et al no longer forced dependencies

You should find that most of the things that led to errors in fastai.vision.data before, such as not including size= or not including a crop_pad transform will now work - and will instead squish images to be the requested size. You can also use a rectangular size= if you’re squishing.

github.com

fastai/fastai/blob/master/CHANGES.md

# Changes

Most recent releases are shown at the top. Each release shows:

- **New**: New classes, methods, functions, etc
- **Changed**: Additional paramaters, changes to inputs or outputs, etc
- **Fixed**: Bug fixes that don't change documented behaviour

Note that the top-most release is changes in the unreleased master branch on
Github. Parentheses after an item show the name or github id of the contributor
of that change.





## 1.0.20.dev0 (Work In Progress)

### New:

This file has been truncated. show original

sgugger · November 6, 2018, 4:25am

Breaking changes in text! To make the API more consistent with the vision side and make it compatible with the data block API, plus cleaning the code that was a bit too messy, I’ve just merged big changes in NLP. Examples and docs are updated to reflate that. There’s also a working lesson notebook in course-v3 (I’d love to know if it doesn’t work for you).

Please note:

RNNLearner.classifier is now text_classifier_learner
RNN_Learner.language_model is now language_model_learner.
You need to manually save/load your TextDataBunch. Doing it automatically was inducing too many subtle bugs.
TextDataBunch.from_csv now takes on csv and valid_pct instead of two csv

If you’re using the dataset methods, changes are more numerous so you’d better check the docs. And remember to use the data bock API as it’s been designed to make your life easier

shaun1 · November 6, 2018, 12:48pm

Docs for the classifier show errors in the page.

sgugger · November 6, 2018, 2:26pm

Thanks for flagging, will correct!

Pomo · November 7, 2018, 5:45am

Hello developers. I ran into a problem with download_images() in fastai/vision/data.py while downloading training images from Google for making a useless but fun classifier. All images are given extension .jpg, even when they are actually .png. This mismatch causes Image Viewer to reject those images, which in turn makes it an extra hassle to clean up the training data.

I am not yet fluent enough in Python to offer a fix, but here is a gist with notebook and url source file that demonstrates the problem. Just put them both in the same folder to test.

https://gist.github.com/PomoML/1836e9f2b9138ecc9fba1586d2118919

Thanks for looking at this issue!

howkhang · November 7, 2018, 12:41pm

Getting an index out of range error when instantiating data_lm. The value used to index the list of folders should not be len?

data_lm = (TextFileList.from_folder(path)         
           #grap all the text files in path
           .label_from_func(lambda x:0)           
           #label them all wiht 0s (the targets aren't positive vs negative review here)
           .split_by_folder(valid='test')         
           #split by folder between train and validation set
           .datasets(TextDataset, is_fnames=True) 
           #use `TextDataset`, the flag `is_fnames=True` indicates to read the content of the files passed
           .tokenize()
           #tokenize with defaults from fastai
           .numericalize()
           #numericalize with defaults from fastai
           .databunch(TextLMDataBunch))
           #use a TextLMDataBunch
data_lm.save('tmp_lm')

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-30-6e16f2eee3d4> in <module>
      3            .label_from_func(lambda x:0)
      4            #label them all wiht 0s (the targets aren't positive vs negative review here)
----> 5            .split_by_folder(valid='test')
      6            #split by folder between train and validation set
      7            .datasets(TextDataset, is_fnames=True)

/fastai/fastai/data_block.py in split_by_folder(self, train, valid)
    135         """
    136         n = len(self.path.parts)
--> 137         folder_name = [o[0].parent.parts[n] for o in self.items]
    138         valid = [o for o in self.items if o[0].parent.parts[n] == valid]
    139         train = [o for o in self.items if o[0].parent.parts[n] == train]

/fastai/fastai/data_block.py in <listcomp>(.0)
    135         """
    136         n = len(self.path.parts)
--> 137         folder_name = [o[0].parent.parts[n] for o in self.items]
    138         valid = [o for o in self.items if o[0].parent.parts[n] == valid]
    139         train = [o for o in self.items if o[0].parent.parts[n] == train]

IndexError: tuple index out of range

sgugger · November 7, 2018, 1:06pm

Update: I fixed the dataset on our S3 bucket so it should run smoothly now. Just be sure to remove imdb/ and imdb.tgz from your .fastai/data/ folder to trigger the download of the new dataset.

jeremy · November 7, 2018, 3:29pm

Just released 1.0.20. Main changes likely to impact folks:

DataBunch.dl replaces the various holdout, is_test, and is_train approaches with a single consistent enum. (h/t @zachcaceres)
download_url reads the get request with iter_content which is robust to ‘content-length’ errors. (h/t @lesscomfortable)
create_cnn should work with models other than resnet now
Data blocks API support for fastai.text
QRNN seems to actually be working properly again.

github.com

fastai/fastai/blob/master/CHANGES.md

# Changes

Most recent releases are shown at the top. Each release shows:

- **New**: New classes, methods, functions, etc
- **Changed**: Additional paramaters, changes to inputs or outputs, etc
- **Fixed**: Bug fixes that don't change documented behaviour

Note that the top-most release is changes in the unreleased master branch on
Github. Parentheses after an item show the name or github id of the contributor
of that change.



## 1.0.21.dev0 (Work In Progress)

### New:

### Changed:

This file has been truncated. show original

rameshsingh · November 8, 2018, 10:41am

Hi Fastai Devs, While I was working on one of the classification problems I found that the plot_top_losses is sometimes confusing and I would personally like to see the current prediction probability as well. In many of the local meetups I found it is being discussed also. I have done the changes locally and its working fine also but I would request the devs here to suggest if we can add it to the master repo.
This being my 1st contribution I would request devs here to excuse any protocol breach and guide me on correct path.
Regards,

marcmuc · November 9, 2018, 7:35am

First real namespace conflict in fastai notebooks?!

I just had a case where the from fastai import * strategy really caused problems.
When running the lesson3-planets notebook (but I guess this would happen in many other notebooks too):
Suddenly data.classes did not exist anymore and gave me errors, and many other things data.xxx that I tried. Turned out after some trial and error, that by rerunning the top cell (with the imports) after adding an additional import is the cause for this:

data = xxxx is the name of the databunch we create using the new (awesome!!) block api.

after running imports again:
data is now fastai.vision.data module. and of course there is no data.classes anymore…

So this is a clear naming/namespace conflict.
Maybe (@sgugger) the variables in the notebooks could be renamed to databunch = xxx so this will not happen to others?

If it helps I can do that and do a PR…

marcmuc · November 9, 2018, 7:41am

And explicitly a big thank you to everyone involved in creating the new data blocks API!
I think it is awesome and will make things much easier, flexible, adaptable and explicit!

stas · February 15, 2019, 4:18pm

10 posts were split to a new topic: AutoLRFinder

jeremy · November 10, 2018, 1:27am

Released 1.0.22 which fixes learn.predict, and also avoids importing submodules directly in to the namespace.

jeremy · November 10, 2018, 1:30am

That was a bug - the submodules weren’t meant to be imported directly. Fixed now. data won’t be clobbered. My fix is really ugly, so if any python experts know how to make our __init__.py files less awful, please let me know

Patrick · November 11, 2018, 1:48am

Hi. Are you accepting PRs from non-core developers? I’ve been looking at the library for the past couple days to find a way to integrate “observation weights” into the codebase. I think the change in code would be very minimal and would be completely confined to fit() function and its dependencies, validate() and loss_batch(). The gist of the PR would be about allowing yb to a be a list where the last item in the list is a tensor representing the observation weights.

sgugger · November 11, 2018, 4:14am

Forgot to reply to your other post. There’s no tweak needed, a target can already be a list of tensors. You have to properly address it with your loss function, that’s all.

For a new features, it’s best to prepare a notebook showing how it works so we can help refactor your code, but otherwise we’re happy to accept PRs from anyone!