Developer chat

[Tom2718]

Would having a Google Cloud functions version of deploying fastai models be worth adding to the list of available production options on the course homepage? I created a repo outlining the general process and could easily change it into a single post tutorial. Currently I’ve been using it for text classification, so that’s what the tutorial would be focused on.

[prosti]

They are used any time you grab any data with fastai… so pretty much all the time.

Thanks @sgugger.

I would like to check with the developers one specific item. I created the example.

import torch
from torch.utils.data import DataLoader, Dataset
import numpy as np

x1 = np.array([1,2,3])
x2 = np.array([2,3,4])
x3 = np.array([4,5,6])

d1 = DataLoader( x1, batch_size=3)
d2 = DataLoader( x2, batch_size=3)
d3 = DataLoader( x3, batch_size=3)

class DataBunch():
    "Bind `train_dl`,`valid_dl` and `test_dl` in a data object."
    def __init__(self, train_dl:DataLoader, valid_dl:DataLoader, test_dl:DataLoader):
        self.train_dl= train_dl
        self.valid_dl= valid_dl
        self.test_dl= test_dl
        
db =DataBunch(d1,d2,d3)

class ItemList:
    _bunch=DataBunch
    def __init__(self, var):
        self.var=var
        
    def __repr__(self):
        return f"class {self.__class__.__name__} holding private _bunch {self._bunch} and var {self.var}"
        
il1 = ItemList(1)
il2 = ItemList(2)
il3 = ItemList(3)


class ItemLists:
    def __init__(self, il1:ItemList, il2:ItemList, il3:ItemList):
        self.il1=il1
        self.il2=il2
        self.il3=il3

ils = ItemLists(il1, il2, il3) # holding 3 ItemList objects
print(vars(ils))

print(hex(id(ils.il1._bunch)))
print(hex(id(ils.il2._bunch)))
print(hex(id(ils.il3._bunch)))


ils.il1._bunch=db

print(hex(id(ils.il1._bunch)))
print(hex(id(ils.il2._bunch)))
print(hex(id(ils.il3._bunch)))

print(ils.il1._bunch)
print(ils.il2._bunch)
print(ils.il3._bunch)

That will output like this:

{'il1': class ItemList holding private _bunch <class '__main__.DataBunch'> and var 1, 'il2': class ItemList holding private _bunch <class '__main__.DataBunch'> and var 2, 'il3': class ItemList holding private _bunch <class '__main__.DataBunch'> and var 3}
0x5ce2598
0x5ce2598
0x5ce2598
0x7f6c733d1550
0x5ce2598
0x5ce2598
<__main__.DataBunch object at 0x7f6c733d1550>
<class '__main__.DataBunch'>
<class '__main__.DataBunch'>

What I wanted to cover here is some architectural dilemma I am having at the moment.
ItemList is a holder for the _bunch=DataBunch, where a DataBunch is a DataSet's or DataLoader's holder and it has self.train_dl,self.valid_dl inside.
_{DataBunch typically holds train and validation datasets. We use these datasets to create DataBunch via create class method, or we initialize DataBunch via __init__ with DataLoaders again holding train and validation datasets.}

My dilemma is why ItemLists doesn’t hold the _bunch as it has more sense.
ItemLists class objects to recall will have train:ItemList, valid:ItemList inside.

This way if I create a DataBunch with three data loaders d1, d2, d3

db=DataBunch(d1,d2,d3) 

and if I set to ItemList 1 _bunch it will be only part of that ItemList. It will not be shared with other ItemLists.

ils.il1._bunch=db

This _bunch will be inside ItemList class dictionary __dict__.

[TomB]

Not a core developer, so could be off, but have done some work on the custom data block stuff for audio so some familiarity with this code.
I think the reason is you don’t generally use a custom ItemLists. The default ItemLists just serves to store multiple custom ItemList instances for the train/validation/test sets. ItemLists just calls the appropriate custom ItemList method on each set instance. This is handled in ItemLists.__getattr__, (which I imagine is how _bunch is accessed when creating a DataBunch).
So yes, it may be ItemLists where it is used but there’s no reason to require a custom ItemLists just to specify this. And it is based on the particular item type handled by an ItemList that you would select an appropriate custom DataBunch if needed.
Though, technically, I think a DataBunch isn’t actually created from an ItemLists but rather from a LabelLists, so it would really be here not in ItemLists that you might move it. But, again you don’t generally use a custom LabelLists, it just dispatches to the ItemLists for the data and labels.

and if I set to ItemList 1 _bunch it will be only part of that ItemList . It will not be shared with other ItemLists .

I don’t think there’s ever a need to do that. You define an appropriate DataBunch when defining a custom ItemList class. For things that do need to be dynamically generated for an instance, you use copynew which defines what needs to be passed to constructors as various new instances are created.

[rpcoelho (Rodrigo Pinto Coelho)]

Hi Jeromy,

I’ve been watching the Deep Learning course 2019 and was inspired by the example you gave of the gentleman that contributed with code. Specially because I’ve been working on something that one of the students asked which was to automatically find the learning rate. I developed an AutoML hyperparameter algo that has been beating random search, and skopt’s Baysian Optimization. I only tested it for Random forests and ExtratreeRegressor. My Dataset is a simplified version of the Rossman Kaggle competition. I feel it would be interesting to try to use it to optimize a Neural network. Would you have a small toy problem (I have a first generation i7 with 24gb of memory and no graphics card) and dataset that we could use to test it? I’m using Pattern Search which is MATLAB’s method for finding global optima (my code is in Python 3.6 and I have adapted it to work in the SKLearn framework, so you can call the code just like you call RandomizedSearchCV). Do you have anyone that I could show the notebook to see if it is something interesting to incorporate into Fast.ai?.

Best regards,

Rodrigo.

[kcturgutlu (Kerem Turgutlu)]

@sgugger Do we need bptt to be a multiple of 8 for fp16? As it will be the input tensor shape I thought it might effect too and I see that it’s still left 70 as the default in 1.0.54.dev0. Just curious…thanks

[sgugger]

Not entirely sure, but yes (and that’s one thing you can easily change )

[kcturgutlu (Kerem Turgutlu)]

Sure, thanks for the reply

[raghavan]

I have been looking into contributing for fast ai , Apart of adding test and documentation , is there any roadmap to things one can contribute to ? Where can I find one ?

[Tom2718]

Probably the best place is here

[jeremy (Jeremy Howard)]

v1.0.54 is released, with just one little addition, but one I’m very happy about: fastai.torch_core.Module. No need for super().__init__ in modules now!

[aschiller (Adam Schiller)]

Hey folks, I’d like the ability to pass additional callbacks (e.g. Teacher Forcing) in the learner.lr_find() function. Currently there isn’t a good way of going about this unless I’m missing something obvious.

It should be as simple as adding an additional callbacks argument which gets added onto the callbacks list and eventually passed to the fit function.

def lr_find(learn:Learner, start_lr:Floats=1e-7, end_lr:Floats=10, num_it:int=100,
            stop_div:bool=True, wd:float=None, callbacks:Optional[CallbackList]=None):
    ...
    cb = [] if callbacks is None else callbacks
    cb.insert(0,LRFinder(learn, start_lr, end_lr, num_it, stop_div))
    ...
    learn.fit(epochs, start_lr, callbacks=cb, wd=wd)

Is this something which I should PR request or is this a bad idea?
Thanks for a great library and course!!

[sgugger]

You can add a PR for this. The use never came up before since usually callbacks are put in the Learner at init.

[aschiller (Adam Schiller)]

Ah, yes of course. It makes much more sense to pass callbacks in to the learner initialization once rather than litter several function calls with the same callbacks! Thanks for the wisdom!

I’ll hold off on the PR unless some other valid use case comes up…

[ilovescience (Tanishq)]

@sgugger would it be helpful to have gradcam as a separate function? because right now it is only under plot_top_losses so we cannot get gradcam output for correctly classified images which could be helpful for model interpretation. It might be useful to be an optional output of get_preds or predict.

Thoughts?

[sgugger]

I have no objection to it being in a separate function if you want to work on a PR that does that.

[Andreas_Daiminger]

Sorry! I am not sure if this is the right place to post this. But there is a bug in the fastai pip and anaconda package:

Bug in fastai 1.0.54 package

I installed fastai with anaconda and found the following bug in the package code. In fastai/text/learner.py def load_encoder(self, name:str, device:torch.device=None): "Load the encoder `name` from the model directory." encoder = get_model(self.model)[0] if device is None: device = self.data.device if hasattr(encoder, 'module'): encoder = encoder.module encoder.load_state_dict(torch.load(self.path/self.model_dir/f'{name}.pth')) encoder.load_state…

[sgugger]

Yes, it has been fixed in master and will be in the next release.

[lewfish (Lewis Fishgold)]

Are there any plans to add an object detection model/learner (similar to how there are is a UnetLearner) to the library in the near term? Just wondering if I should hold out a little longer, or figure out how to use the object detection models in the new version of torchvision in conjunction with the fastai object detection data loader and transforms.

[sgugger]

This is more midterm than short-term, as we are fully focused on v2 for now.

[alien35 (Alex Leon)]

Hi everybody. Question regarding fast.ai and render.com

1. I trained a model using crestle
2. I exported the model to .pkl
3. I put the .pkl file on dropbox
4. I have a server load the pkl as per the instructions
5. When I try to build the dockerfile, I get this error

/usr/local/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class ‘torch.nn.modules.loss.CrossEntropyLoss’ has changed. you can retrieve the original source code by accessing the object’s source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes.
warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class ‘torch.nn.modules.conv.Conv2d’ has changed. you can retrieve the original source code by accessing the object’s source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes.
warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class ‘torch.nn.modules.batchnorm.BatchNorm2d’ has changed. you can retrieve the original source code by accessing the object’s source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes.
warnings.warn(msg, SourceChangeWarning)
/usr/local/lib/python3.7/site-packages/torch/serialization.py:454: SourceChangeWarning: source code of class ‘torch.nn.modules.activation.ReLU’ has changed. you can retrieve the original source code by accessing the object’s source attribute or set torch.nn.Module.dump_patches = True and use the patch tool to revert the changes.
warnings.warn(msg, SourceChangeWarning)
…

I have tried clicking export twice in crestle, as someone suggested ‘re-exporting.’ Did I misinterpret what ‘re-exporting’ means? Any ideas what is happening? My requirements.txt file looks like this:

aiofiles==0.4.0
aiohttp==3.5.4
asyncio==3.4.3
fastai==1.0.52
https://download.pytorch.org/whl/cpu/torch-1.1.0-cp37-cp37m-linux_x86_64.whl
https://download.pytorch.org/whl/cpu/torchvision-0.3.0-cp37-cp37m-linux_x86_64.whl
numpy==1.16.3
starlette==0.12.0
uvicorn==0.7.1
python-multipart==0.0.5