Fastai v2 chat

Thanks, silly me, I don’t know how I missed that. I usually search within Github, which works reasonably well. Also PyCharm ctr + click takes you to the source, even for external code. For some reason I forgot about that. Interestingly I found the Cuda reference as well, in fastai2/data/transforms.py, but it’s not on the online version. There has probably been a recent change there.

Defined in fastcore/01_foundation.ipynb as shown,

class _T(metaclass=PrePostInitMeta):
    def __pre_init__(self):  self.a  = 0; assert self.a==0
    def __init__(self,b=0):  self.a += 1; assert self.a==1
    def __post_init__(self): self.a += 1; assert self.a==2

_T() raises error in pre_init when initialized with an argument intended for b:

_T(1)
TypeError: __pre_init__() takes 1 positional argument but 2 were given

Thus, a better definition:

class _T(metaclass=PrePostInitMeta):
    def __pre_init__(self,b=0):  self.a  = 0; assert self.a==0
    def __init__(self,b=0):  self.a += 1; assert self.a==1
    def __post_init__(self,b=0): self.a += 1; assert self.a==2

Sylvain @sgugger, nota bene.

I don’t get any error with _T(1).

@muellerzr,

Deep down in the dungeons I find ‘TitledFloat:grinning:
so that I can use dbunch.show_batch() for regression… some thing like below works . It might be useful to others …

class TitledFloatShort(Float, ShowTitle):

_show_args = {‘label’: ‘text’}
def show(self, ctx=None, **kwargs):
“Show self”

return show_title(f’{self:.2f}’, ctx=ctx, **merge(self._show_args, kwargs))

class ToFloatTensor(Transform):

“Transform to float tensor”
order = 10 #Need to run after PIL transforms on the GPU
_show_args = {‘label’: ‘text’}
def init(self, split_idx=None, as_item=True):
super().init(split_idx=split_idx,as_item=as_item)
def encodes(self, o): return o.astype(np.float32)
def decodes(self, o): return TitledFloatShort(o)

def FloatBlock(vocab=None, add_na=False):
"TransformBlock for single-label float targets"
return TransformBlock(type_tfms=ToFloatTensor())

den_db = DataBlock(blocks=(ImageBlock, FloatBlock),
get_x=lambda x:x[0],
get_y=lambda x:x[1],
splitter=RandomSplitter())

item_tfms = [Resize(224)]
dbunch = den_db.databunch(data2, path=path_img, bs=128, num_workers=0, item_tfms=item_tfms)
dbunch.c = 1

and now works with float titles :slight_smile:

dbunch.show_batch()

2 Likes

Good job finding that! I was about to post about it :slight_smile:

2 Likes

Sorry if this is too obvious. I am still working on moving a text model from v1 to v2. I am currently getting an AttributeError for mixed precision:

self.learner = language_model_learner(data, arch=AWD_LSTM, opt_func=opt_func, path=path, drop_mult=0.3).to_fp16()
AttributeError: 'LMLearner' object has no attribute 'to_fp16'

However, the train_imagenette example contains the line
if fp16: learn = learn.to_fp16()
so I am a bit confused :sweat_smile:

:thinking: the problem may be different than I assumed. After removing to_fp16 it protested about not having train_one_epoch, which makes no sense, so I wrote back the import * line. Now I am having different trouble. I will report when I understand things a bit better.

Can someone help me to understand what the purpose of assigning/checking delwrap is here inside delegates? To me it seems to be preventing multiple nested calls to delegates from working successfully. The function being wrapped has it’s __delwrap__ attr set on the first delegates call, and on subsequent calls the function is returned with it’s signature unchanged. For reference, here’s an example of the type of nested delegates I’m playing around with, and the problem is that only the arguments from the nearest delegats call (_ToDB) appear in the altered AudioToSpec signature

@delegates(_GenSpec.__init__)
@delegates(_GenMelSpec.__init__, keep=True)
@delegates(_ToDB.__init__, keep=True)
class AudioToSpec(Transform):
def delegates(to=None, keep=False):
    "Decorator: replace `**kwargs` in signature with params from `to`"
    def _f(f):
        if to is None: to_f,from_f = f.__base__.__init__,f.__init__
        else:          to_f,from_f = to,f
        from_f = getattr(from_f,'__func__',from_f)
        if hasattr(from_f,'__delwrap__'): return f #*****THIS LINE*****
        sig = inspect.signature(from_f)
        sigd = dict(sig.parameters)
        k = sigd.pop('kwargs')
        s2 = {k:v for k,v in inspect.signature(to_f).parameters.items()
              if v.default != inspect.Parameter.empty and k not in sigd}
        sigd.update(s2)
        if keep: sigd['kwargs'] = k
        from_f.__signature__ = sig.replace(parameters=sigd.values())
        from_f.__delwrap__ = to_f #*****AND THIS LINE*****
        return f
    return _f

Hmmm… I really don’t recall. We should have put a comment in when we added that code - sorry about that!

Maybe try removing the if hasattr line, and see if anything breaks?

1 Like

I’m trying to implement ULMFIT for multi label classification for many texts (so texts load from folder).

The language model seems fine, as seems the databunch. (Incidentally; how do you save this? data.save(my_paths.multi_label_data) fails.)

For my learner I am using:

opt_func = partial(Adam, wd=0.1)
learner = text_classifier_learner(data,
                                  AWD_LSTM,
                                  metrics=[FBeta],
                                  path=path,
                                  drop_mult=0.5,
                                  opt_func=opt_func)
learner = learner.load_encoder(my_paths.fine_tuned_encoder)

But then at train I get (note 780 = batch x labels):

ValueError: Expected input batch_size (12) to match target batch_size (780).

If you want to double check, databunch tensors look like:

Text (yes, the first words are the same in all docs):
tensor([[ 2, 96, 9, …, 17, 48, 16],
[ 2, 96, 9, …, 1, 1, 1],
…,
[ 2, 96, 9, …, 1, 1, 1],
[ 2, 96, 9, …, 1, 1, 1]], device=‘cuda:0’)

Labels (correct length: batch 12 x 65 labels):
tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
…,
[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]], device=‘cuda:0’)

Thanks for the suggestion, I did this and ran nbdev_test_nbs on fastai2 and fastcore and nothing broke (procedure I followed included below). Would you guys prefer to delete the delwrap lines or would it be easier if I submit a PR? I’ll note your preference going forward as well so I don’t ask each time :grinning:

  1. Force-pull fastai2 and reinstall with pip install -e .
  2. Edit fastcore/foundations.py to remove the hasattr line
  3. pip uninstall fastcore, pip install -e . inside fastcore root

You can directly use pickle to save/load your DataBunch object. It seems like you have a multilabel problem so you probably need to change the loss function.

Ah, I see the problem. text_classifier_learner hard-codes the loss function instead of picking it in the data. Will fix.

1 Like

Yes, that fixed the problem! Now I have a new one, but things are moving forward :slight_smile:

Edit: it seems like basic training is finally working! Now I just have to iron out some details.

I have some trouble with my data loading for the classifier. It works, but it seems to repeat tokenization (already done for the language model). Currently I do:

data_label_blocks = (TextBlock.from_folder(path=unsupervised_folder, vocab=self.vocab), 
                     MultiCategoryBlock(vocab=my_classes))
dsrc = DataBlock(blocks=data_label_blocks,
                 splitter=RandomSplitter(),
                 get_x=lambda x: unsupervised_folder / f'{x[0]}.txt',
                 get_y=lambda x: x[1].split(' '),
                 dl_type=SortedDL)

But I already have the data tokenized in another folder! Very naively I tried to pass that folder but then Fastai created a tokenized version of the tokenized version (which is mostly right, but not quite, and still repeating work). I suppose there is some option to say “skip tokenization”? I tried some silly ideas like TextBlock(None, vocab=self.vocab) but no luck so far.

Update: We have been exploring the code and perhaps this is already fixed by design?

From the Tokenizer class in fastai2/text/core.py we have:

@delegates(tokenize_folder, keep=True)
def from_folder(cls, path, tok_func=SpacyTokenizer, **kwargs):
    path = Path(path)
    output_dir = Path(ifnone(kwargs.get('output_dir'), path.parent/f'{path.name}_tok'))
    if not output_dir.exists(): tokenize_folder(path, **kwargs)
    res = cls(get_tokenizer(tok_func, **kwargs), counter=(output_dir/fn_counter_pkl).load(),
              lengths=(output_dir/fn_lengths_pkl).load(), mode='folder')
    res.path,res.output_dir = path,output_dir
    return res

if not output_dir.exists(): tokenize_folder(path, kwargs)
So it seems like no double work is being done :slight_smile:

1 Like

What would be thoughts on including another dataset? This would be the one in question:

It needs a bit of cleaning for it to work (I’ve done this), the goal would be for a keypoint/pose detection based dataset :slight_smile:

(The heatmap tutorial will be using this dataset)

1 Like

I see there are some interesting changes in callbacks.

A good change is that we don’t need to pass the learner as a parameter (rather the callback is a parameter of the learner, which makes more sense) and that we don’t need to pass the callback at train.

The method (learn.show_training_loop()) to show all active callbacks at different parts of the loop is AMAZING. (Credit to David Cato.)

It’s a bit confusing that method names for events in callbacks have changed, and the base class is much harder to understand now. But there is a helpful (not trivial to find) list of events at fastai2/learner.py, called _loop

Event methods at the callbacks now don’t receive any parameters. The information is already available, although with new names as well. For instance, for the former parameter train, now we have to use self.learn.training. And other information is directly an attribute. For example, before we had the parameters last_target and last_output, and now we have self.pred (this is before sigmoid/softmax, I assume?) and self.yb (why yb?).

Do you have a quick explanation for why some info is stored at the learner and other is directly at the callback?

Update: I have been trying out combinations and it seems all attributes are accessible either through the learner or as attributes of the callback. I think this is the expected behaviour when inheriting from GetAttr.

The documentation is lacking, in my opinion, a small part explaining how to do your own callback (not much more info than in this mini-post would help a lot, I think).

2 Likes

No one said v2 was ready yet. The documentation is not done, and we will add tutorials, but we are still in development for now.

Of course! I didn’t mean it like that. The documentation on callbacks seems rather advanced otherwise, so I was just pointing this out.

Attention, we have made some renaming that breaks everything:

  • DataBunch is now DataLoaders
  • DataSource is now Datasets
  • TfmdList is now TfmdLists

To automatically change your code, run this in the folder it lays

find . -type f -exec perl -pi -e 's/\bDataSource\b/Datasets/g' {} +
find . -type f -exec perl -pi -e 's/\bdatasource\b/datasets/g' {} +
find . -type f -exec perl -pi -e 's/\bDataBunch\b/DataLoaders/g' {} +
find . -type f -exec perl -pi -e 's/\bdatabunch\b/dataloaders/g' {} +
find . -type f -exec perl -pi -e 's/\bTfmdList\b/TfmdLists/g' {} +
10 Likes

Is RandomResizeCropGPU currently the only example of GPU Transform?