Underlying `Transforms` machinery and `MetaClasses`

fastai fastai dev
4

Okay, so in my quest to find an answer to this I believe I’ve picked up something else about MetaClasses which is worth sharing.

I think the three most important functions in understand Transforms in V2 are:

class Transform(metaclass=_TfmMeta):
    "Delegates (`__call__`,`decode`) to (`encodes`,`decodes`) if `filt` matches"
    filt,init_enc,as_item_force,as_item,order = None,False,None,True,0
    def __init__(self, enc=None, dec=None, filt=None, as_item=False):
        self.filt,self.as_item = ifnone(filt, self.filt),as_item
        self.init_enc = enc or dec
        if not self.init_enc: return

        # Passing enc/dec, so need to remove (base) class level enc/dec
        del(self.__class__.encodes,self.__class__.decodes)
        self.encodes,self.decodes = (TypeDispatch(),TypeDispatch())
        if enc:
            self.encodes.add(enc)
            self.order = getattr(self.encodes,'order',self.order)
        if dec: self.decodes.add(dec)

    @property
    def use_as_item(self): return ifnone(self.as_item_force, self.as_item)
    def __call__(self, x, **kwargs): return self._call('encodes', x, **kwargs)
    def decode  (self, x, **kwargs): return self._call('decodes', x, **kwargs)
    def __repr__(self): return f'{self.__class__.__name__}: {self.use_as_item} {self.encodes} {self.decodes}'

    def _call(self, fn, x, filt=None, **kwargs):
        if filt!=self.filt and self.filt is not None: return x
        f = getattr(self, fn)
        if self.use_as_item or not is_listy(x): return self._do_call(f, x, **kwargs)
        res = tuple(self._do_call(f, x_, **kwargs) for x_ in x)
        return retain_type(res, x)

    def _do_call(self, f, x, **kwargs):
        return x if f is None else retain_type(f(x, **kwargs), x, f.returns_none(x))

Ofcourse, the class Transform itself.

Then, its metaclass _TfmMeta

#export
class _TfmMeta(type):
    def __new__(cls, name, bases, dict):
        print("I'm alive inside `__new__` in `_TfmMeta`")
        res = super().__new__(cls, name, bases, dict)
        res.__signature__ = inspect.signature(res.__init__)
        return res

    def __call__(cls, *args, **kwargs):
        f = args[0] if args else None
        n = getattr(f,'__name__',None)
        if not hasattr(cls,'encodes'): cls.encodes=TypeDispatch()
        if not hasattr(cls,'decodes'): cls.decodes=TypeDispatch()
        if isinstance(f,Callable) and n in ('decodes','encodes','_'):
            getattr(cls,'encodes' if n=='_' else n).add(f)
            return f
        return super().__call__(*args, **kwargs)

    @classmethod
    def __prepare__(cls, name, bases): return _TfmDict()

And finally _TfmDict

#export
class _TfmDict(dict):
    def __setitem__(self,k,v):
        if k=='_': k='encodes'
        if k not in ('encodes','decodes') or not isinstance(v,Callable): return super().__setitem__(k,v)
        if k not in self: super().__setitem__(k,TypeDispatch())
        res = self[k]
        res.add(v)

According, to section 3.3.3.4. of Python Data Model,

Once the appropriate metaclass has been identified, then the class namespace is prepared. If the metaclass has a __prepare__ attribute, it is called as namespace = metaclass.__prepare__(name, bases, **kwds) (where the additional keyword arguments, if any, come from the class definition).

Which means when we first define our class Transform like so:
class Transform(metaclass=_TfmMeta):

The __prepare__ method inside _TfmMeta get’s called which in turn changes the way a normal __prepare__ method would work by using _TfmDict which inherits from dict.

Since the __setitem__ is updated inside _TfmDict, if k ie., the attribute to be set, is not encodes or decodes the normal machinery of dict is used ie., super().__setitem__(k,v)

From my understanding, while preparing the Transform namespace the encodes or decodes does not actually get passed and the final dict that we get at this stage is

{'__module__': '__main__', '__qualname__': 'Transform', '__doc__': 'Delegates (`__call__`,`decode`) to (`encodes`,`decodes`) if `filt` matches', 'filt': None, 'init_enc': False, 'as_item_force': None, 'as_item': True, 'order': 0, '__init__': <function Transform.__init__ at 0x7f2f3ae80d08>, 'use_as_item': <property object at 0x7f2f3ae97818>, '__call__': <function Transform.__call__ at 0x7f2f3ae80c80>, 'decode': <function Transform.decode at 0x7f2f3ae80bf8>, '__repr__': <function Transform.__repr__ at 0x7f2f3ae88048>, '_call': <function Transform._call at 0x7f2f3ae88268>, '_do_call': <function Transform._do_call at 0x7f2f3ae881e0>, '__return__': None}

Once, the dict is setup, __new__ inside _TfmMeta get’s called which I think comes from here and inside the Python Data Model it is said:

When using the default metaclass type , or any metaclass that ultimately calls type.__new__ , the following additional customisation steps are invoked after creating the class object:

  • first, type.__new__ collects all of the descriptors in the class namespace that define a __set_name__() method;
  • second, all of these __set_name__ methods are called with the class being defined and the assigned name of that particular descriptor;
  • finally, the __init_subclass__() hook is called on the immediate parent of the new class in its method resolution order.
1 Like