So we provide wgts for each class and it’ll draw examples for that class with given probability (wgts) ?
I’m not sure if this is a bug, or just unintuitive behavior.
When I try to setup a learner with no weight decay, it doesn’t appear to be so.
learn = cnn_learner(dls, resnet18, wd=0.0)
When I enter learn.opt_func in a notebook and run the cell, it gives me the following output
<function fastai2.optimizer.Adam(params, lr, mom=0.9, sqr_mom=0.99, eps=1e-05, wd=0.01, decouple_wd=True)>
which suggests that the opt_func is still setup with wd=0.01.
Do I need to pass in learn.fit(wd=0.0) to truly fit without weight decay?
The learn.wd is not used when you create the optimizer but it is passed each time you call a fit method with no wd. So don’t worry, it will fit without weight decay.
1 Like
Good to know, thank you!
What’s the difference between passing in wd when creating cnn_learner(...) vs calling learn.fit(wd=...)? If they refer to the same, then does the wd passed in when calling learn.fit() override the other option?
You might want to call fit with a different wd each time, so that’s why we have that
and as Sylvain pointed out:
So yes, if you pass wd to fit it overrides
3 Likes
That makes it crystal clear. Thank you.
What is the correct way of exporting just the model from unet_learner?
I tried:
import dill
torch.save(learn.model,"unet-best-2.pth",pickle_module=dill)
I also tried withouth using dill and it throws me an error when I make model=torch.load("unet-best-2.pth") in other project.
SOLUTION: using torchscript for saving the model made the trick!
Do you know any better approach?
1 Like
After installing pytorch-nightly I am getting
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-1-1285ae11e650> in <module>
----> 1 from fastai2.basics import *
2 from fastai2.vision import models
3 from fastai2.vision.all import *
4 from fastai2.metrics import *
5 from fastai2.data.all import *
~/anaconda3/envs/seg1/lib/python3.7/site-packages/fastai2/basics.py in <module>
----> 1 from .data.all import *
2 from .optimizer import *
3 from .callback.core import *
4 from .learner import *
5 from .metrics import *
~/anaconda3/envs/seg1/lib/python3.7/site-packages/fastai2/data/all.py in <module>
----> 1 from ..torch_basics import *
2 from .core import *
3 from .load import *
4 from .external import *
5 from .transforms import *
~/anaconda3/envs/seg1/lib/python3.7/site-packages/fastai2/torch_basics.py in <module>
2 from .imports import *
3 from .torch_imports import *
----> 4 from .torch_core import *
5 from .layers import *
~/anaconda3/envs/seg1/lib/python3.7/site-packages/fastai2/torch_core.py in <module>
283 setattr(TensorBase, fn, get_f(fn))
284
--> 285 _patch_tb()
286
287 # Cell
~/anaconda3/envs/seg1/lib/python3.7/site-packages/fastai2/torch_core.py in _patch_tb()
279 for fn in dir(t):
280 if fn in skips: continue
--> 281 f = getattr(t, fn)
282 if isinstance(f, (MethodWrapperType, BuiltinFunctionType, BuiltinMethodType, MethodType, FunctionType)):
283 setattr(TensorBase, fn, get_f(fn))
RuntimeError: imag is not implemented for tensors with non-complex dtypes.
I don’t believe fastai has been made for nightly (I don’t even think it worked with nightly in v1?) @sgugger can comment on that more
1 Like
Sad, I have my learners exported. However, I need to load them in pytorch-nightly so I can export the models again to TorchScript
1 Like
Hmmm… I may try looking into this myself then for my fastinference library 
(as if you need it, many people do too). I won’t promise I’ll get it working but I’ll try!
What are you going to add exactly? I am interested.
I follow you on github and I saw that you make this repository. What is the aim of this package?
A light-weight inference module for fastai (in terms of what’s needed, fastai2 is still required, and speed optimizations), along with a suite of inference functions to help out with it too. There’s currently ONNX support along with optimized inference pipelines. I’d add specifically export to torch script and nightly.
(For instance text here: https://muellerzr.github.io/fastinference/text.inference/)
2 Likes
Good to know. The other day I mentioned that making inference with fastai2 on new data was some kind of strange.
How so? Normally you just do learn.dls.test_dl, and depending on the dl transforms it should be able to generate it
I find the need of passing the data to learn.dls.test_dl strange.
Let me know if you need any help with that.
I have been facing may issues with that in the past 2 weeks. However, I have understood how it works now.
I’m working on getting a method for Tab + Vision working (that makes sense to me). For the most part it seems like when we’re actually training, from the DataLoader itself we just call .one_batch(). With this in mind will this keep my batches together how I want if I choose to shuffle the DataLoader? Or is this the wrong way to look at it (cc @sgugger)
class MixedDL():
def __init__(self, tab_dl, vision_dl, device='cuda:0'):
self.device = device
tab_dl.shuffle_fn = self.shuffle_fn
vision_dl.shuffle_fn = self.shuffle_fn
self.dls = [tab_dl, vision_dl]
self.count = 0
self.fake_l = _FakeLoader(self, False, 0, 0)
def __len__(self): return len(self.dls[0])
def __iter__(self):
z = zip(*[_loaders[i.fake_l.num_workers==0](i.fake_l) for i in self.dls])
for b in z:
if self.device is not None:
b = to_device(b, self.device)
x,y = [], []
x.extend(self.dls[0].after_batch(b[0])[:2])
x.append(self.dls[1].after_batch(b[1][0]))
y.append(b[1][1])
yield tuple([x,y])
def one_batch(self):
with self.fake_l.no_multiproc(): res = first(self)
if hasattr(self, 'it'): delattr(self, 'it')
return res
def shuffle_fn(self, idxs):
if self.count == 0:
self.rng = self.dls[0].rng.sample(idxs, len(idxs))
self.count += 1
return self.rng
if self.count == 1:
self.count = 0
return self.rng
def show_batch(self):
for dl in self.dls:
dl.show_batch()
def to(self, device): self.device = device
4 Likes