Actually @pbanavara you do know; PCA (principal components analysis) forms a set of new features that are independent linear combinations of the original input features, and keeps only those new features that have the strongest effect. The idea is, you gain a simpler model at the expense of some information loss. If you want to, you can find out exactly which new features have been dropped. However, in most cases it would be difficult to interpret these dropped features.
In this paper they use attention mechanism to outperform factorization machines.
Happy to discuss – https://arxiv.org/abs/1810.11921
2 Likes
Hi,
I am looking for resources to get started on processing 3D images. As a simple 3D image categorization task but I am not sure where to get started. I found 3D mnist data. Will the same principles that we have been using to identify the right learning rate work for 3D images as well.
I’m still stuck at how should I use fastai collaborative filtering learner to recommend in production. anyone have any end-to-end example ? I already trained it but I’m not sure how should I use it
Update: I could make it work. My next post will describe how.
=================================================
I am trying to import models from the Cadene Pytorch model libraries.
I am trying for 2 days to make create_cnn working, but seems only learn = Learner(data, model) works. create_cnn is much better, but there is something I am missing. I understand that create_cnn expects a callable function and tried the following examples.
Here are working examples:
# this works
def get_model(pretrained=True, model_name = 'resnet50', **kwargs ): # pretrained key should be the first
arch = models.resnet50(pretrained, **kwargs )
return arch
learn = Learner(data, get_model(), metrics=[accuracy])
# this works
def get_model(pretrained=True, model_name = 'resnet50', **kwargs ): # pretrained key should be the first
arch = pretrainedmodels.__dict__[model_name](num_classes=1000, pretrained='imagenet')
return arch
learn = Learner(data, get_model(), metrics=[accuracy])
Here are not working examples:
# this does not work
def get_model(pretrained=True, model_name = 'resnet50', **kwargs ): # pretrained key should be the first
arch = pretrainedmodels.__dict__[model_name](num_classes=1000, pretrained='imagenet')
return arch
learn = Learner(data, get_model, metrics=[accuracy])
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-45-814e6d5c8795> in <module>
4 return arch
5
----> 6 learn = Learner(data, get_model, metrics=[accuracy])
<string> in __init__(self, data, model, opt_func, loss_func, metrics, true_wd, bn_wd, wd, train_bn, path, model_dir, callback_fns, callbacks, layer_groups)
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/basic_train.py in __post_init__(self)
145 self.path = Path(ifnone(self.path, self.data.path))
146 (self.path/self.model_dir).mkdir(parents=True, exist_ok=True)
--> 147 self.model = self.model.to(self.data.device)
148 self.loss_func = ifnone(self.loss_func, self.data.loss_func)
149 self.metrics=listify(self.metrics)
AttributeError: 'function' object has no attribute 'to'
# this does not work
def get_model(pretrained=True, model_name = 'resnet50', **kwargs ): # pretrained key should be the first
arch = pretrainedmodels.__dict__[model_name](num_classes=1000, pretrained='imagenet')
return arch
learn = create_cnn(data, get_model(), metrics=[accuracy])
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-43-74cb1e5cde9c> in <module>
4 return arch
5
----> 6 learn = create_cnn(data, get_model(), metrics=[accuracy])
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/vision/learner.py in create_cnn(data, arch, cut, pretrained, lin_ftrs, ps, custom_head, split_on, bn_final, **kwargs)
53 "Build convnet style learners."
54 meta = cnn_config(arch)
---> 55 body = create_body(arch, pretrained, cut)
56 nf = num_features_model(body) * 2
57 head = custom_head or create_head(nf, data.c, lin_ftrs, ps=ps, bn_final=bn_final)
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/vision/learner.py in create_body(arch, pretrained, cut, body_fn)
30 def create_body(arch:Callable, pretrained:bool=True, cut:Optional[int]=None, body_fn:Callable[[nn.Module],nn.Module]=None):
31 "Cut off the body of a typically pretrained `model` at `cut` or as specified by `body_fn`."
---> 32 model = arch(pretrained)
33 if not cut and not body_fn: cut = cnn_config(arch)['cut']
34 return (nn.Sequential(*list(model.children())[:cut]) if cut
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
487 result = self._slow_forward(*input, **kwargs)
488 else:
--> 489 result = self.forward(*input, **kwargs)
490 for hook in self._forward_hooks.values():
491 hook_result = hook(self, input, result)
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/pretrainedmodels/models/torchvision_models.py in forward(self, input)
338
339 def forward(self, input):
--> 340 x = self.features(input)
341 x = self.logits(x)
342 return x
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/pretrainedmodels/models/torchvision_models.py in features(self, input)
320
321 def features(self, input):
--> 322 x = self.conv1(input)
323 x = self.bn1(x)
324 x = self.relu(x)
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
487 result = self._slow_forward(*input, **kwargs)
488 else:
--> 489 result = self.forward(*input, **kwargs)
490 for hook in self._forward_hooks.values():
491 hook_result = hook(self, input, result)
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
318 def forward(self, input):
319 return F.conv2d(input, self.weight, self.bias, self.stride,
--> 320 self.padding, self.dilation, self.groups)
321
322
TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not bool
# this does not work
def get_model(pretrained=True, model_name = 'resnet50', **kwargs ): # pretrained key should be the first
arch = pretrainedmodels.__dict__[model_name](num_classes=1000, pretrained='imagenet')
return arch
learn = create_cnn(data, get_model, metrics=[accuracy])
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-44-86a48fcf2401> in <module>
4 return arch
5
----> 6 learn = create_cnn(data, get_model, metrics=[accuracy])
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/vision/learner.py in create_cnn(data, arch, cut, pretrained, lin_ftrs, ps, custom_head, split_on, bn_final, **kwargs)
54 meta = cnn_config(arch)
55 body = create_body(arch, pretrained, cut)
---> 56 nf = num_features_model(body) * 2
57 head = custom_head or create_head(nf, data.c, lin_ftrs, ps=ps, bn_final=bn_final)
58 model = nn.Sequential(body, head)
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/callbacks/hooks.py in num_features_model(m)
115 def num_features_model(m:nn.Module)->int:
116 "Return the number of output features for `model`."
--> 117 return model_sizes(m)[-1][1]
118
119 def total_params(m:nn.Module)->int:
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/callbacks/hooks.py in model_sizes(m, size)
110 "Pass a dummy input through the model `m` to get the various sizes of activations."
111 with hook_outputs(m) as hooks:
--> 112 x = dummy_eval(m, size)
113 return [o.stored.shape for o in hooks]
114
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/fastai/callbacks/hooks.py in dummy_eval(m, size)
105 def dummy_eval(m:nn.Module, size:tuple=(64,64)):
106 "Pass a `dummy_batch` in evaluation mode in `m` with `size`."
--> 107 return m.eval()(dummy_batch(m, size))
108
109 def model_sizes(m:nn.Module, size:tuple=(64,64))->Tuple[Sizes,Tensor,Hooks]:
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
487 result = self._slow_forward(*input, **kwargs)
488 else:
--> 489 result = self.forward(*input, **kwargs)
490 for hook in self._forward_hooks.values():
491 hook_result = hook(self, input, result)
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
90 def forward(self, input):
91 for module in self._modules.values():
---> 92 input = module(input)
93 return input
94
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
487 result = self._slow_forward(*input, **kwargs)
488 else:
--> 489 result = self.forward(*input, **kwargs)
490 for hook in self._forward_hooks.values():
491 hook_result = hook(self, input, result)
~/anaconda3/envs/fastai-v1/lib/python3.7/site-packages/torch/nn/modules/pooling.py in forward(self, input)
563 def forward(self, input):
564 return F.avg_pool2d(input, self.kernel_size, self.stride,
--> 565 self.padding, self.ceil_mode, self.count_include_pad)
566
567
RuntimeError: Given input size: (2048x2x2). Calculated output size: (2048x-4x-4). Output size is too small at /opt/conda/conda-bld/pytorch_1544202130060/work/aten/src/THNN/generic/SpatialAveragePooling.c:48
A working example shared here to import any Cadene model is very helpful. It will expand the usable models in fastai for everybody. I heard in Kaggle competitions that there are people could make it work with fastai v1, but could not do it myself. Here is an 8th Kaggle winner in Human Protein Atlas comp. that just did that!
2 Likes
Here is my code importing resnet50 from Cadene Pytorch pretrained models repo. I used resnet50 to make sure that I did not make any mistake which can be validated by comparing it with a normal fastai resnet50 method. The below code can give you access to ~45 models most of them still not available in fastai by simply changing the model’s name.
pretrainedmodels should be installed by:
pip install pretrainedmodels
Our lesson-1 example using pretrained models import:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
from fastai import *
from fastai.vision import *
import pretrainedmodels
path = untar_data(URLs.PETS); path
path_anno = path/'annotations'
path_img = path/'images'
fnames = get_image_files(path_img)
np.random.seed(2)
pat = re.compile(r'/([^/]+)_\d+.jpg$')
bs = 64
data = ImageDataBunch.from_name_re(path_img, fnames, pat, ds_tfms=get_transforms(),
size=299, bs=bs//2).normalize(imagenet_stats)
def get_model(pretrained=True, model_name = 'resnet50', **kwargs ):
if pretrained:
arch = pretrainedmodels.__dict__[model_name](num_classes=1000, pretrained='imagenet')
else:
arch = pretrainedmodels.__dict__[model_name](num_classes=1000, pretrained=None)
return arch
custom_head = create_head(nf=2048*2, nc=37, ps=0.5, bn_final=False)
# Below you can change the imported model into any of the models available in the `pretrainedmodels`
# which can be shown by: pretrainedmodels.model_names
fastai_resnet50=nn.Sequential(*list(children(get_model(model_name = 'resnet50'))[:-2]),custom_head)
def get_fastai_model(pretrained=True, **kwargs ):
return fastai_resnet50
learn = create_cnn(data, get_fastai_model, metrics=error_rate)
learn.fit_one_cycle(5)
learn.unfreeze()
learn.fit_one_cycle(1, max_lr=slice(1e-6,1e-4))
Update-1:
Maybe it is safer to create the custom_head as the following:
custom_head = create_head(nf=num_features_model(self.cnn)*2, nc=data.c, ps=0.5, bn_final=False)
Just in case the output features of the body of a model that you will use is not 2048
Update-2:
Jeremy tweeted an even better implementation for importing architectures from cadene. What I missed in the above code, the points of split for the discriminative learning.
16 Likes
Looking for an answer to your point 2. That makes me wonder, the weights would not change much if a level in a variable doesn’t appear many times in the dataset.
Thanks for this excellent contribution, Haider @hwasiti I plan to try some of these as I progress further into modeling.
1 Like
@sgugger Some of my friends asked me to make pull request to include those pretrained models into fastai.
There are a lot of models claimed to be better on the imageNet benchmark than ResNet which I could import them into fastai using this simple code.
Any chance that you will add them to fastai in the future? or can I make a pull request but then where exactly in the library should I make the changes to add them?
2 Likes
i have learnt that if we pass lin_ftrs=[1024] then soon after adaptive create cnn will create Linear layers with no of Inputs as we put in the list… so we dont need toput our own custom head ,if our requirement is just to change the last Fc layer features with output as no of classses for various trained models…
2 Likes
@jeremy I had some trouble with your macro in the basic worksheet, with the last line in your onestep subroutine. It kept giving me a 400 error. I fixed it with the following changes. First, I changed the subroutine to accept an integer as an argument:
Sub onestep(ByVal i as Integer)
Then, I changed the last line to
Range("msestart").Offset(i, 0).PasteSpecial Paste:=xlPasteValues
And calling the subroutine is then simply
For i = 1 To 5: onestep (i): NextThanks for the post! The link you provided in the second update is a really nice reference. I was able to get everything working on my end, though I had a one thing not working as expected:
Are arch_summary() and get_groups() Fastai methods? If so I think they’ve been replaced by something else or just removed completely. I can’t find anything on the docs. There’s other stuff in the github repo that isn’t updated with the most recent Fastai version, but those two seemed really nice to learn how these models are structured/setting layer groups.
Thanks again for the post!
@jeremy @sgugger How to use Collaborative filtering for the non-rating dataset.e.g. product recommendation based on transaction history. I used TuriCreate API.But, how to implement product recommendation problem based on the transaction in fastai?
Secondly, do fastai support various type of recommendations as present in TuriCreate API?
That’s pretty cool. Do you know of anyone using this in practice?
Thank you @jeremy for the excellent lecture. I have been thinking much about collaborative filtering and I have a few questions to ask.
-
In the example given, the item list in the collaborative filtering dataset does not necessarily need to be movies. It can be anything with a rating to it. Potentially one can put demographic or other user data in there (e.g. if the person is male, then the entry male can be made in the movie column with a rating of 5 to capture this) and the model can learn how these data can influence the user embedding. This may avoid requiring a separate tabular model to use demographic data for the cold start problem. I wonder if anyone has tried this approach?
-
The more interesting idea is to use collaborative filtering in medical diagnostics. Most patients only have a few diseases, and most diseases only affect a small proportion of patients, so this is not dissimilar to the movie recommendation problem. Potentially, if there is a dataset of a large number of patients with their diagnostic coding, then a collaborative filtering system can figure out if what diseases a patient may be most susceptible to given their previous diagnoses.
-
Collaborative filtering can also be used to impute missing values. This is again of importance in medicine as most patients will not have had all the tests / investigations that is available. Potentially as more information is known about a patient, a system can impute and predict test results that have not been performed.
Does anyone know if a collaborative filtering approach has been used in medical research / diagnostics?
Hi! I’m trying to implement a net as discussed in this lesson, but I’m getting the following error:
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-30-36d1490bd9f3> in <module>()
----> 1 losses = [update(x,y,lr) for x,y in data.train_dl]
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
1404 if input.dim() == 2 and bias is not None:
1405 # fused op is marginally faster
-> 1406 ret = torch.addmm(bias, input, weight.t())
1407 else:
1408 output = input.matmul(weight.t())
RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1'
I have checked out my datatypes and everything that I can find is a float, there are no doubles, so I’m confused as to what could be causing this error. Thoughts?
If it helps, I’ve figured out (via source diving), that mat1 here is the bias.
I’ve figured out my problem, when I run:
torch.tensor(some_array)
I get back a DoubleTensor when what I want is a FloatTensor. What is also strange is that if you print that tensor you will see its dtype is float64. Anyways, I’ve gotten around this by calling torch.tensor(some_array).float().
Sorry for the late reply.
Those should be imported from the utils.py (check out this file in the same repo):
from utils import *
BTW. the post for that notebook that I have referred with the link in the 2nd update:
I am uploading link to my hand-written notes on things you can do to improve your neural networks. I wrote these notes while doing deeplearning.ai specialization taught by Andrew Ng. Hope you find these useful. These notes cover in detail topics such as regularization, weight decay, adam optimization , momentum etc. These notes go well with the lesson 5 where Jeremy teaches about Adam optimizing algo ,weight decays,RMSprop etc.
https://drive.google.com/open?id=1dXjZ2boL5pqvxSB7J-zLEdCbTo7XL8MO
2 Likes