Don’t delete that, that’s much more useful than what I was doing.
Thanks for it!
After I have the Data object, how do I pass it to the DataBlock?
I do have two folders with images, so I create two Data objects pointing to each folder and then pass them to DataBlock?
If you’ve already created a DataLoaders object using the ImageDataLoaders.from_folder method, you don’t need to worry about passing it to a DataBlock . The DataBlock API is just another way to create a DataLoaders object that gives you more control over how your data is processed.
I see I see
Thank you, I will give it a try using your method as it looks more clean.
After training my model, I got terrible results. It’s not getting the categorization accurately.
What’s the process of fixing this? I suppose I will learn in other lessons, as I just started lesson 2.
1.When you run a Colab notebook and download images using code in the notebook, the images are downloaded and stored in the cloud where the Colab notebook is running(same of Kaggle nbs). They are not stored on your local computer.
2. The verify_images function is a utility function provided by the fastai library to check if a set of image files can be opened. It does this by attempting to open each image file and returning a list of files that cannot be opened.
For more information about the function you can use these tricks:
-
?function-name:: Shows the definition and docstring for that function -
??function-name:: Shows the source code for that function -
doc(function-name):: Shows the definition, docstring and links to the documentation of the function
(only works with fastai library imported) - Shift+Tab (press once):: See which parameters to pass to a function
- Shift+Tab (press three times):: Get additional information on the method
3.Depends on how imbalanced the classes are in the dataset.This link might help
EDIT: a related answear
4.You can identify overfitting by looking at the traning loss and validation loss,if at some point the training loss is low and the validation loss is high that means that the model is memorizing the traning that and it dosent do well on unseen data (aka overfitting).
1 Like
Thanks so much (Y)
1 Like
try this:
from torchvision import models
resnet34 = models.resnet34(weights='ResNet34_Weights.DEFAULT')
before your code
Facing the following error after making the modificationr:
TypeError: ResNet.forward() got an unexpected keyword argument ‘pretrained’
Code:
from fastai.vision.all import *
from torchvision import models
resnet34 = models.resnet34(weights=‘ResNet34_Weights.DEFAULT’)
path = untar_data(URLs.PETS)/‘images’
def is_cat(x):
return x[0].isupper()
dls = ImageDataLoaders.from_name_func(
path, get_image_files(path), valid_pct=0.2,
label_func=is_cat, item_tfms=Resize(224))
learn = vision_learner(dls, resnet34, metrics=error_rate)
learn.fine_tune(2)
1 Like
You can disable warnings with the following line of code.
import logging; logging.disable(logging.warning)
I see, so do just as @ForBo7 said and just disable warnings
I’m currently going through the is it a bird? notebook on Kaggle, but I ran into an error.
from duckduckgo_search import ddg_images
from fastcore.all import *
def search_images(term, max_images=30):
print(f"Searching for '{term}'")
return L(ddg_images(term, max_results=max_images)).itemgot('image')
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
/tmp/ipykernel_17/1717929076.py in <module>
----> 1 from duckduckgo_search import ddg_images
2 from fastcore.all import *
3
4 def search_images(term, max_images=30):
5 print(f"Searching for '{term}'")
/opt/conda/lib/python3.7/site-packages/duckduckgo_search/__init__.py in <module>
8
9 # isort: off
---> 10 from .compat import (
11 ddg,
12 ddg_answers,
/opt/conda/lib/python3.7/site-packages/duckduckgo_search/compat.py in <module>
2 import warnings
3
----> 4 from .duckduckgo_search import DDGS
5
6 logger = logging.getLogger(__name__)
/opt/conda/lib/python3.7/site-packages/duckduckgo_search/duckduckgo_search.py in <module>
11
12 import requests
---> 13 from requests.exceptions import HTTPError, JSONDecodeError, Timeout
14 from requests.models import Response
15
ImportError: cannot import name 'JSONDecodeError' from 'requests.exceptions' (/opt/conda/lib/python3.7/site-packages/requests/exceptions.py)
It looks like an error from the duckduckgo_search package, but is it because the code is out of date? What else can I use instead? Is there a latest version of the code maybe?
2 Likes
you can try Google Images Search: Google-Images-Search · PyPI
I tried substituting googleimagesearch into the pip install and got a bunch of errors.
Then got this 'Nonetype" object is not subscriptable error when I add this code
gis = GoogleImagesSearch('XXXXXXXXXX', 'XXXXXX')
_search_params = {
'q': 'birds',
'num': 1,
}
urls = gis.search(_search_params)
urls[0]
What have I done wrong?
I’m running into the same issue as well
This is a “known issue”, at least there is a thread about this.
The following post solved it for me: Duckduckgo search not working - #18 by maitland
Thank you that one was solved. Got a new one though.
download_url(search_images_ddg(‘forest photos’, max_images=1)[0], ‘forest.jpg’, show_progress=False)
Image.open(‘forest.jpg’).to_thumb(256,256)
Missing: Training Diagnostics
When I run learn.fine_tune(3) in the notebooks or in iPython, I don’t get any tables returned with the training log. Can anyone suggest why this is happening and how I could fix it?
I’ve tried the following:
- running the cell as-is in the given notebook (no changes)
- using a rewritten notebook and python file
- rerouting the stdout to a text file, and there was nothing there
For all, there’s no output. Can anyone suggest why this is happening and how I could fix it?
TIA!
Solved!
Refer to this post: fastprogress progress bar not showing · Issue #13163 · microsoft/vscode-jupyter · GitHub
1 Like
AssertionError: Exception occured in Recorder when calling event after_batch:
For lab1 (Is it a bird?) I’m trying to modify the learner to work in a multiclass setting, i.e. with 3 categories instead of 2.
According to the tutorial in the docs this is straightforward, but I’m having trouble with fine-tuning the model.
When re-training with an optimal learning rate for 3 epochs the following rather complex error message was produced:
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
/tmp/ipykernel_18/568062170.py in <module>
10 learn2=vision_learner(dls2, resnet18, metrics=[partial(accuracy_multi, thresh=0.5), fbeta_macro, fbeta_sample] )
11 learn2.lr_find() #find the optimal learning rate
---> 12 learn2.fine_tune(3, 1e-3) #args: n epochs, learning rate. Not in documentation for some reason...ugh
/opt/conda/lib/python3.7/site-packages/fastai/callback/schedule.py in fine_tune(self, epochs, base_lr, freeze_epochs, lr_mult, pct_start, div, **kwargs)
163 "Fine tune with `Learner.freeze` for `freeze_epochs`, then with `Learner.unfreeze` for `epochs`, using discriminative LR."
164 self.freeze()
--> 165 self.fit_one_cycle(freeze_epochs, slice(base_lr), pct_start=0.99, **kwargs)
166 base_lr /= 2
167 self.unfreeze()
/opt/conda/lib/python3.7/site-packages/fastai/callback/schedule.py in fit_one_cycle(self, n_epoch, lr_max, div, div_final, pct_start, wd, moms, cbs, reset_opt, start_epoch)
117 scheds = {'lr': combined_cos(pct_start, lr_max/div, lr_max, lr_max/div_final),
118 'mom': combined_cos(pct_start, *(self.moms if moms is None else moms))}
--> 119 self.fit(n_epoch, cbs=ParamScheduler(scheds)+L(cbs), reset_opt=reset_opt, wd=wd, start_epoch=start_epoch)
120
121 # %% ../../nbs/14_callback.schedule.ipynb 50
/opt/conda/lib/python3.7/site-packages/fastai/learner.py in fit(self, n_epoch, lr, wd, cbs, reset_opt, start_epoch)
262 self.opt.set_hypers(lr=self.lr if lr is None else lr)
263 self.n_epoch = n_epoch
--> 264 self._with_events(self._do_fit, 'fit', CancelFitException, self._end_cleanup)
265
266 def _end_cleanup(self): self.dl,self.xb,self.yb,self.pred,self.loss = None,(None,),(None,),None,None
/opt/conda/lib/python3.7/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
197
198 def _with_events(self, f, event_type, ex, final=noop):
--> 199 try: self(f'before_{event_type}'); f()
200 except ex: self(f'after_cancel_{event_type}')
201 self(f'after_{event_type}'); final()
/opt/conda/lib/python3.7/site-packages/fastai/learner.py in _do_fit(self)
251 for epoch in range(self.n_epoch):
252 self.epoch=epoch
--> 253 self._with_events(self._do_epoch, 'epoch', CancelEpochException)
254
255 def fit(self, n_epoch, lr=None, wd=None, cbs=None, reset_opt=False, start_epoch=0):
/opt/conda/lib/python3.7/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
197
198 def _with_events(self, f, event_type, ex, final=noop):
--> 199 try: self(f'before_{event_type}'); f()
200 except ex: self(f'after_cancel_{event_type}')
201 self(f'after_{event_type}'); final()
/opt/conda/lib/python3.7/site-packages/fastai/learner.py in _do_epoch(self)
246 def _do_epoch(self):
247 self._do_epoch_train()
--> 248 self._do_epoch_validate()
249
250 def _do_fit(self):
/opt/conda/lib/python3.7/site-packages/fastai/learner.py in _do_epoch_validate(self, ds_idx, dl)
242 if dl is None: dl = self.dls[ds_idx]
243 self.dl = dl
--> 244 with torch.no_grad(): self._with_events(self.all_batches, 'validate', CancelValidException)
245
246 def _do_epoch(self):
/opt/conda/lib/python3.7/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
197
198 def _with_events(self, f, event_type, ex, final=noop):
--> 199 try: self(f'before_{event_type}'); f()
200 except ex: self(f'after_cancel_{event_type}')
201 self(f'after_{event_type}'); final()
/opt/conda/lib/python3.7/site-packages/fastai/learner.py in all_batches(self)
203 def all_batches(self):
204 self.n_iter = len(self.dl)
--> 205 for o in enumerate(self.dl): self.one_batch(*o)
206
207 def _backward(self): self.loss_grad.backward()
/opt/conda/lib/python3.7/site-packages/fastai/learner.py in one_batch(self, i, b)
233 b = self._set_device(b)
234 self._split(b)
--> 235 self._with_events(self._do_one_batch, 'batch', CancelBatchException)
236
237 def _do_epoch_train(self):
/opt/conda/lib/python3.7/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
199 try: self(f'before_{event_type}'); f()
200 except ex: self(f'after_cancel_{event_type}')
--> 201 self(f'after_{event_type}'); final()
202
203 def all_batches(self):
/opt/conda/lib/python3.7/site-packages/fastai/learner.py in __call__(self, event_name)
170
171 def ordered_cbs(self, event): return [cb for cb in self.cbs.sorted('order') if hasattr(cb, event)]
--> 172 def __call__(self, event_name): L(event_name).map(self._call_one)
173
174 def _call_one(self, event_name):
/opt/conda/lib/python3.7/site-packages/fastcore/foundation.py in map(self, f, *args, **kwargs)
154 def range(cls, a, b=None, step=None): return cls(range_of(a, b=b, step=step))
155
--> 156 def map(self, f, *args, **kwargs): return self._new(map_ex(self, f, *args, gen=False, **kwargs))
157 def argwhere(self, f, negate=False, **kwargs): return self._new(argwhere(self, f, negate, **kwargs))
158 def argfirst(self, f, negate=False):
/opt/conda/lib/python3.7/site-packages/fastcore/basics.py in map_ex(iterable, f, gen, *args, **kwargs)
838 res = map(g, iterable)
839 if gen: return res
--> 840 return list(res)
841
842 # %% ../nbs/01_basics.ipynb 336
/opt/conda/lib/python3.7/site-packages/fastcore/basics.py in __call__(self, *args, **kwargs)
823 if isinstance(v,_Arg): kwargs[k] = args.pop(v.i)
824 fargs = [args[x.i] if isinstance(x, _Arg) else x for x in self.pargs] + args[self.maxi+1:]
--> 825 return self.func(*fargs, **kwargs)
826
827 # %% ../nbs/01_basics.ipynb 326
/opt/conda/lib/python3.7/site-packages/fastai/learner.py in _call_one(self, event_name)
174 def _call_one(self, event_name):
175 if not hasattr(event, event_name): raise Exception(f'missing {event_name}')
--> 176 for cb in self.cbs.sorted('order'): cb(event_name)
177
178 def _bn_bias_state(self, with_bias): return norm_bias_params(self.model, with_bias).map(self.opt.state)
/opt/conda/lib/python3.7/site-packages/fastai/callback/core.py in __call__(self, event_name)
60 try: res = getcallable(self, event_name)()
61 except (CancelBatchException, CancelBackwardException, CancelEpochException, CancelFitException, CancelStepException, CancelTrainException, CancelValidException): raise
---> 62 except Exception as e: raise modify_exception(e, f'Exception occured in `{self.__class__.__name__}` when calling event `{event_name}`:\n\t{e.args[0]}', replace=True)
63 if event_name=='after_fit': self.run=True #Reset self.run to True at each end of fit
64 return res
/opt/conda/lib/python3.7/site-packages/fastai/callback/core.py in __call__(self, event_name)
58 res = None
59 if self.run and _run:
---> 60 try: res = getcallable(self, event_name)()
61 except (CancelBatchException, CancelBackwardException, CancelEpochException, CancelFitException, CancelStepException, CancelTrainException, CancelValidException): raise
62 except Exception as e: raise modify_exception(e, f'Exception occured in `{self.__class__.__name__}` when calling event `{event_name}`:\n\t{e.args[0]}', replace=True)
/opt/conda/lib/python3.7/site-packages/fastai/learner.py in after_batch(self)
558 if len(self.yb) == 0: return
559 mets = self._train_mets if self.training else self._valid_mets
--> 560 for met in mets: met.accumulate(self.learn)
561 if not self.training: return
562 self.lrs.append(self.opt.hypers[-1]['lr'])
/opt/conda/lib/python3.7/site-packages/fastai/learner.py in accumulate(self, learn)
480 def accumulate(self, learn):
481 bs = find_bs(learn.yb)
--> 482 self.total += learn.to_detach(self.func(learn.pred, *learn.yb))*bs
483 self.count += bs
484 @property
/opt/conda/lib/python3.7/site-packages/fastai/metrics.py in accuracy_multi(inp, targ, thresh, sigmoid)
199 def accuracy_multi(inp, targ, thresh=0.5, sigmoid=True):
200 "Compute accuracy when `inp` and `targ` are the same size."
--> 201 inp,targ = flatten_check(inp,targ)
202 if sigmoid: inp = inp.sigmoid()
203 return ((inp>thresh)==targ.bool()).float().mean()
/opt/conda/lib/python3.7/site-packages/fastai/torch_core.py in flatten_check(inp, targ)
785 "Check that `inp` and `targ` have the same number of elements and flatten them."
786 inp,targ = TensorBase(inp.contiguous()).view(-1),TensorBase(targ.contiguous()).view(-1)
--> 787 test_eq(len(inp), len(targ))
788 return inp,targ
789
/opt/conda/lib/python3.7/site-packages/fastcore/test.py in test_eq(a, b)
35 def test_eq(a,b):
36 "`test` that `a==b`"
---> 37 test(a,b,equals, cname='==')
38
39 # %% ../nbs/00_test.ipynb 25
/opt/conda/lib/python3.7/site-packages/fastcore/test.py in test(a, b, cmp, cname)
25 "`assert` that `cmp(a,b)`; display inputs and `cname or cmp.__name__` if it fails"
26 if cname is None: cname=cmp.__name__
---> 27 assert cmp(a,b),f"{cname}:\n{a}\n{b}"
28
29 # %% ../nbs/00_test.ipynb 16
AssertionError: Exception occured in `Recorder` when calling event `after_batch`:
==:
96
32
Here is my code:
#establish which DL model to use
#here we need to change the perforamnce metric as unidim error rate won't work
#brie_pop=BrierScoreMulti(thresh=0.4) did not work due to non-mutual exclusivity of strictly proper scoring rule?
fbeta_macro=FBetaMulti(beta=1.1, thresh=0.5, average='macro')
fbeta_macro.name='FBeta (macro)'
fbeta_sample=FBetaMulti(beta=1.1, thresh=0.5, average='samples')
fbeta_macro.name='FBeta (samples)'
#multiclass modification
learn2=vision_learner(dls2, resnet18, metrics=[partial(accuracy_multi, thresh=0.5), fbeta_macro, fbeta_sample] )
learn2.lr_find() #find the optimal learning rate
learn2.fine_tune(3, 1e-3) #args: n epochs, learning rate. Not in documentation for some reason...ugh
Has anyone else experienced a similar error, or possesses the API expertise to find the root cause of this issue?
Thanks for your time!