See I can not find a way to use duckduckgo to download images
Thank you for your advice! Actually I tried to upgrade my ddgs yesterday, but now the ERROR has turned into “No results found”. It seems that I have to wait for a few seconds before I could use ddgs to search for images again. I also tried some different proxies, but that didn’t work as well. It really confuses me.
Really? I never encountered this error before. Can you share your code snippet?
I’m using:
from ddgs import DDGS # DuckDuckGo has changed the api so we need to update (again)
from fastcore.all import *
from fastdownload import download_url
from fastai.vision.all import *
def search_images(keywords, max_images=200): return L(DDGS().images(keywords, max_results=max_images)).itemgot('image')
urls = search_images('bird photos', max_images=10)
print(urls[0])
dest = 'bird.jpg'
download_url(urls[0], dest, show_progress=False)
im = Image.open(dest)
im.to_thumb(256,256)
works every time for me
Here is my code.
from ddgs import DDGS #DuckDuckGo has changed the api so we need to update
from fastcore.all import *
from itertools import islice
def search_images(keywords, max_images=200):
with DDGS() as ddgs:
return L(r['image'] for r in islice(ddgs.images(query=keywords), max_images))
import time, json
urls = search_images('bird', max_images=1)
urls[0]```
I tried your code on my computer, the first visit was OK but then it came up with the "No result found" ERROR as before. I guess it might be related to the permission difference of ddgs all over the world.I reproduced your problem. I then changed my ip using a vpn and it works again.
So it might be an ip problem where ddgs rate limits you…
I think so, too. I have tried some different VPNs, but did not find a stable one. According to my previous tries I found the American VPN is the most stable but it still can not support a search for even a small dataset. Maybe I could try a different engine or just download some datasets locally to follow up. Thank you for your assistance!
I have tried DDGS multiple times and always run into the DDGS ratelimit error. I switched to bing_image_downloader and it has been working like a charm every time. I use the below code -
!pip install bing_image_downloader
from bing_image_downloader import downloader
from pathlib import Path
CLASSES = ["Forest", "Bird"]
IMAGES_PER_CLASS = 50 # change as needed
BASE_DIR = Path("./images")
def download_images():
for cls in CLASSES:
downloader.download(
query=cls,
limit=IMAGES_PER_CLASS,
output_dir=str(BASE_DIR),
adult_filter_off=True,
force_replace=False,
timeout=60,
verbose=True
)
download_images()
Thank you! Bing is much stabler than DDGS, but it sometimes also fails. I got an answer from stackoverflow, which suggests that we could use multi-region. That works well for me. I share it here.
from ddgs import DDGS
from fastcore.all import *
from itertools import islice
import random
import time
def search_images(keywords, max_images=200):
# Try 'de-de' region first, then fallback to default
regions = ['de-de','uk-en', 'us-en', 'fr-fr','ar-es', 'pl-pl', 'cn-zh','ar-en', 'pl-en', 'cn-en']
for region in regions:
try:
with DDGS(timeout=20) as ddgs:
return L(r['image'] for r in islice(ddgs.images(query=keywords, region=region), max_images))
except:
time.sleep(random.uniform(1.2, 3.6))
continue
return L()
import time, json
from fastdownload import download_url
from fastai.vision.all import *
download_url(search_images('forest', max_images=1)[0], 'forest.jpg', show_progress=False)
Image.open('forest.jpg').to_thumb(256,256)
2 Likes
Brooooo youre a life saverrr 
Instead of using duckduckgo-search , use ddgs .It works fine for me (in india).
I have also tried Bing Image search, but it doesn’t work for more than 15+images per class/keyword.
Any advice on how to use tab autocomplete feature for fastai functions and methods in Kaggle ?
I am unable to get suggestions , also the CPU shoots up 101% on trying to use it (even after being failed to use it)
I tried running %config Completer.use_jedi = False ,
but it doesn’t work .
Any solutions ? Please help.
I’m new to Fastai. I’ve explored the course book’s code in Colab with very encouraging results, but two weeks later, when trying to reproduce the code, I get an error in learner.fine_tune. Could someone help me solve this?
from fastai.vision.all import *
path = untar_data(URLs.PETS)/‘images’
def is_cat(x): return x[0].isupper()
dls = ImageDataLoaders.from_name_func(
path, get_image_files(path), valid_pct=0.2, seed=42,
label_func=is_cat, item_tfms=Resize(224))
learn = vision_learner(dls, resnet34, metrics=error_rate)
learn.fine_tune(1)
progress { appearance: none; border: none; border-radius: 4px; width: 300px;
height: 20px; vertical-align: middle; background: #e0e0e0; }
progress::-webkit-progress-bar { background: #e0e0e0; border-radius: 4px; }
progress::-webkit-progress-value { background: #2196F3; border-radius: 4px; }
progress::-moz-progress-bar { background: #2196F3; border-radius: 4px; }
progress:not([value]) {
background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px); }
progress.progress-bar-interrupted::-webkit-progress-value { background: #F44336; }
progress.progress-bar-interrupted::-moz-progress-value { background: #F44336; }
progress.progress-bar-interrupted::-webkit-progress-bar { background: #F44336; }
progress.progress-bar-interrupted::-moz-progress-bar { background: #F44336; }
progress.progress-bar-interrupted { background: #F44336; }
table.fastprogress { border-collapse: collapse; margin: 1em 0; font-size: 0.9em; }
table.fastprogress th, table.fastprogress td { padding: 8px 12px; border: 1px solid #ddd; text-align: left; }
table.fastprogress thead tr { background: #f8f9fa; font-weight: bold; }
table.fastprogress tbody tr:nth-of-type(even) { background: #f8f9fa; }
100.00% [811712512/811706944 00:12<00:00]Downloading: “https://download.pytorch.org/models/resnet34-b627a593.pth” to /root/.cache/torch/hub/checkpoints/resnet34-b627a593.pth
100%|██████████| 83.3M/83.3M [00:00<00:00, 153MB/s]
AttributeError Traceback (most recent call last)
/tmp/ipython-input-780811548.py in <cell line: 0>()
8
9 learn = vision_learner(dls, resnet34, metrics=error_rate)
—> 10 learn.fine_tune(1)
15 frames/usr/local/lib/python3.12/dist-packages/fastai/callback/schedule.py in fine_tune(self, epochs, base_lr, freeze_epochs, lr_mult, pct_start, div, **kwargs)
165 “Fine tune with Learner.freeze for freeze_epochs, then with Learner.unfreeze for epochs, using discriminative LR.”
166 self.freeze()
→ 167 self.fit_one_cycle(freeze_epochs, slice(base_lr), pct_start=0.99, **kwargs)
168 base_lr /= 2
169 self.unfreeze()
/usr/local/lib/python3.12/dist-packages/fastai/callback/schedule.py in fit_one_cycle(self, n_epoch, lr_max, div, div_final, pct_start, wd, moms, cbs, reset_opt, start_epoch)
119 scheds = {‘lr’: combined_cos(pct_start, lr_max/div, lr_max, lr_max/div_final),
120 ‘mom’: combined_cos(pct_start, *(self.moms if moms is None else moms))}
→ 121 self.fit(n_epoch, cbs=ParamScheduler(scheds)+L(cbs), reset_opt=reset_opt, wd=wd, start_epoch=start_epoch)
122
123 # %% ../../nbs/14_callback.schedule.ipynb 50
/usr/local/lib/python3.12/dist-packages/fastai/learner.py in fit(self, n_epoch, lr, wd, cbs, reset_opt, start_epoch)
270 self.opt.set_hypers(lr=self.lr if lr is None else lr)
271 self.n_epoch = n_epoch
→ 272 self._with_events(self._do_fit, ‘fit’, CancelFitException, self._end_cleanup)
273
274 def _end_cleanup(self): self.dl,self.xb,self.yb,self.pred,self.loss = None,(None,),(None,),None,None
/usr/local/lib/python3.12/dist-packages/fastai/learner.py in with_events(self, f, event_type, ex, final)
205
206 def with_events(self, f, event_type, ex, final=noop):
→ 207 try: self(f’before{event_type}'); f()
208 except ex: self(f’after_cancel{event_type}‘)
209 self(f’after_{event_type}’); final()
/usr/local/lib/python3.12/dist-packages/fastai/learner.py in call(self, event_name)
178
179 def ordered_cbs(self, event): return [cb for cb in self.cbs.sorted(‘order’) if hasattr(cb, event)]
→ 180 def call(self, event_name): L(event_name).map(self._call_one)
181
182 def _call_one(self, event_name):
/usr/local/lib/python3.12/dist-packages/fastcore/foundation.py in wrapper(self, *args, **kwargs)
223 def wrapper(self, *args, **kwargs):
224 if not isinstance(self, L): return lambda items: f(L(items), self, *args, **kwargs)
→ 225 return f(L(self), *args, **kwargs)
226 return wrapper
227
/usr/local/lib/python3.12/dist-packages/fastcore/foundation.py in map(self, f, *args, **kwargs)
231 def map(self:L, f, *args, **kwargs):
232 “Create new L with f applied to all items, passing args and kwargs to f”
→ 233 return self._new(map_ex(self, f, *args, gen=False, **kwargs))
234
235 # %% ../nbs/02_foundation.ipynb #9c5a4633
/usr/local/lib/python3.12/dist-packages/fastcore/basics.py in map_ex(iterable, f, gen, *args, **kwargs)
940 res = map(g, iterable)
941 if gen: return res
→ 942 return list(res)
943
944 # %% ../nbs/01_basics.ipynb #02d08a1b
/usr/local/lib/python3.12/dist-packages/fastcore/basics.py in call(self, *args, **kwargs)
925 if isinstance(v,_Arg): kwargs[k] = args.pop(v.i)
926 fargs = [args[x.i] if isinstance(x, _Arg) else x for x in self.pargs] + args[self.maxi+1:]
→ 927 return self.func(*fargs, **kwargs)
928
929 # %% ../nbs/01_basics.ipynb #84779a02
/usr/local/lib/python3.12/dist-packages/fastai/learner.py in _call_one(self, event_name)
182 def _call_one(self, event_name):
183 if not hasattr(event, event_name): raise Exception(f’missing {event_name}')
→ 184 for cb in self.cbs.sorted(‘order’): cb(event_name)
185
186 def _bn_bias_state(self, with_bias): return norm_bias_params(self.model, with_bias).map(self.opt.state)
/usr/local/lib/python3.12/dist-packages/fastai/callback/core.py in call(self, event_name)
62 try: res = getcallable(self, event_name)()
63 except (CancelBatchException, CancelBackwardException, CancelEpochException, CancelFitException, CancelStepException, CancelTrainException, CancelValidException): raise
—> 64 except Exception as e: raise modify_exception(e, f’Exception occured in {self.__class__.__name__} when calling event {event_name}:\n\t{e.args[0]}', replace=True)
65 if event_name==‘after_fit’: self.run=True #Reset self.run to True at each end of fit
66 return res
/usr/local/lib/python3.12/dist-packages/fastai/callback/core.py in call(self, event_name)
60 res = None
61 if self.run and _run:
—> 62 try: res = getcallable(self, event_name)()
63 except (CancelBatchException, CancelBackwardException, CancelEpochException, CancelFitException, CancelStepException, CancelTrainException, CancelValidException): raise
64 except Exception as e: raise modify_exception(e, f’Exception occured in {self.__class__.__name__} when calling event {event_name}:\n\t{e.args[0]}', replace=True)
/usr/local/lib/python3.12/dist-packages/fastai/callback/progress.py in before_fit(self)
21 if self.learn.logger != noop:
22 self.old_logger,self.learn.logger = self.logger,self._write_stats
—> 23 self._write_stats(self.recorder.metric_names)
24 else: self.old_logger = noop
25
/usr/local/lib/python3.12/dist-packages/fastai/callback/progress.py in _write_stats(self, log)
46
47 def _write_stats(self, log):
—> 48 if getattr(self, ‘mbar’, False): self.mbar.write([f’{l:.6f}’ if isinstance(l, float) else str(l) for l in log], table=True)
49
50 _docs = dict(before_fit=“Setup the master bar over the epochs”,
/usr/local/lib/python3.12/dist-packages/fastprogress/fastprogress.py in write(self, line, table)
237 if table: self.lines.append(line); self.text_parts = [text2html_table(self.lines)]
238 else: self.text_parts.append(P(line))
→ 239 self.show()
240
241 # %% ../nbs/01_fastprogress.ipynb
/usr/local/lib/python3.12/dist-packages/fastprogress/fastprogress.py in show(self)
232 children = [getattr(item, ‘progress’, None) or item for n in self.order
233 if (item := self.inner_dict.get(n))]
→ 234 self.out.update(Div(*children))
235
236 def write(self, line, table=False):
AttributeError: Exception occured in ProgressCallback when calling event before_fit:
‘NBMasterBar’ object has no attribute ‘out’
Hi, I got this error before:
AttributeError: Exception occured in ProgressCallback when calling event before_fit :
‘NBMasterBar’ object has no attribute ‘out’
you just need to set a version of FastProgress lib:
!pip install fastprogress==1.0.3
This might not be the right solution. You could simply call learn.remove_cbs(ProgressCallback) before invoking fine_tune.
Neither duckduckgo-search nor ddgs worked for me. But I got it working using icrawler.
!pip install icrawler
# download with Bing (works well in Colab)
from icrawler.builtin import BingImageCrawler
crawler = BingImageCrawler(storage={'root_dir':'images/grizzly'})
crawler.crawl(keyword='grizzly bear', max_num=200)
# use with fastai
from fastai.vision.all import *
ims = get_image_files('images/grizzly')