Duckduckgo search not working

Part 1 2022
1

Hi all!
It’s my first post here, I just started trying to run the cells by myself on the course and I’m already having trouble in the beginning, when we’re supposed to download the bird image from duckduckgo. That’s the code:

from duckduckgo_search import ddg_images
from fastcore.all import *
def search_images(term, max_images=200): return L(ddg_images(term,max_results=max_images)).itemgot('image')
urls = search_images('bird photos', max_images=1)
urls[0]

error:

IndexError                                Traceback (most recent call last)
<ipython-input-35-aa080084010c> in <cell line: 3>()
      1 # urls = search_images('bird photos', max_images=1)
      2 urls = search_images('bird photos', max_images=1)
----> 3 urls[0]

1 frames
/usr/local/lib/python3.10/dist-packages/fastcore/foundation.py in __getitem__(self, idx)
    110     def _xtra(self): return None
    111     def _new(self, items, *args, **kwargs): return type(self)(items, *args, use_list=None, **kwargs)
--> 112     def __getitem__(self, idx): return self._get(idx) if is_indexer(idx) else L(self._get(idx), use_list=None)
    113     def copy(self): return self._new(self.items.copy())
    114 

/usr/local/lib/python3.10/dist-packages/fastcore/foundation.py in _get(self, i)
    114 
    115     def _get(self, i):
--> 116         if is_indexer(i) or isinstance(i,slice): return getattr(self.items,'iloc',self.items)[i]
    117         i = mask2idxs(i)
    118         return (self.items.iloc[list(i)] if hasattr(self.items,'iloc')

IndexError: list index out of range

it seems like the search isn’t returning any results

print(urls)
[]

urls is an empty array. ive basically just copy and pasted the code from the 00-is-it-a-bird-creating-a-model-from-your-own-data.ipynb notebook on colab to my personal studying colab. Actually on both the function runs fine only once but if I try to run the cell again it won’t let me.
I tried to use only the ddg_images function according to duckduckgo-search · PyPI

result = ddg_images('bird photo')
print(result)
None

I searched in the forum and saw some suggestions to see the underlying code and rewrite it but I don’t understand why it works once and then it stops working and also the doc on DDG seems pretty straightforward. So if anyone has some input about it would be great because I am not sure how to go about rewriting the underlying function.
I guess I can always go and download an image myself but I since recommendation number 1 is to try to run the notebooks myself and be active in the forums I guess it’s worth the try.

Appreciate the time and look forward to interact in the forums over the next weeks :slight_smile:

4 Likes
2

I think one cause might be the API rate limiting of DuckDuckGo. It imposes rate limits on its API to prevent abuse and ensure reliable service. If the rate limit is exceeded, the API may return an empty list.

Try disconnecting your run time and restart. This time let max_images=30 for search_images. See if that works.

3

Same thing. O actually set it to 10 (def search_images(term, max_images=10)). It prints the image of a bird once but if I run the cell again it brings an empty array in all following tries. It’s weird that it keeps bringing always the same image from the example notebook since if you actually search for “bird photos” on DDG that will be like the 5th picture. I actually tried to use Bing or Google but on Bing I couldn’t even find the API key in the Azure console and Google actually asks for an API key and a search engine ID. I am not sure if those aren’t matching but I’m getting error 403 when trying to use the API. Anyway those would be just a workaround, if would be best to get the DDG API to work so I could follow the course the way it’s set up

4

I am having the same issue right now. Working for the first image but not the rest…

5

I have the same issue. What is DuckDuckGo? Is there another way to easily collect images given a search term?

6

Same is happening for me, maybe something going with DuckDuckGo search at the moment?

7

Think I’ve found an Issue about this on duckduckgo_search github, it’s fresh.

The problem seems to be happening after latest update, unfortunately we can’t install an older version, since they are no longer working.

1 Like
8

since you posted the link to the issue on github a contributor said that the API has changed and caused the issue and he is already working on repairing it. I guess we should wait a few hours/days :crossed_fingers:t2:

2 Likes
9

Having same issue from last 2 days. I think after re starting the router(changing public ip address) It worked for once.

10

yes it only works once also if you clean API cache which someone on the Github issues used as a workaround. The owner looks like is still working on the issue and by the quantity of likes on his comment it looks like several other people are having the same issue. If anyone can provide a link to another image search api documentation to get the job done I’ll be happy to stop waiting for the fix at DDG at this point :slight_smile:

11

I think maybe you should use the “fastbook” repo to learn?
However, search_images_ddg(your_topic) could work

12

facing the same issue. switched to search_images_bing. It isn’t very difficult to secure their api keys and they let you query thrice every one second as part of the free trial.

13

Use search_images_ddg from fastbook instead it still works.

2 Likes
14

Could you share how you were to secure the api key? I’ve searched all over that azure console and couldn’t find it

15

from duckduckgo_search import ddg_images
from fastcore.all import *
def search_images(term, max_images=200): return L(ddg_images(term, max_results=max_images)).itemgot(‘image’)
urls = search_images(‘bird photos’, max_images=1)

is the implementation like this?

16

I have recorded a loom for you explaining the steps.

1 Like
17

you don’t need an api key to use duck duck go, you may be trying to use bing search, finding and using an api key is described in the video tutorial as a “pain”, so not the recommended path.

18 
# Check fastbook is installed
!pip install fastbook
# Import fastbook functions, in our case we'll be using: search_images_ddg
from fastbook import *
# create a function that will take an arbitrary search term and return a list of urls.
def search_images(term, max=30):
  print(f"Searching for '{term}'")
  # search_images_ddg comes from fastbook: https://github.com/fastai/fastbook/blob/master/utils.py#L45
  return search_images_ddg(term, max_images=max)

urls = search_images("cats eating ice cream")
14 Likes
19

with given function it returns

name ‘L’ is not defined

Do I need to install fastcore also?

20

Yes, since ddg was giving me the same issues, decided to go for bing search API.
The above video was really helpful, but would like to add that it ll only work once you have started the free trial (had to add my CC) but bing search api offers free 1K searches per month, so I m guessing it should be fine.

Also, for those who are looking to shift to bing search instead of ddg, you can refer to the fastai book (production).

I ll share the snipped of the code I used, in case it ll be helpful for others who want to shift to bing api. Don’t forget to import the necessary fastai /fastbook libraries.


key = os.environ.get('AZURE_SEARCH_KEY', 'your key here') //add key variable


bear_types = 'grizzly','black','teddy'
path = Path('bears')

from time import sleep

if not path.exists():
    path.mkdir()
    for o in bear_types:
        dest = (path/o)
        dest.mkdir(exist_ok=True)
        results = search_images_bing(key, f'{o} bear',  min_sz=128, max_images=150)
        download_images(dest, urls=results.attrgot('contentUrl'))
        sleep(10)  # Pause between searches to avoid over-loading server
1 Like