Duckduckgo search not working

For anyone running into this, until the fix in the linked github issue is resolved:
The docs indicate that beginning May 12 2023 older duckduckgo versions will not work. I thought the update (-U) flag in !pip install -Uqq duckduckgo_search would pull the latest version but either it didn’t for some reason or a regression was introduced in a later version.
At any rate, explicitly asking for a version worked for me:

!pip install -Uqq duckduckgo_search==2.9.4

2 Likes

Looks like the fix has been committed. Release v3.0.1 · deedy5/duckduckgo_search · GitHub

2 Likes

I got {“code”:“DeploymentFailed”,“target”:“/subscriptions/03eef13f-3f60-4106-9c3c-ec622619d84c/resourceGroups/searchforfastaicourse/providers/Microsoft.Resources/deployments/Microsoft.BingSearch-20230518084223”,“message”:“At least one resource deployment operation failed. Please list deployment operations for details. Please see Deployment history - Azure Resource Manager | Microsoft Learn for usage details.”,“details”:[{“code”:“InsufficientQuota”,“message”:“Insufficient Quota”}]}
eve though I chose free tier

Yes! It has been fixed and everything works fine for me and hopefully for all you too. Thanks to all who participated and replied. Let’s go!!

I seem to get this error when doing !pip install -Uqq duckduckgo_search==2.9.4

Is there something I’m doing wrong?

what changes did you make to the code?

maybe remove one of the =
=2.9.4 instead of ==2.9.4 or even don’t specify version at all

Thanks a lot Iain - this is the only working solutions for me for now.

I just needed to change slightly the definition of the function:

def search_images(term, max_images=30):
    print(f"Searching for ‘{term}’")
     return search_images_ddg(term, max_images)

The function search_images_ddg expects max_n instead of max_images:
    def search_images_ddg(key,max_n=200)

2 Likes

Hey, I’ve just tried doing the same thing and I get the following error:

TypeError: search_images_ddg() got an unexpected keyword argument ‘max_n’

Although I see that in the definition of search_images_ddg I see it expects max_n. Very weird. Any ideas how to fix this?

OK it now seems to work with max_images instead of max_n for some reason, for anyone having the same issue.

@jsk123 , @rosh-eth and others who are running into this issue, I struggled with it as well and finally found that importing fastbook and using ‘search_images_ddg’ is what got everything to work in the end.

I’ve documented what I did in a kaggle notebook in the hope that it can help others get through this. I found this issue quite frustrating until I finally got it to work.

2 Likes

Thanks for your time putting this up. This is exactly what worked for me, and the only method that worked for me essentially. Cheers!

Thanks, this was the only solution that worked for me (or at least first one – I’ll stop digging now :laughing:)

What did not work:

  • Upgrading duckduckgo_search to >= 2.9.4, even 3.3.0 failed with HTTP 403 forbidden, although the same URL worked when used within the browser. Might be the User-Agent header that is used in fastai’s util function.
  • Fixing the deprecation warning and using DDGS(), same error in request library call.
  • Restarting the kernel because some caching apparently prevented the usage of the updated/changed library, so using a working library version might still fail (but no version of plain duckduckgo_search fixed my problem).

So thanks again, @maitland :slight_smile:

PS: I did not have to fiddle with the parameter names, max_images worked just fine.

PPS: This is equivalent to what @cmondorf did in his kaggle notebook linked above.

2 Likes

Yes @sebkraemer ! restarting the runtime works, I am using colab.

1 Like


Anyone know how to fix this?

If someone Still having Issues with duckduckgo on Kaggle (403 forbidden for me), they can use Google Colab for this. It worked nicely for me.

Hey there, I found a workaround for the duckduckgo search issue.

Here is my notebook that works:

Long story short: there seems to be a problem with version 3.8.5 of duckduckgo library, once I updated to the latest 3.9.5 it started working. Note, that this requires python version >=3.9
In kaggle there is an option in Notebook options I had to change: Always use latest environment (see screenshot)

Hope that helps!

6 Likes

Hi y’all, as of today, the Client error “403 forbidden” occurs for me no matter what: locally, Paperspace Gradient, Colab, Kaggle … Also the workarounds mentioned above and utilizing the alternative context manager implementation like

def search_images_ddg(term, max_images=200):
    with DDGS() as ddgs:
        results = ddgs.images(keywords=term)
        images = [next(results).get("image") for _ in range(max_images)]
        return L(images)

don’t work anymore or make no difference.

Anybody familiar with the API chiming in on what’s going on and/or a (new) fix would be highly appreciated. I pulled too many hairs trying to get it working on a Friday evening :slightly_smiling_face:

I found this GitHub issue which lists a solution to pass the dictionary {"Accept-Encoding": "gzip, deflate, br"} to the headers parameter of DDGS as follows:

def search_images(term, max_images=200):
    with DDGS(headers = {"Accept-Encoding": "gzip, deflate, br"}) as ddgs:
        results = ddgs.images(keywords=term)
        images = [next(results).get("image") for _ in range(max_images)]
        return L(images)

I tested this in a Kaggle notebook and it seems to work:

5 Likes

Can confirm, tested it locally and this solves the issue :+1:

2 Likes