Mercury Challenge - Deep Learning for Forecasting

(Leslie N. Smith) #1


I just saw a notification of an IARPA challenge on applying DL for forecasting global events. I know has entered and won other challenges (i.e., DAWNBench, Kaggle, etc.) so perhaps a team might enter this one too. If so, I’d like to participate on the team.


Here is some information on the Mercury Challenge:


Who We Are: The Intelligence Advanced Research Projects Activity (IARPA), within the Office of the Director of National Intelligence (ODNI), focuses on high-risk, high-payoff research programs to tackle difficult challenges of the agencies and disciplines in the intelligence community. IARPA’s challenges invite experts from the broader research community to participate in IARPA research in a convenient, efficient, and non-contractual way.

What We’re Doing: IARPA is looking for innovative machine learning techniques that would improve the accuracy and timeliness of global forecasts. IARPA hosts these challenges to identify ways that individuals, academia, and others with a passion for forecasting can showcase their skills easily.

Why We’re Doing This: Surprise events such as the fall of the Berlin Wall, Iraq’s invasion of Kuwait, the civil unrest that gave rise to the Arab Spring, and Russian incursions into Ukraine, forced rapid responses in the absence of data related to the underlying causes of these events.
In an effort to strengthen global crisis response capabilities, The IARPA Mercury Challenge seeks innovative solutions using machine learning and artificial intelligence methods to automatically predict the occurrence of critical events involving military action, non-violent civil unrest, and infectious diseases in Egypt, Saudi Arabia, Iraq, Syria, Qatar, Lebanon, Jordan, and Bahrain.

Who Should Participate: Technologists, data scientists, and machine learning engineers who are skilled at breaking down complex data are encouraged to join. Individuals ranging from private industry and academia are all eligible to participate and win prizes. The Mercury Challenge Team believes success in this challenge can prove to be a strong addition to any data science practitioner’s portfolio.
Certain individuals and groups with existing agreements with IARPA may not be eligible for cash prizes but may be able to compete for standing on the leaderboard and other non-monetary incentives. Additional eligibility rules will be available closer to launch.

Why Should You Participate: This challenge gives you a chance to join a community of leading experts to advance your research, contribute to global security and humanitarian activities, and compete for cash prizes. This is your chance to test your forecasting skills and prove yourself against the state-of-the-art, and to demonstrate your superiority over political pundits. By participating, you may:
• Network with collaborators and experts to advance your research
• Gain recognition for your work and your methods
• Test your methods and monitor how you stack up amongst competitors
• Win prizes from a total prize purse of $100,000
Throughout the challenge, an online leaderboard will display solvers’ rankings and accomplishments, giving you opportunities to have your work viewed and appreciated by leaders from industry, government and academia.

(RobG) #2

This looks interesting. The task reminds me of the book Superforecasting, enough to skim read it again to get some pointers as to how to perhaps structure an approach.

(Francisco Ingham) #3

Hey, I’d be interested in participating!

(Even Oldridge) #4

Even after looking at the website I’m a little unclear on what the data aspect of this challenge entails. Is the idea to predict catastrophes based on other catastrophe data? Or are you supposed to be building your own data sources and ingest?

(RobG) #5

I haven’t looked in great detail, but I imagine a core part of the challenge is identifying which datasets hold the best signal vs processing effort. Eg, one could imagine using sources from channels like social media (twitter), general news (bbc), survey companies (ipsos), ‘chroniclers’ (Wikipedia), economic (stock market, treasury reports), gov press releases etc, weather (and weather alerts), time based info (eg election dates) etc.

The iarpa forecast challenges have been running a few years and were covered in the book Superforecasting. IIRC the gist is that people overestimate short term likelihood and under estimate long term likelihood. You can see example forecast questions here:

(Leslie N. Smith) #6

I expect IARPA will specify the data sources. From their website:
“In an effort to provide early warning capabilities, the Department of Defense’s Integrated Crisis Early Warning System (ICEWS) and IARPA’s Open Source Indicators (OSI) programs want to leverage novel statistical and machine learning techniques using publicly available data sources to forecast societal such as civil unrest and disease outbreaks with a high degree of accuracy.
Participants are encouraged to develop and test innovative forecasting methods that ingest and process publicly available data sources to predict Military Activity, Non-violent Civil Unrest, and Infectious Disease in specific places of interest.”

So they state “publicly available data sources”, @digitalspecialists seems on the right track when he says:

I am wondering if we have enough interest to register a fastai team.

(Leslie N. Smith) #7

I just saw that many questions about the specifics can be answered by looking at their site on github:

(Kevin Bird) #8

I would be interested in working on this. I am also working through the ML lessons so I can’t promise 100% of my focus, but I will work on it on this end as well. Really cool concept!

(Leslie N. Smith) #9


My sincerest apologies. I just saw the following under eligibility:
“May not be a federal entity or federal employee acting within the scope of their employment. An individual or entity shall not be deemed ineligible because the individual or entity used federal facilities or consulted with federal employees during a competition if the facilities and employees are made available to all individuals and entities participating in the competition on an equitable basis.”

I am a federal employee and it is ambiguous if I can consult with the team outside the scope of my employment. How unfortunate as I see this as a fascinating and challenging problem. :slightly_frowning_face:


(Nick) #10

I’ve run a program in this area in Australia which used previous IARPA programs as inspiration. is a decent overview of what we did.