Recommender system benchmarks

(Ben Johnson) #1

Hi All –

Jeremy works with the MovieLens dataset in Lecture 5 – I’ve been trying to find somewhere that benchmarks the performance of various recommender system algorithms. There are a number of open source tools for this:

but I haven’t been able to find anywhere (papers, blogs, etc) that compare performance of different methods on some standard benchmark dataset?

~ Ben

(Even Oldridge) #2

The most common one I’ve seen is from criteo, but there’s not really a consensus amongst the community. Some research reports on MovieLens20M but a lot of companies report only on their own data.

Recsys, the main conference in recommender systems has a competition every year that’s also worth checking out. This year is a spotify playlist recommendation problem but unfortunately it was only open to academics and finishes at the end of June.

Their previous datasets for their challenges have been good and are often included in Recsys papers, but there isn’t a gold standard for evaluation so different papers tend to use different datasets.

I’m working in the field right now and will be at Recsys. If you’re interested in the intersection between deep learning and recommender systems hit me up and I’m happy to talk shop. Let me know if you come to a different conclusion about this because I’m working on a paper right now and I’d like to do a few public datasets.