Benchmark scores for all image models in Keras and PyTorch

[marvin (Marvin Hansen)]

Curtis Northcutt from MIT took the time time to benchmark all image models available in Keras and PyTorch and created a ranked list.

Now you can benchmark and compare your best image model against all verified SOTA scores of all standard image models. Nice.

Find the benchmark, data, and results here:

cgnorthcutt/benchmarking-keras-pytorch

[kushaj (Kushajveer Singh)]

Something that was really needed.

[LessW2020 (Less)]

Thanks for posting this!

Nice to see that ResNet152 is the leader for PyTorch. I’ve certainly had strong results from it.

I’m surprised the #1 rank is 3% higher though, that’s a fairly large difference. Have not heard of nasnetlarge, but wonder if a wide resnet would beat it (no wides were shown in this list).

*Found the background for nasnet - https://arxiv.org/abs/1707.07012

[jeremy (Jeremy Howard)]

I’m surprised the #1 rank is 3% higher though, that’s a fairly large difference. Have not heard of nasnetlarge, but wonder if a wide resnet would beat it (no wides were shown in this list).

I’ve found it too memory intensive and slow to be useful. Been a while since I tried though.

[LessW2020 (Less)]

Looking at the NAS cell, that would definitely make sense regarding slow and memory intensive. Each normal cell is running 8 convolutions inside…it’s like a mini CNN in each cell:

(image from their paper)

Thanks for the feedback on it!

[amitkayal]

this is excellent one