Meet Mish: New Activation function, possible successor to ReLU?

[LessW2020 (Less)]

@Diganta

So there was a paper by lastminutepapers where they showcased combination of activations in a single NN. The recent project I’m working on called XTAN is basically that. It’s a gridsearch for getting the best activation for each layer for any model defined. Link to the paper - https://arxiv.org/pdf/1801.09403.pdf

That’s exactly the paper I was thinking of

Here’s two more related to that concept of HAN (heterogeneous activation networks). I think this is a space that might yield new innovation and results, along with leveraging scale inside the networks (ala res2net).

Evolving Parsimonious Networks by Mixing Activation Functions

Ensemble of Convolutional Neural Networks Trained with Different Activation...