Thanks @jianshen92, I think I am just too unimaginative to try something else 
Have been trying Radam + LookAhead (Ranger) alright, RangerLars looks like it should be better again from the discussion here: https://forums.fast.ai/t/imagenette-woof-leaderboards-guidelines-for-proving-new-high-scores/52714/57:
RangerLars: https://github.com/lessw2020/mish/blob/master/rangerlars.py
Have you seen this new Mish activation too? Going to swap it in the EfficientNet and see how it goes: Meet Mish: New Activation function, possible successor to ReLU?
Crazy time for new techniques!!