@Philipp - I like the way you’re thinking about this. The good news is that there’s a terrific paper that talks about how to deal with just that issue: http://arxiv.org/abs/1412.6596 . The ‘soft loss’ they discuss is similar to your idea - but it’s done in a more dynamic way during training.
1 Like