When should we not use random forests?

Random forests seem to be quite robust algorithms in general, but when should we try and avoid using them?