Not at all! 
Just because a latent factor could end up in column 2 vs column 3 of the matrix, doesn’t make its interpretation any different…
More importantly, the main way to interpret models is not to look at their weights, but to “ask them questions”. See Feeling unfulfilled pushing 98% accuracy on State Farm