Word Importance(most contributing words) from sentiment analysis

Hi, I have been wondering if it was possible to do something similar to feature importance in sentiment analysis, something like which word is contributing more to the sentiment of the sentence. I don’t want to use a separate dictionary with positive, negative words but instead back track on the sentiment analysis values to find the contributing words.

I was thinking of replacing each word with random embedding and check its contribution, but sometimes combination of 2 words give different meaning so I wanted some suggestion on this problem in a more automated way than brute force method.