Categorical variables in deep learning

Part 2 (2017)
1

Hi everyone!

Since we are going to work on structured data in Part 2 I thought this paper could be interesting: Categorical Reparameterization with Gumbel-Softmax.

I just saw this concept used in a deep learning online seminar and have not applied it myself yet. The idea is to replace samples from categorical variables on which we cannot apply backpropagation by something differentiable. From what I understand, this allows us to work with a differentiable version of our data in the network and recover the categorical values at the end.

2

We will be discussing categorical variables at length in a later lesson - the paper you mention above is a very interesting idea, which is also discussed in this paper: http://arxiv-sanity.com/1611.00712v2 . Note however that we can already use categorical variables as inputs (with embeddings) and outputs (with regular softmax).

1 Like