Embedding with Categorically-Continuous variables?

Deep Learning
1

Before asking my question, let me clarify what I mean by “categorically-continuous”. Imagine you’re trying to determine whether a given soup recipe will taste good or bad. There are many possible ingredients you could use (>25), but you only use a few at a time. You can describe a recipe by the weight fractions of each of the ingredients (e.g. 40% chicken broth, 30% chicken, 2% salt, 28% noodles). Thus, the weight fractions themselves are continuous but each relates to the amount of a certain category. That is what I mean by “categorically-continuous.” This soup example is obviously silly, but there are many complicated practical examples that are analogous.

My question is can you use categorical embedding on the categories but also include the scalar values associated with them? I feel like other people must have tried this, but I don’t know the keywords to search for it.

I’ll return to the soup example to illustrate why I think this combined scalar-embedding would be useful. Each ingredient has different qualities that it brings to the soup. The more weight % of that ingredient, the more of its qualities that it brings to the soup. For example the broth has high fluidity and low calories, while the chicken is the opposite. Salt brings…saltiness. For a tasty soup their is probably some ideal range of saltiness, fluidity, etc. So by weighting the qualities of ingredients by their weight fractions you could better predict tastiness by the overall qualities. In this simple soup example, I can think of several important qualities, but for more complicated problems this isn’t always possible. Therefore I want the categorical embeddings to learn the important qualities, and the scalar values to weight them.

2

You could certainly do a little feature engineering, and make associated col’s with them. I don’t see why that couldn’t work, unless it’s dynamic. You could add some layer (or sth) that just applies that fractional value to the output of the embedding matrix perhaps?

4

For me the problem with creating explicit features is that in some of my applications I usually don’t know what the important underlying qualities are. Your last suggestion about applying the scalars to embedding output sounds like what I want. Could you sketch out how to do that with a Tabular_Learner? (I don’t have much experience with changing stuff under the hood).

5

I’m still interested in doing this. Could you nudge me in the right direction on how to start implementing a layer that applies the fractional values to the output of the embedding matrix?

6

Hi @sgp - did you figure this out by any chance?