Stuck debugging my simple model


I’m having a hard time understanding what’s going wrong with my model.

I’m trying to reproduce some matrix factorization with biases on a real estate dataset.

This is the code for my model:

    class MF(nn.Module):
    def __init__(self, config):
        self.user_embedding = nn.Embedding(config['user_pool'], config['latent_dim'])
        self.item_embedding = nn.Embedding(config['item_pool'], config['latent_dim'])
        self.user_biases = nn.Embedding(config['user_pool'], 1)
        self.item_biases = nn.Embedding(config['item_pool'], 1)

    def forward(self, users, items):
        users_emb, items_emb = self.user_embedding(users), self.item_embedding(items)
        biases_u, biases_i = self.user_biases(users).squeeze(), self.item_biases(items).squeeze()
        dots = (users_emb * items_emb).sum(1)
        res = dots + biases_u + biases_i
        return res 

I ran over 50 experiments and all the results I got were very bad (ie. awful recommendations), even though I had an almost perfect decrease in validation loss in some cases:

I will write my setup so that maybe one of you guys will give me a hint of what am I missing.

    config = {
       'user_pool': 96249,
       'item_pool': 25874,
       'latent_dim': 32,
       'model': 'MF',
       'lr': 1.65e-3,
       'wd': 1e-6,
       'bs': 256,
       'optim': 'Adam',
       'loss': 'RMSE'

The way I constructed my datasets is as follows:

  1. Because I had users that viewed an item multiple times and I wanted to take this into account, my target column was transformed using this method
data = data.groupby(['users', 'items']).target.count().reset_index()
data['itemId'] = data.items.apply(lambda x: item_to_int[x])
data['userId'] = data.users.apply(lambda x: user_to_int[x])
data['target'] = x: np.log2(x) + 1)
  1. Training and test split was done based on leave-one-out cross validation, ie. for each interaction I had in the training, I left one out for validation.

Now, this is just a summary, I explored some other alternatives too, but this is the latest one.
An insight will be welcomed.

I think that you need to improve your model. Check EmbeddingDotBias fastai model. It has some tricks like y_range and sigmoid at the output that helps to increase accuracy as said by Jeremy during the course.