Tabular Transfer Learning and/or retraining with fastai

Deep Learning
7

I’m facing a similar problem with transfer learning of embeddings. I’ve taken the approach of copying the tensor values from an embedding to a CSV file and reloading them into a new embedding which may have some different categories. I’m still having a problem freezing and unfreezing them, but otherwise it seems to work. Here’s what I have so far (I would appreciate ANY critique on the approach or the code itself.)

 import csv
 def write_encoding_dict(filename,df,cat,input_embeds):
     embeds=input_embeds.cpu()
     source_vocab= df[cat].astype('category').cat.categories.values
     with open(filename, 'w') as csvFile:
         writer = csv.writer(csvFile, lineterminator='\n')
         for i in range(len(source_vocab)):
             myvals = np.array(embeds(torch.tensor(i))).tolist()
             writer.writerow([source_vocab[i],*myvals])
         csvFile.close()

In my model, I want to save the first embedding variable, and I do it like this:

write_encoding_dict(‘embedding0.csv’,panda_dataframe,category_var0, learn.model.embeds[0])

Then the file contains rows of “class,embeddings value list” like this:

ACE,-0.00013918841432314366, 3.610396379372105e-05, -7.69308189774165e-06, -2.2517966499435715e-05, -2.284333822899498e-05

Then to read them back in and load the embedding values into a different model:

 def get_encoding_dict(filename):
     with open(filename, 'r') as csvFile:
         reader = csv.reader(csvFile)
         lines = list(reader)
         d = OrderedDict()
         for i in range(len(lines)):
             d[lines[i][0]] = [float(lines[i][j]) for j in range(1,len(lines[i]))]
         csvFile.close()
         return d
 
 def load_embed_weights(df, cat, embeds, file):
     encodings = get_encoding_dict(file)
     target_vocab = df[cat].astype('category').cat.categories.values
     weights_matrix = embeds.weight
     #weights_matrix.requires_grad = False
     emb_dim=weights_matrix.shape[1]
     words_found = 0
     for i, word in enumerate(target_vocab):
         try: 
             enc = encodings[word]
             for j in range(emb_dim):
                 weights_matrix[i][j] = enc[j]
             words_found += 1
         except KeyError:
             for j in range(emb_dim):
                 weights_matrix[i][j] = np.random.normal(scale=0.6)
     print(weights_matrix.shape[0], words_found)

So - seems to work. The problem I’m having is when I try to freeze the weights in the new model, like this:

weights_matrix.requires_grad = False

I get an error that I can’t freeze a non-leaf node. So when I try to freeze the embedded tensor directly, like this:

weights_matrix.data.requires_grad = False

I get a different error that the optimizer can’t optimize a non-leaf variable.

I feel like I’ve made real progress, but this last hurdle is killing me…