Getting weights in an AWD_LSTM model, *with* and *without* the doc

Part 1 2022
1

Hello,
I’m working on Kaggle and currently, I’m playing with models by trying to “unbox” them, in particular, to get the weights they store.

TL;DR

My questions are:

  1. How can I get all the weights and parameters of, for instance, the first trained model of Chapter 10 of the Fastbook
  2. How can I do this with the help of the doc or ??foo (as I will explain below, I’m quite at loss to find the attribute lists)

Basically, it’s less a post about finding the weights than a post about the method of reading and find one’s way about the doc, so now I include more precise examples

Long version + more specific questions

**You can find the questions by searching the string “question:” :slight_smile:

Context just after the first training of Chapter 10, i.e. just after the cell:

learn.fit_one_cycle(1, 2e-2)    # ~22 minutes to run of Kaggle

So now, I have my trained model and I want to unbox pretty much anything there is in it. So let’s start:

learn.model

gives me:

SequentialRNN(
  (0): AWD_LSTM(
    (encoder): Embedding(60008, 400, padding_idx=1)
    (encoder_dp): EmbeddingDropout(
      (emb): Embedding(60008, 400, padding_idx=1) )
    (rnns): ModuleList(
      (0): WeightDropout(
        (module): LSTM(400, 1152, batch_first=True) )
      (1): WeightDropout(
        (module): LSTM(1152, 1152, batch_first=True) )
      (2): WeightDropout(
        (module): LSTM(1152, 400, batch_first=True) ))
    (input_dp): RNNDropout()
    (hidden_dps): ModuleList(
      (0-2): 3 x RNNDropout() ))
  (1): LinearDecoder(
    (decoder): Linear(in_features=400, out_features=60008, bias=True)
    (output_dp): RNNDropout() ))

Then, I focus on the first element of the list:

mod0 = learn.model[0]
  1. First attribute:
mod0.encoder

I get Embedding(60008, 400, padding_idx=1) so I can write learn.model.encoder.weight.data to get the weights of the encoder. Thins work also fine with mod0.input_dp and mod0.hidden_dps (4th and 5th attributes)

  • Second attribute (fail): the cell above has suggested that learn.model has also encoder_dp and an attribute, but:
mod0.encoder_dp

gives AttributeError: 'AWD_LSTM' object has no attribute 'encoding_dp'

Question 1: why is this so? Whereas in the cell learn.model, encoder and encoder_dp appear in a similar fashion?

Question 2: if I understand correctly, AWD_LSTM is a fastai (and not a Pytorch) feature. So I looked at this page of the Fastai doc. There is no indication on the possible attribute that I can read from AWD_LSTM. Where shoud I find them? What am I missing?

  • 3rd attribute:
mod0.rnns

works fine and I get

ModuleList(
  (0): WeightDropout(
    (module): LSTM(400, 1152, batch_first=True) )
  (1): WeightDropout(
    (module): LSTM(1152, 1152, batch_first=True) )
  (2): WeightDropout(
    (module): LSTM(1152, 400, batch_first=True) ))

So, now, I’ll have a look at the element of rank 0 of this ModuleList:

mods0.rns[0]

I get:

ModuleList(
  (0): WeightDropout(
    (module): LSTM(400, 1152, batch_first=True))
  (1): WeightDropout(
    (module): LSTM(1152, 1152, batch_first=True))
  (2): WeightDropout(
    (module): LSTM(1152, 400, batch_first=True) ))

So, I write

lstm0 = mod0.rnns[0].module  ; lstm0

which displays LSTM(400, 1152, batch_first=True)

  • Question 3 : how do I get the weights in lstm0 ? Are they any?
  • Question 4 : how is this information indicated for instance in the Pytorch doc for LSTM ? Is this page the right place to get this info?

I tried the thing which seems rational to me, i.e. doing ??lstm0 and looking at the Pytorch doc page on LSTM which in the *Variables section, suggested that I could get the attribute weight_ih_l[k] .
Thus I wrote

lstm0.weight_ih_l[0]

which gives me the error: AttributeError: 'LSTM' object has no attribute 'weight_ih_l'

Question 5 What am I reading wrong in the pytorch doc? How should I get the weights of lstm0 ?