Question on labeling text for sentiment analysis

I pored over the nlp.py and lm_rnn.py scripts but could’nt figure out if there is support for multi-task yet. I’ll wait for @jeremy too :slight_smile:

1 Like

Simply treating it as a multi-label classification problem should solve this very nicely! :slight_smile: I don’t think you need additional fastai stuff for this…

1 Like

Ah well that’s a good point. The suggestion in my previous post would involve just treating it all as sigmoid, which would be slight sub-optimal since you’re not taking advantage of the softmax. I still suspect it would work OK - worth trying!

Otherwise, you’d have to create your own classifer layer. It wouldn’t be hard, but would require some code-reading to see how it’s done in fastai. (I’m happy to help, if anyone decides to try this).

Point me in the right direction, and it shall be done.

I’m imagining it looks something like this (the layers on top of the pre-trained LSTM) and that it works similar to the part1v1 lesson 7 model that predicts both fish and bounding box coordinates:

That’s exactly right. Here’s the code that creates the model:

You could just replace the LinearDecoder call with a call to your own little classifier class. Then set learn.models.model to your new language model object.

2 Likes

Your code snippet has def get_rnn_classifier highlighted, and so I wanted to make sure that your recommendation was indeed to swap out the call to LinearDecoder, and not PoolingLinearClassifier, with a call to my custom classifier (that will attempt to classify the three things noted above simultaneously.

Is that correct?

If so, the process would be for me to:

1.Create an instance of a LanguageModelData object
md = LanguageModelData(...)

2.Define my own pytorch model using my custom classifier that will output three values:

rnn_enc = RNN_Encoder(bs, n_tok, emb_sz, nhid, nlayers, pad_token, dropouth=dropouth, dropouti=dropouti, dropoute=dropoute, wdrop=wdrop)
enc = rnn_enc.encoder if tie_weights else None
model = SequentialRNN(rnn_enc, VerbatimClassifier(n_tok, emb_sz, dropout, tie_encoder=enc))

3.Then set the learner to use my pytorch model
learner.models.model = model

4.Then train in usual way


In the notebooks the approach has been to train a language model and then use it in another model for sentiment analysis. But, if I’m understanding you correctly, we are bypassing the second step and doing both tasks at the same time.

1 Like

Ah yes I guess I should have said PoolingLinearClassifier

Ok.

So then the process is the same … 1) train a language model and then 2) use that pre-trained model for the sentiment task.

So step 2 would look something like this after I have the language model (encoder saved as “adam3_20_enc”):

md2 = TextData.from_splits(PATH, splits, bs)

rnn_enc = MultiBatchRNN(max_sl, bptt, bs, n_tok, emb_sz, n_hid, n_layers, pad_token=pad_token, dropouth=dropouth, dropouti=dropouti, dropoute=dropoute, wdrop=wdrop)
custom_model =  SequentialRNN(rnn_enc, VerbatimClassifier(n_class, 3*emb_sz, dropout))

learner = RNN_Learner(md2, TextModel(to_gpu(custom_model)), opt_fn=opt_fn)
learner.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)
learner.clip=25.
learner.load_encoder(f'adam3_20_enc')

… and then train as normal?

Give it a try! :smiley:

I’m trying to get the IMDB code working, but it’s throwing an error that I think might be related to this thread.

I’ve tried a few values, but I can’t get the model to actually fit, any guidance on what the new attributes should be?

m3 = md2.get_model(opt_fn, 1500, bptt, emb_sz=em_sz, n_hid=nh, n_layers=nl,
dropouti=0.4, wdrop=0.5, dropoute=0.05, dropouth=0.3)


TypeError Traceback (most recent call last)
in ()
1 m3 = md2.get_model(opt_fn, 1500, bptt, emb_sz=em_sz, n_hid=nh, n_layers=nl,
----> 2 dropouti=0.4, wdrop=0.5, dropoute=0.05, dropouth=0.3)

~/fastai/courses/dl1/fastai/nlp.py in get_model(self, opt_fn, max_sl, bptt, emb_sz, n_hid, n_layers, **kwargs)
391 def get_model(self, opt_fn, max_sl, bptt, emb_sz, n_hid, n_layers, **kwargs):
392 m = get_rnn_classifer(max_sl, bptt, self.bs, self.c, self.nt,
–> 393 emb_sz=emb_sz, n_hid=n_hid, n_layers=n_layers, pad_token=self.pad_idx, **kwargs)
394 return self.to_model(m, opt_fn)
395

TypeError: get_rnn_classifer() missing 2 required positional arguments: ‘layers’ and ‘drops’

Apologies - try git pull, should be fixed now.

Legend, thanks :slight_smile:

@wgpubs If you have implemented this can you share me your notebook. I m facing some similar problems. My predictions are giving accuracy of ~18% I might be going wrong in some step.

I haven’t had the time to do this (yet).

If you want to share what you got as a gist I can take a look and see if I see anything odd in it.

Jeremy,
I think that the fix of the invocation to get_rnn_classifier(), in line 359 of nlp.py
has an extra parameter (the first argument, max_sl) which creates a mismatch with the
def in lm_rnn.py,

As I understand MultiBatchRNN divides the text sequence into sizes of bptt and concatenates to generate final output/raw_output. Since the output of last layer current output is considered for the sentiment/label classification, does it mean that the classification will depend on the last (bptt) chunk of text?

Hi wgpubs, I am currently working on the toxic comment classification challenge and your discussions were really helpful. I assume that the classifier “VerbatimClassifier” is a custom classification function of yours? Would you mind sharing how to create a classifier? I am presently only on lecture 5 of the deep learning course and how to add additional layers to a NN is still beyond me.

I haven’t gotten to this yet, but yes, it would be a custom classifier. I think you’ll find helpful anything from lesson 5-7 in particular where Jeremy digs into the pytorch code and then try to simply subclass nn.Module yourself and create a forward method with multiple outputs.

If you try something and get it working, or kinda working, post a gist and I’ll take a look.

1 Like

I’ve also been following @wgpubs whilst attempting the toxic comments challenge and have been enormously helped by their comments. I’m only a little ahead of you in the course and am no expert, but eventually managed to put a multiclass classifier together. Unfortunately I can’t get it to train! So much to learn still…

As I understand it, the only need for a custom classifier is to handle the various softmax / sigmoid layers for the different outputs in the original question. For toxic comments we just need softmax, so I only modified the original PoolingLinearClassifier slightly by adding softmax. Maybe I’m wrong, hopefully somebody will correct me if so.

I’ve put my custom classifier here in case it helps. Custom PLC Gist

Happy to share my whole notebook if you want as there’s a lot of custom code, but since it’s not training I don’t want to send someone else down the wrong path too…

1 Like

Thanks I looked at your gist and it is very interesting. However I don’t understand the pooling aspect. What do we need the pooling for? I probably haven’t gotten far enough in the course yet (I am now in the middle of lecture 5) so if you point me to the right lecture I will gladly take a look. Also I don’t understand the

for l in self.layers:
    l_x = l(x)
    x = F.relu(l_x)

Why are we applying those relus if we want a sigmoid at the end?

2 Likes