Yah exactly.
Check out Lesson 7 of the original part 1 if you have the chance, because by “multi-task”, it sounds a lot like using a single model to predict both a) type of fish and b) where that fish is located in the image.
Here the model would try to simultaneously predict a) how positive the comment was, b) was there a threat in there, and c) does it include a suggestion. Each would require a different final non-linearity function … sigmoid, softmax, and softmax respectively.
will do, thx! that does sound like separate classification tasks jointly solved, i’ll look at it again
I pored over the nlp.py and lm_rnn.py scripts but could’nt figure out if there is support for multi-task yet. I’ll wait for @jeremy too 
Simply treating it as a multi-label classification problem should solve this very nicely! 
I don’t think you need additional fastai stuff for this…
Ah well that’s a good point. The suggestion in my previous post would involve just treating it all as sigmoid, which would be slight sub-optimal since you’re not taking advantage of the softmax. I still suspect it would work OK - worth trying!
Otherwise, you’d have to create your own classifer layer. It wouldn’t be hard, but would require some code-reading to see how it’s done in fastai. (I’m happy to help, if anyone decides to try this).
Point me in the right direction, and it shall be done.
I’m imagining it looks something like this (the layers on top of the pre-trained LSTM) and that it works similar to the part1v1 lesson 7 model that predicts both fish and bounding box coordinates:
That’s exactly right. Here’s the code that creates the model:
You could just replace the LinearDecoder call with a call to your own little classifier class. Then set learn.models.model to your new language model object.
Your code snippet has def get_rnn_classifier highlighted, and so I wanted to make sure that your recommendation was indeed to swap out the call to LinearDecoder, and not PoolingLinearClassifier, with a call to my custom classifier (that will attempt to classify the three things noted above simultaneously.
Is that correct?
If so, the process would be for me to:
1.Create an instance of a LanguageModelData object
md = LanguageModelData(...)
2.Define my own pytorch model using my custom classifier that will output three values:
rnn_enc = RNN_Encoder(bs, n_tok, emb_sz, nhid, nlayers, pad_token, dropouth=dropouth, dropouti=dropouti, dropoute=dropoute, wdrop=wdrop)
enc = rnn_enc.encoder if tie_weights else None
model = SequentialRNN(rnn_enc, VerbatimClassifier(n_tok, emb_sz, dropout, tie_encoder=enc))
3.Then set the learner to use my pytorch model
learner.models.model = model
4.Then train in usual way
In the notebooks the approach has been to train a language model and then use it in another model for sentiment analysis. But, if I’m understanding you correctly, we are bypassing the second step and doing both tasks at the same time.
Ah yes I guess I should have said PoolingLinearClassifier…
Ok.
So then the process is the same … 1) train a language model and then 2) use that pre-trained model for the sentiment task.
So step 2 would look something like this after I have the language model (encoder saved as “adam3_20_enc”):
md2 = TextData.from_splits(PATH, splits, bs)
rnn_enc = MultiBatchRNN(max_sl, bptt, bs, n_tok, emb_sz, n_hid, n_layers, pad_token=pad_token, dropouth=dropouth, dropouti=dropouti, dropoute=dropoute, wdrop=wdrop)
custom_model = SequentialRNN(rnn_enc, VerbatimClassifier(n_class, 3*emb_sz, dropout))
learner = RNN_Learner(md2, TextModel(to_gpu(custom_model)), opt_fn=opt_fn)
learner.reg_fn = partial(seq2seq_reg, alpha=2, beta=1)
learner.clip=25.
learner.load_encoder(f'adam3_20_enc')
… and then train as normal?
Give it a try! 
I’m trying to get the IMDB code working, but it’s throwing an error that I think might be related to this thread.
I’ve tried a few values, but I can’t get the model to actually fit, any guidance on what the new attributes should be?
m3 = md2.get_model(opt_fn, 1500, bptt, emb_sz=em_sz, n_hid=nh, n_layers=nl,
dropouti=0.4, wdrop=0.5, dropoute=0.05, dropouth=0.3)
TypeError Traceback (most recent call last)
in ()
1 m3 = md2.get_model(opt_fn, 1500, bptt, emb_sz=em_sz, n_hid=nh, n_layers=nl,
----> 2 dropouti=0.4, wdrop=0.5, dropoute=0.05, dropouth=0.3)
~/fastai/courses/dl1/fastai/nlp.py in get_model(self, opt_fn, max_sl, bptt, emb_sz, n_hid, n_layers, **kwargs)
391 def get_model(self, opt_fn, max_sl, bptt, emb_sz, n_hid, n_layers, **kwargs):
392 m = get_rnn_classifer(max_sl, bptt, self.bs, self.c, self.nt,
–> 393 emb_sz=emb_sz, n_hid=n_hid, n_layers=n_layers, pad_token=self.pad_idx, **kwargs)
394 return self.to_model(m, opt_fn)
395
TypeError: get_rnn_classifer() missing 2 required positional arguments: ‘layers’ and ‘drops’
Apologies - try git pull, should be fixed now.
Legend, thanks
@wgpubs If you have implemented this can you share me your notebook. I m facing some similar problems. My predictions are giving accuracy of ~18% I might be going wrong in some step.
I haven’t had the time to do this (yet).
If you want to share what you got as a gist I can take a look and see if I see anything odd in it.
Jeremy,
I think that the fix of the invocation to get_rnn_classifier(), in line 359 of nlp.py
has an extra parameter (the first argument, max_sl) which creates a mismatch with the
def in lm_rnn.py,
As I understand MultiBatchRNN divides the text sequence into sizes of bptt and concatenates to generate final output/raw_output. Since the output of last layer current output is considered for the sentiment/label classification, does it mean that the classification will depend on the last (bptt) chunk of text?
Hi wgpubs, I am currently working on the toxic comment classification challenge and your discussions were really helpful. I assume that the classifier “VerbatimClassifier” is a custom classification function of yours? Would you mind sharing how to create a classifier? I am presently only on lecture 5 of the deep learning course and how to add additional layers to a NN is still beyond me.
I haven’t gotten to this yet, but yes, it would be a custom classifier. I think you’ll find helpful anything from lesson 5-7 in particular where Jeremy digs into the pytorch code and then try to simply subclass nn.Module yourself and create a forward method with multiple outputs.
If you try something and get it working, or kinda working, post a gist and I’ll take a look.