How different is it to prediction of bbox coordinates?
Thanks for your prompt response, I think for the prediction of bbox, it is a regression problem, and it make sense to look at average loss as individual coordinate have equal importance.
In particular, I was trying to do multi-label classification. Although the evaluation metrics is average AUC of 6 columns, there are need to monitor individual columns.
- it is a imbalance dataset, certain label has much more example/ less example (thus the task for making classification for each column has different difficulty. Only monitor the average loss may loss the chance to exploit the model that maybe good at classified a particular column.
- Consider an extreme case, if i have 2 models.
1 model has accuracy [ 99.9%,99.9%,99.9%,90%,90%,90%], the other has [ 90%,90%,90%,99.9%,99.9%,99.9%]. The obvious thing to do is ensemble 2 model, which I which never know if I only monitor the average.
Sure, but bbox numbers also have to be compared against each of the target box coordinates. If you use a list of predictions to compare to a Y list, simply checking for equality along the 1 axis while taking the mean should give you accuracy for each element in the list. Agree?
Actually, I was struggle to pass a batch to the model too. For the previous model, usually I can just do m(VV(xs)) to get a sample of output. However, for the language model classifier, I was getting self.hidden attribute does not exist when I do so. (this is another problem though, which makes debugging not easy as well.
I try to read through the source code, I think the reason why it does not work as it use a np.stack, which expect a list only. If i return a list with a custom metric, it become list of mix of elements and list, and thus doesn’t work. My instance just hang and I cannot get into it.
I had a similar problem when passing 2 lists as input to the model where the lists could have different lengths. np.stack will complain about the sizes.
so, i wrote a custom metric but didn’t run it as part of the eval at the end of each epoch. I was manually taking the preds and calculating accuracy separately.
Could you guide me how to save the output? I actually struggle to change the model structure, as the Language model class is not as straight forward, which have multiple funciton call and use a sequentialRNN.
I want to change the output to sigmoid layer (As required by binary cross entropy), but I cannot find where I should modify the code, as a result. I actually tweak the crit directly, I use a custom_loss function, to add a sigmoid layer in the end. It would be much easier
def custom_loss(input,target):
return F.binary_cross_entropy(F.sigmoid(input),target,weight=weight_normalized)
sure. pytorch has a binary cross entropy loss variant called BCEWithLogitsLoss() which does the sigmoid for you. you can see the pytorch documentation for this fn. For more details see how I implemented the pairdataset, the pairbatchrnn and linearcomparator classes in this notebook: https://github.com/arvind-cp/fastai/blob/arvind-cp-LM-eval/courses/dl2/FiTLAM%20-%20Quora%20Duplicate%20Pairs%20-%20STS%20task.ipynb
In this case you define the model your self. How could I add a layer on top of the current model? I use the default get_cnn_classifier model, which return a SequentialRNN.
I would like to do 2 things.
- add sigmoid layer at the end( I test the nn.functional.binary_cross_entropy_with_logits, which does not take a sigmoid for you, it is expecting a log(sigmoid) - log(1-sigmoid) instead)
- tweak the model output so that I can actually save the prediction and thus do whatever metric i want outside the fit function.
That’s what I have done. get_rnn_comparator returns a SequentialRNN which adds my custom model on top of the fitlam backbone. it’s in the notebook.
the sequential RNN returns an output which is a list of 3 things.
[0] the o/p of the classifier module
[1] the o/p of the backbone with dropouts applied
[2] the o/p of backbone minus the dropouts.
You can take [1] or [2] and pass it to your module if you wish.
Also, remember…if you want to write your custom logistic loss crit and you don’t have a squashing fn like sigmoid, you need to clamp(clip in numpy) the values or the log will blow up.
def binary_loss_crit(y, p):
p = p.clamp(min=0.001,max=0.999)
return torch.mean(-(y * torch.log(p) + (1-y)*torch.log(1-p)))
Could you point me to the source code where you find these? I was trying to see what the output is but I don’t know where I should read.
the sequential RNN returns an output which is a list of 3 things.
[0] the o/p of the classifier module
[1] the o/p of the backbone with dropouts applied
[2] the o/p of backbone minus the dropouts.
Sure, it’s in lm_rnn.py inside the fastai library.
see what’s returned by LinearDecoder or PoolingLinearClassifier.
In the notebook I showed you, I wrote a custom head but I chose to return only the first piece - the classifier output itself.
Hope this helps.
Ah, I see it now. Thanks, really appreciate your help. My last question is how do I pass a sample to the model for evaluation? the usual way seems not working.
You’re welcome!
Passing lists is a bit tricky. See my earlier comment about this in another thread:
The change is in model.py:
def step(self, xs, y, epoch):
xtra = []
output = self.m(*xs)
becomes:
def step(self, xs, y, epoch):
xtra = []
if len(xs)>1 :output = self.m([*xs])
else: output = self.m(*xs)
And the same change in the evaluate method as well.
I try all m(xs), m(*xs), m([*xs]), none of them work. In this case I think len(xs) =1, but I was getting self.hidden does not exist.
See my earlier screenshot.
MultiBatchRNN inherits hidden from RNN_Encoder where it is defined as self.hidden = repackage_var(new_hidden).
Is it missing in your copy of lm_rnn.py ?
I did have it, the weird thing is, I can run the training loop, lr_find, everything are fine, but I cannot pass a batch to the sample myself.
I will continue tomorrow, I just copy the whole lm_rnn to my notebook so it is easier to add pdb.set_trace() to debug. I run lr_find() and I have self.hidden and able to run m(*VV(xs)) when I was inside lr_find(), not sure why it is missing outside the function…
the self.hidden attribute was created by reset(), I notice that by printing a checkpoint inside the function, it prints when I do lr_find() but not when I am evaluating a batch.
Turns out Jeremy actually answer my question already… but I did not see it.
1 Like