While using y_range, not adding Sigmoid layer

krisho007 · March 6, 2019, 1:21pm

Hi, I am trying to create a TabularModel. I am trying to guess a probability, so y_range=(0,1). I have couple of problems.

Whenever I have y_range, learning rate plot is not looking right. See below.
Whenever I add y_range, learn.model should automatically have a Sigmoid layer at the end. But this is not happening.

marcmuc · March 6, 2019, 2:33pm

re 2.:
The sigmoid is not added as an nn.layer but rather applied as F.sigmoid() in the forward method of the model, so it does happen, but you will not see it in the model layer list. See here (line 155 below):

github.com

fastai/fastai/blob/7cf2f19193a313f0d2a03d8448489d140667c8dc/old/fastai/column_data.py#L146


def forward(self, x_cat, x_cont):
    if self.n_emb != 0:
        x = [e(x_cat[:,i]) for i,e in enumerate(self.embs)]
        x = torch.cat(x, 1)
        x = self.emb_drop(x)
    if self.n_cont != 0:
        x2 = self.bn(x_cont)
        x = torch.cat([x, x2], 1) if self.n_emb != 0 else x2
    for l,d,b in zip(self.lins, self.drops, self.bns):
        x = F.relu(l(x))
        if self.use_bn: x = b(x)
        x = d(x)
    x = self.outp(x)
    if not self.is_reg:
        if self.is_multi:
            x = F.sigmoid(x)
        else:
            x = F.log_softmax(x)
    elif self.y_range:
        x = F.sigmoid(x)
        x = x*(self.y_range[1] - self.y_range[0])

krisho007 · March 6, 2019, 2:58pm

Thanks @marcmuc. Code would have been more readable if they had added it to the Sequential model.

Anyone can help me please with Question 1?

miko · March 6, 2019, 3:10pm

add skip_end=0 to your learn.recorder.plot()

krisho007 · March 8, 2019, 10:32am

It did not help.

miko · March 8, 2019, 10:35am

have a look at how lr_find works and try experimenting with different parameters