I’m looking to try and use dropout at prediction time to get a distribution of predictions.
I was inspired by this thread:
How I understand it is:
Run 100~ predictions with dropout applied to get 100~ different predictions, then by measuring the variance of the predictions you can get a measure of ‘uncertainty’ for each sample.
Some boilerplate code I’ve been experimenting with using MNIST:
class Net(nn.Module): def __init__(self): super().__init__() self.conv1 = nn.Conv2d(1, 10, kernel_size=5, padding=2) self.conv2 = nn.Conv2d(10, 20, kernel_size=5, padding=2) self.dropout = nn.Dropout2d(p=0.3) self.fc1 = nn.Linear(7*7*20, 1000) self.fc2 = nn.Linear(1000, 10) def forward(self, x): # dropout at every activation layer x = F.relu(F.max_pool2d(self.dropout(self.conv1(x)), 2)) x = F.relu(F.max_pool2d(self.dropout(self.conv2(x)), 2)) #print(x.shape) x = x.view(-1, 7*7*20) x = F.relu(self.fc1(self.dropout(x))) x = F.relu(self.fc2(self.dropout(x))) return F.log_softmax(x, dim=1) model = Net().to(device) ...[Train model]... # Create function to apply to model at eval() time def apply_dropout(m): if type(m) == nn.Dropout2d: m.train() # Predict the class of X using model def predict_class(model, X): model = model.eval() model.apply(apply_dropout) # apply dropout at pred time (see func above) outputs = model(Variable(X)) #print(outputs) _, pred = torch.max(outputs.data, 1) return pred.numpy() # Predict T times to get a distribution of predictions def predict(model, X, T=100): list_of_preds =  standard_pred=predict_class(model, X) y1 =  y2 =  for _ in range(T): _y1 = model(Variable(X)) _y2 = F.softmax(_y1, dim=1) y1.append(_y1.data.numpy()) y2.append(_y2.data.numpy()) list_of_preds.append(predict_class(model, X)) # predict T times return standard_pred, np.array(y1), np.array(y2), np.array(list_of_preds)
.predict() and passing it
model, 100 predictions are made with
What my empirical testing has shown is examples of data which are ‘bad’ (don’t look good compared to others) have a higher variance compared to ‘good’ examples.
Does anyone know how I would go about doing this with the fastai library?
Could I do it with a wrapper function on the
I’m currently investigating this now but if anyone has a clue, that’d be amazing.