A few thoughts on the validate function

fastai fastai dev
1

I try to evaluate a trained model in v1- which I assume should be done by calling the validate(...) function (possibly for a different set of metrics/a different dataloader compared to the training process). I was only able to make it work with a call of the form validate(learn.model, learn.data.valid_dl,metrics=[accuracy],loss_fn=learn.loss_fn,cb_handler=CallbackHandler(learn.callbacks))

  • It took me a while to figure out that validate(...) should not be called with loss_fn=None as this would lead loss_batch to return the data in a different format. Is this the intended behavior?
  • the metrics argument has to be passed as list- unlike fit(...) which seems to provide a workaround if one provides just a single argument.
  • calling validate without a cb_handler did not work for me (as the shape of the input tensor seems to modified by one of the callbacks- although I didn’t figure out where exactly)
  • given these issues- is there a reason why there is no validate member function in the learner class very much like the get_preds member function (that would in the ideal case also allow to pass a custom dataloader and a custom list of metrics to evaluate)?

I think the new library is great step forward. Thank you for your hard work so far.

1 Like
2

Please note that the validate function is more for internal use than external, which is why it’s not the easiest to use. We will implement something more practical shortly.

3

that’s perfectly fine. My suggestion would just be to add a member function like
def validate(self, dl:DataLoader=None, loss_fn:OptLossFunc=None, metrics:OptMetrics=None, average:bool=True)->Iterator[Tuple[Union[Tensor,int],...]]: return validate(self.model, dl=(dl if dl is not None else self.data.valid_dl), loss_fn=(loss_fn if loss_fn is not None else self.loss_fn), metrics=listify(metrics) if metrics is not None else self.metrics, cb_handler=CallbackHandler(self.callbacks), average=average)
which would make it already much more convenient for the time being.

4

Check out Learner.validate :wink:

1 Like