A few thoughts on the validate function

I try to evaluate a trained model in v1- which I assume should be done by calling the validate(...) function (possibly for a different set of metrics/a different dataloader compared to the training process). I was only able to make it work with a call of the form validate(learn.model, learn.data.valid_dl,metrics=[accuracy],loss_fn=learn.loss_fn,cb_handler=CallbackHandler(learn.callbacks))

  • It took me a while to figure out that validate(...) should not be called with loss_fn=None as this would lead loss_batch to return the data in a different format. Is this the intended behavior?
  • the metrics argument has to be passed as list- unlike fit(...) which seems to provide a workaround if one provides just a single argument.
  • calling validate without a cb_handler did not work for me (as the shape of the input tensor seems to modified by one of the callbacks- although I didn’t figure out where exactly)
  • given these issues- is there a reason why there is no validate member function in the learner class very much like the get_preds member function (that would in the ideal case also allow to pass a custom dataloader and a custom list of metrics to evaluate)?

I think the new library is great step forward. Thank you for your hard work so far.

1 Like

Please note that the validate function is more for internal use than external, which is why it’s not the easiest to use. We will implement something more practical shortly.

that’s perfectly fine. My suggestion would just be to add a member function like
def validate(self, dl:DataLoader=None, loss_fn:OptLossFunc=None, metrics:OptMetrics=None, average:bool=True)->Iterator[Tuple[Union[Tensor,int],...]]: return validate(self.model, dl=(dl if dl is not None else self.data.valid_dl), loss_fn=(loss_fn if loss_fn is not None else self.loss_fn), metrics=listify(metrics) if metrics is not None else self.metrics, cb_handler=CallbackHandler(self.callbacks), average=average)
which would make it already much more convenient for the time being.

Check out Learner.validate :wink:

1 Like