I understand that transforms should not be applied during inference.
The issue I’m facing is that it does not even seem possible to add a Test DataSet to a Learner that has been restored through a call to load_learner.
Using load_learner with a test ItemList as input fails immediately. Using load_learner without providing a test ItemList works, but later attempts to add a Test DataSet to the Learner’s DataBunch also fail.
Looking at the values of the restored Learner and DataBunch I find that bunch.dl_tfms = [ ].
I would appreciate the following clarifications:
Where in the restored Learner are the transforms that are causing the exception? Is it bunch.dl_tfms or other?
Does fastai support (without modifying source code) adding a Test DataSet to a Learner restored from load_learner? If yes, is there documentation somewhere explaining this?
If the answer to 2. is Yes – Does fastai support (without modifying source code) doing batch inference for semantic segmentation?
Performing inference on single images through learn.predict(img) does work after calling load_learner (without providing a test ItemList). How does this workflow avoid the problems of doing batch processing? Can this be replicated to the batch case?
The transforms are applied at the dataset level, which is why you can’t find them in dl_tfms. They are inherited from what you set during training, and you can modify them with data.test_ds (once you set it), attributes are going to be things like tfms, tfms_y, tfm_y…
Yes, as long as you pre-process your images to be all of the same size, otherwise they can’t be collated in a batch without being transformed, which will make your predictions wrong (and you will have to apply a post-process to your predictions afterward)
This workflow avoid the problem of resizing because you only have one image and don’t need to collate it in a batch.
@sgugger thanks for all your help in this thread. I had most of the same issues as @turntwo463. One clarifying question about his final question:
Can this be replicated to the batch case?
Is there a way to apply the transforms that are done when calling learner.predict on an inference batch (potentially using learner.pred_batch)? Thanks in advance for any guidance you can provide
That’s done automatically, the problem here is that you can’t apply those transforms to your labels. You can pass tfm_y=False when adding your test data to avoid having those transforms (like resize) applied there but your predictions won’t match your original images (if you resize for instance, they will have the resized size).
I’m not using any labels and approaching the problem from the perspective of using load_learner but having no access to the code/data used during training. I tried to mimic the behavior of .predict on .pred_batch instead but wasn’t able to match the one_item data method behavior:
from PIL import Image
from fastai.vision import open_image, pil2tensor, load_learner
import torch
import numpy as np
learner = load_learner(...)
# .predict behavior
img = open_image(...)
batch = learner.data.one_item(img) # first line of .predict to transform data correctly
# .pred_batch behavior
img2 = Image.open(...)
tensor = pil2tensor(img2, dtype=np.float32).div_(255) # mimic open_image behavior
tfm_tensor, _ = learner.data.valid_dl.tfms[0]((tensor, torch.zeros(0))) # apply validation transforms
assert(np.array_equal(batch[0], tfm_tensor))
# AssertionError
The final tensors are very nearly equal with the exception that they are slightly different sizes. In this specific case the training data is resized on input (src.transform(..., size=src_size)) and that transform isn’t applied.
You’re passing dataloader transforms and not dataset transforms. Resize is a dataset transform (needs to be applied before you collate your samples in a batch).
You can always remove the transforms applied in the add_test method by passing tfms=None. Note that you will need to have all your images of the same size otherwise you won’t be able to collate them in batches.
According to the documentation, at https://docs.fast.ai/data_block.html#LabelLists.add_test, add_test function should be able to accept tfms and tfm_y arguments. However, i was prompted the error below. Is this the correct behaviour for the function?
data = (SegmentationItemList
.from_folder(images, presort=True)
.split_by_rand_pct(seed=seed)
.label_from_func(get_y_fn, classes=codes)
.add_test(tests, tfms=None))
Thanks for sharing your findings. Unfortunately I have a similar problem for inference:
After loading the learner using load_learner without specifying test set: learn = load_learner(test_dir, ‘learner.pkl’).load(model), I intend to make a prediction using one of the images from validation set:
learn.predict(data.valid_ds[0][0]). But I got an error from FlowField in the function ‘create’ in class ‘ImageBBox’: “shape ‘[-1, 2]’ is invalid for input of size 4725”. The problem is it reads the whole prediction (189 * 25) as bounding boxes. The 189 is the number of boxes in total and the ‘25’ consists of 21 classes and 4 coordinates per box.
Could you give me some hint what could go wrong? Thanks!
hi，i have a question about inference,my dataset is define by :
data = (SegmentationItemList.from_folder(img_dir_path)
.split_by_files(val_list)
.label_from_func(get_y_fn, classes=category)
.transform(get_transforms(), tfm_y=True, size=400)
.databunch(bs=2, path=dataset_path, num_workers=6)
.normalize())
when i inference, i got the result size 400*400,that is not strange by what your explanation,(For inference, it applies the transforms you had set on the validation set),what i want to know is how to set train set transforms and validation set transforms respectively.
i seem the transforms do not apply to the validation set except the size, FCN should run with any size, i don’t want to resize src to inference then resize.what should i do to let the export model to inference picture in any size without resize to 400?
thank you.
In your call to transform, you pass a tuple of list of transforms (the return from get_transforms()). First list is for the training set, second list for the validation/test set.
Thank you very much.
The parameter of size seems not in the tfms list .Could it work for any size of input picture without resize?
When train the model I want to resize the input,but when i test and verify I want it could keep the input size.What can i do?
I do test and val without batch. Just one by one. Or you can think that there is not validation set in my model. I just train and then predict(I will computer the metrics by myself). What can i do to set the dataset?
Just create a different databunch for your test data then. There is no point having it in the same databunch as the training set since you don’t want to validate on it.