V2 images = questions on how to get predictions properly (multilple sources with diff?

fastai fastai dev
16

ok to answer some of my own questions thanks to the tips above and some debugging today:
1 - An exported model appears to work basically the same as a ‘loaded’ model that was saved.
2 - Use load_learner(path/file) for exported models and of course learn.load(name) for regular models saved during training.
3 - Both .predict and .get_preds ultimately use the same code of setting a callback and running a validation batch:
cb = GatherPredsCallback(with_input=with_input, with_loss=with_loss, **kwargs)

4 - .predict does a lot nicer job of unpacking the relevant details, but also only works on one image.

So full prediction code ends up like this for get_preds:

inference_folder = Path.cwd()/'inference'

#exported model predictions - steps

#get images to run
images = get_image_files(inference_folder);images

#get model name
name = 'exported_model_name.pkl'

#load model with file/path
modelex = autosave_path/name;modelex

#load exported model
learn = load_learner(modelex);learn

#pass in images to create test batch
dl = learn.dls.test_dl(images)

#get preds for batch
pred_tensor, ignored, preds = learn.get_preds(dl=dl, with_decoded=True)

#outputs of above
pred_tensor
#tensor([[0.8150, 0.0348, 0.0220, 0.0258, 0.1023]])

#category index
preds
#tensor([0])


#index into vocab to turn int index into label 
learn.dls.vocab[0] 
#"my_label

learn.dls.vocab.o2i
#output full dictionary of category index and label
#{'label1': 0, 'label2': 1, 'label3': 2, 'invalid': 3, 'negative': 4}


results = learn.predict(images[0])
#output of results

#'category_name', tensor(0), tensor([0.8150, 0.0348, 0.0220, 0.0258, 0.1023]))


#loop to spit out formatted results from get_preds
for index,item in enumerate(pred_tensor):
    prediction = learn.dls.categorize.decode(np.argmax(item)).upper()
    confidence = max(item)
    percent = float(confidence)
    print(f"{prediction}   {percent*100:.2f}% confidence.   Image = {learn.dl.items[index].name}")

#get file name(s)
learn.dl.items[0].name

#show tested image(s)
learn.dl.show_batch()
8 Likes
17

related question - is there a way to list out the transforms at prediction time/ applied to verify what is happening during prediction?
and possibly add transforms as well?
(
specifically thinking of the need to apply a RatioResize transform as incoming production images would vary in size, vs in training and validation datasets are sometimes all prepped already).

18

the docs say:
On the validation set, the crop is always a center crop (on the dimension that’s cropped).
Is there a way to override this for test predictions? a center crop in my case my eliminate the very item trying to be classified.
I’m working around by presizing before prediction but again having the ability to control what transforms are used at prediction time would be very handy.

19

@LessW2020 you might find TTA helpful, or doing squish resizing instead of crop resizing. Or you could try ‘pad’ mode.

1 Like
20

How do we loop through and show the specific images in the test set?
I see learn.dl.items will give me the list of files that just ran, but I want to view the images exactly as they were shown to the model (i.e. to see if data went missing due to crop resize etc) and to do heatmaps on specific images.
I see learn.dls.valid seems to be live, but when trying to access a specific item within it it wants a b object.
examples that don’t work:
idx=0
x = learn.dls.valid_ds[idx] #test_ds etc. all fail
show_at(learn.dls.test_ds, idx);

Anyway if anyone could advise how to view the test images as seen by the model that would be a huge help. (learn.show_batch() does show some images but it’s pulling random images and want to see specific images).
I’ll try and trace out learn.show_batch() as next step but any leads/info appreciated!

21

Your DataLoaders is learn.dls. So you can access each DataLoader as learn.dls.valid and learn.dls.train. So you can do next(iter(learn.dls.valid)) or just first(learn.dls.valid) or even learn.dls.valid.one_batch() to grab a batch, exactly as the model will see it.

2 Likes
22

Thanks very much @jeremy - greatly appreciate it!

23

happened again - seeing random preds that don’t add up to 1:
preds:
tensor([[ 1.4974, -0.1202, 0.4346, 1.1905, -1.8412],
[ 2.0858, -1.0677, -1.5246, -1.3662, 0.0106]])

also had similar garbage coming back from single image preds.
I’ll restart the kernel but it seems if the server is up and running for a while then later results are off…or maybe I somehow mucked with the dls settings after trying to get the test images… :slight_smile:

24

The fact they’re all not above 0 concerns me, as they all should be. You’re sure those aren’t losses? Is it multi-category or just classification?

25

same classification (single result from 5 classes) as I was just using and showed in above code :). just left the server running and needed to run a new images.
I’m restarting server now but just wanted to post since I saw simliar oddity last week which was then making me unclear if it was returning losses,etc.

Note - restarting the kernel fixed it - same 2 images:
tensor([[0.4320, 0.0857, 0.1492, 0.3178, 0.0153],
[0.8150, 0.0348, 0.0220, 0.0258, 0.1023]])

I’ll have to put some code in that if we get a negative number in these preds to alert and not make a prediction.

26

Hey @LessW2020,
This is really helpful as I am trying to use a similar end2end scenario for Tabular Data.
Once you are done can you please add a working extended example on any example notebook of V2 Class notebooks for Image one which is similar to your dataset.

1 Like
27

Hi @navneetkrch - glad it is helpful and yes I can make a more extended example and post it as a notebook. It’s a good idea and I’ll try to do it for ImageWoof or similar common dataset so it’s ties in with the course notebooks.

28

Is it possible to make a seperate data loader for the test data?

I made data loaders for the training data by doing the following:

train_data = DataBlock(blocks = (ImageBlock(cls=RGBYPILImage), MultiCategoryBlock),
                 get_x = lambda x:os.path.join(TRAIN, f'{x[0]}'),
                 splitter = RandomSplitter(),
                 get_y = lambda x:x[1].split(' '))

dls = train_data.dataloaders(train_df, bs=batch_size)

(here RGBYPILImage is my own class that takes 4 images and combines them into a single RGB image)

Training with this works fine.

I then thought that I’d be able to do the same thing for the test data, so I made a data loader like this:

test_data = DataBlock(blocks = (ImageBlock(cls=RGBYPILImage)),
                 get_x = lambda x:os.path.join(TEST, f'{x[0]}'))

test_dls = test_data.dataloaders(test_df, bs=batch_size)

So its exactly the same as the training data loader, but has no split and no y values. This works fine with “show_batch”, however when I try to set this data loader into the “get_preds” function it returns me a “‘DataLoaders’ has no len()” error

preds,_ = learn.get_preds(dl=test_dls)

Would anyone know what I’m doing wrong & how I can create a data loader for inference?

29

You should do dis.test_dl() and pass in the items you want.

30

Can “test_dl()” take a data loader?

The trouble is that I don’t have individual images - I have a red, green, blue and yellow image that I then need to combine into a single RGB image, which is what the “ImageBlock(cls=RGBYPILImage)” is doing.

If its not possible for test_dl() to take a data loader can my test data loader be appended onto the “learn.dls” set, so I can then reference it using the “ds_idx” value? - I tried appending it but got a “‘L’ object is not callable” error.

31

The test_dl is associated with your original type transforms in your DataLoaders. So if you trained with your special image type it should also then build it from your passed in file name as well and make a new DataLoader for you. Try it, report back what happened :slight_smile:

32

Just to add to this - test_dl is really a validation set internally in terms of transforms and image prep. So that is what will be applied to your test images.
I found this part confusing until I traced it through but test/validation set are the same transform wise.
Hope that helps.

33

That’s worked thanks! - so I supply it my data frame, containing the list of test images, and it uses this as it did with my training data.

I’d tried this before, but mistakenly was giving it the data loader I’d created for the test data, rather than simply supplying the data frame.

Thanks again.

2 Likes
34

Going mad here with predictions, what’s the proper way to inference for train\valid\test?

I have this, but it gives different results each time I run it:

def show_results(learn, start=0, n=5, detect_thresh=0.35, figsize=(10,25), ds="train"):
    ds = learn.dls.train_ds
    dl = learn.dls[0].new(shuffled=False, drop_last=False)

    _,axs = plt.subplots(n, 2, figsize=figsize)

    for b in range(start, n, BS):
        x,_,_ = dl.one_batch()
        with torch.no_grad():
            output = learn.model.eval()(x)
    ...
35

You should do ‘test_dl(learn.dls, items)’

And then do dl.one_batch() should work