Doing predictions and showing results with v2...questions & best practice thread

I switched my work over to FastAI2 last week and today got called on to use the first model to provide predictions on live data…and found it was a struggle as most of the info here on predictions is buried in long threads with reams of obsolete code (i.e. last summer).

Thus, I’d like to make some new, how to threads to keep up with the best practice way to accomplish a given task in v2 now that’s its closer to launch.

here’s an example of what you get right now in v2:

But what everyone usually wants is :
category prediction / confidence of that prediction / file name of what was processed for that prediction?

Thanks to help from @muellerzr and esp @vijayabhaskar we have the following code snippet to decompress this into much more readable/actionable results:

preds = learn.get_preds(dl=dls.test)
for index,item in enumerate(preds):
    prediction = dls.categorize.decode(np.argmax(item)).upper()
    confidence = max(item)
    percent = float(confidence)
    print(f"{prediction}   {percent*100:.2f}% confidence.   Image = {dls.test_ds.items[index].name}")

note the image path might not work exactly…I had to change to Image = {dl2.items[index].name}")

Related questions to help make v2 predictions as usable as possible:

1 - are heatmaps going to be supported so we can see esp for failure cases?

2 - Are resize and normalize transforms automatically done when we call get_preds?

3 - Is predict() for one item and get_preds for a batch?

4 - Can we feed the results from get_preds into show batch to be able to visualize the images and results?

I did browse the source code but there’s zero comments so I don’t know what things like with_input are supposed to mean, nor rm_type_transforms.

Anyway some more definitive tutorials and best practices for actually employing the models would help everyone :slight_smile: as people pay dl practictioners to get results from their models and not just to build models for model building sake.

2 Likes

My notebooks cover a lot of that. Remember. better documentation /tutorials is still being done

  1. Yes, eventually. It’s still being developed.
  2. Item transforms are applied to your image + normalize.

3 Yes. Predict is one item and batch is batch.

  1. Your results should have a show() option depending on what it is. Can you provide a direct example of a situation?

Thanks @muellerzr.

To try and help out:

1 - I did look for your notebooks…but there are several issues;
A - the forums have many broken links to your notebooks…so that’s hard to find. (I guess you moved them over time?) so even digginng in threads and then jumping doesn’t work in many cases.

B - you have two different repos for FastAI2 notebooks…not clear which one is the master/current beyond checking lastest checkin times. (might be better to hide one as private if one is old?)

C - The inference server one doesn’t show how to do the main task (as I see it) of printing result / confidence factor / file that was processed.

I do see you enumerating the decoded preds list but I think most people want prediction / confidence / file by default.
And I’d argue that this type of summary display really should be a built in function in v2 ala learn.prediction_results(preds) or similar…otherwise everyone has to roll their own like in v1.

re: 4 - should have a .show() option - I don’t see an option. I’m referrinng to my example posted above.
I tried to pass learn.show_results(max_n=2,dl=dl2) where dl2 was my prediction batch as above but it errs out.
If there’s a .show() function that would be great but please provide a use example?

Re: Remember. better documentation /tutorials is still being done
I know and I appreciate all the work you are doing here! I’m not here to bash, I’m trying to point out what’s hard to do atm and esp for scenarios that are valid use cases.

Thanks!

1 Like

I’d keep an eye on the ones linked here. Yes, that is something I intend to fix (as I did originally make two seperate repos and then just moved to 1) A walk with fastai2 - Study Group and Online Lectures Megathread

This should then answer 1 B :slight_smile:

For confidence:

You are saying you would like % dog for example? (This is certainly something I can change, I’ve been thinking about that. See the segmentation notebook now for its inference)

  1. get_preds(even in v1) was always raw. I need to play around with decoded=True more to understand what it really does myself.

  2. I’ll get back to you on that :slight_smile:

1 Like

Here’s the code I wrote for v1 to display - if you can help translate it to v2 that would help a ton:

preds = learn.get_preds(ds_type=DatasetType.Test)    
for index,item in enumerate(preds[0]):
    prediction = classes[np.argmax(item)].upper()
    confidence = max(item)
    percent = float(confidence)
    print(f"{prediction}   {percent*100:.2f}% confidence.   Image = {data.test_ds.items[index].name}")

here’s my progress - just missing the file names:

I think this would do, I tried on validation set and it worked, it should work on test set but I’m not sure.

preds = learn.get_preds(dl=dls.test)
for index,item in enumerate(preds[0]):
    prediction = dls.categorize.decode(np.argmax(item)).upper()
    confidence = max(item)
    percent = float(confidence)
    print(f"{prediction}   {percent*100:.2f}% confidence.   Image = {dls.test_ds.items[index].name}")
3 Likes

@vijayabhaskar I’d agree with you but keep it as a dl as test_dl doesn’t exist anymore. IE:

preds = learn.get_preds(dl=dl)
for index,item in enumerate(preds[0]):
    prediction = dls.categorize.decode(np.argmax(item)).upper()
    confidence = max(item)
    percent = float(confidence)
    print(f"{prediction}   {percent*100:.2f}% confidence.   Image = {dl.items[index].name}")

I’m unsure on the tail end, I’ll play with that tonight and see what I get. This way we generalize it to any dl :slight_smile:

2 Likes

I didn’t knew test_dl doesn’t exist anymore, I just tried it couple of days back and it this worked. Thanks for informing.

1 Like

Test_dl generates a DataLoader (IIRC). It may if you have one set but we can have any amount of them. So that’s why I say may be better to generalize :slight_smile:

By generate I mean:

learn.dls.test_dl(items)

1 Like

it works! Thanks @vijayabhaskar!

The only thing I had to change was to adjust the test_ds for the image name:
ala Image = {dl2.items[index].name}")

Otherwise I now get a nice printout exactly as needed.
I’ll update my first post with your code and credit you.
Thanks again!

3 Likes