Issues with coordinate system and labels not in training set

fastai
1

I’m trying to train a model to predict where a cursor should be for a given screenshot of gameplay. I have created the training data by using template matching with each frame of some gameplay to locate the cursor. Each image is labeled with the x and y coordinates of the cursor in the form xxx-yyy-numberOfScreenshot.png, however, since I do not have data for every pixel in a 600x600 window, I get the error:

KeyError: "Label '321-333' was not included in the training dataset"

Is there any way to work around this? I’m still pretty new to this.
I realise that it is imminent for the program to come across labels that aren’t in its training set, especially when coming across unseen gameplay. My code for training the model is as shown below:

from fastai.vision.all import *
import time
import cv2
from utils.getScreen import grab_screen

def label_func(x): return x.name[0:7]

def run():
    path = Path("C:/Users/Joshua/Desktop/Data/osu-ai/")
    fnames = get_image_files(path)
    print(f"Total Images:{len(fnames)}")

    dls = ImageDataLoaders.from_path_func(path, fnames, label_func,bs=20, num_workers=0)
    learn = cnn_learner(dls, resnet18, metrics=error_rate)
    print("Loaded")
    learn.remove_cb(ProgressCallback)
    learn.fine_tune(2, base_lr=1.0e-02)

    learn.export()

If you need me to provide anything else, please let me know.

2

It might be easier to treat the labels as points rather than a string label, and predict the point(x,y) for the position.

Take a look at the Keypoint prediction example

I think this approach would work for you with minor modification.

2 Likes
3

I shall give this a try, thank you for the advice :slight_smile:

1 Like