Does anyone here know how to do multi-label classification where the order matters?

fastai
1

Hi! I’ve been trying to read three ordered symbols from an image using fastai, but I can’t figure it out.

The best I’ve managed to do is to create one model for each of the three positions, such that model 1 reads the symbol at position 1, model 2 reads the symbol at position 2 and so on. Then I’ll just run all three and combine the result. This works, but I would like to do it using just one model.

After a lot of searching I’ve stumbled upon one-hot encoding, but I’ve yet to find some examples for fastaiv2. It seems like I can’t use an numpy array or a list(the one-hot encoded label) with eg. ImageDataLoaders.from_name_func as I get the “unhashable type” error.

I’ve learned that the normal multi-label classification uses one-hot encoding, but as some symbols are rare in some positions I sometimes hit the “Labels ‘position1:half-star’ were not included in the training dataset” error. Any way to define what the one-hot encoding should look like, and also make it always use the three most likely signs?

Here are some examples of data using emojis, representing symbols:
:sunflower: :cat: :sun_with_face:
:oncoming_automobile: :sun_with_face: :oncoming_automobile:

The order matters, and there might be duplicates.

Can anyone show me an example or point me in the right direction for how to use one-hot encoded labels? And maybe some pointers on how to calculate the loss?

I would really appreciate if someone could help as I’ve spent quite a few days without getting any closer to a solution.

2

Maybe you could try image detection instead of classification. This way, the model could return the label and coordinates of each emoji. You can then use that information to get the order.
Fastai v2 does not really support object detection, but icevision does.

If you need to create bounding boxes, I would recommend labelImg. This is a lightweight python program available on GitHub.

1 Like
3

Hi Alex

Half an answer. Here is a text/category example.

Create the Panda data from with two columns category and text.
The category is string “Emogy-1;Emogy-2;Emogy-3”. The label_delim=";" creates three categories.

dfr=pd.DataFrame(pdload,columns=[‘category’,‘text’])

dlsr = TextDataLoaders.from_df(df=dfr, text_col=‘text’, label_col=‘category’, label_delim=";",y_block=MultiCategoryBlock,splitter=RandomSplitter(0.2) )

Obviously this not solve the ordering issue. My only thought was maybe the loss function could reward emojy 1 and punish emjoy 3.

The warning message means there are no examples of the categories. Try playing with the Random Splitter value or if possible increase the size of the data source.

Regards Conwyn