Ordinal classes in an ImageDataBunch

I have ordinal categorical labels I want to use in ImageDataBunch.

In my DataFrame I specify this as such:

from pandas.api.types import CategoricalDtype
cat_type = CategoricalDtype(categories=['L', 'M', 'H'], ordered=True)
label_df["adj_gr"] = label_df["adj_gr"].astype(cat_type)

#Series shows as: Categories (3, object): [L < M < H]

However when I made an ImageDataBunch from this column it reverts the categories to alphabetical order:

data = ImageDataBunch.from_df("", df=label_df, fn_col="path", label_col="adj_gr", valid_pct=0.2,
        ds_tfms=tfms, size=224, num_workers=4, padding_mode='border').normalize(imagenet_stats)
OUT: [H, L, M]

Does the order not matter for NN like it does for ML models? I checked the docs and Google but see nothing about this.