What seems to happen is that the annotation file (cars_annos.mat) that comes with the two folders is not the correct annotation file. It is the one for the consolidated dataset where all images (train and test) are in the same folder. To download the correct annotation file, must get the two files from https://ai.stanford.edu/~jkrause/cars/car_devkit.tgz
I posted a notebook with the code higher in this thread.
Hope it helps.