Dog Breed Identification challenge


(Chris Palmer) #202

Still getting the error!

Here is my entire script (just the actual setup parts):

from fastai.conv_learner import *
PATH = "data/dogbreed/"
arch=resnext101_64
sz=224
bs=64

label_csv = f'{PATH}labels.csv'
n = len(list(open(label_csv)))-1
#val_idxs = get_cv_idxs(n, val_pct=1e-4)
#val_idxs = get_cv_idxs(n, val_pct=0.01)

val_idxs = [0]
n, len(val_idxs), val_idxs
# (10222, 1, [0])

label_df = pd.read_csv(label_csv)

tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)
data = ImageClassifierData.from_csv(PATH, 'train', f'{PATH}labels.csv', bs=bs, tfms=tfms, 
                                    val_idxs=val_idxs, suffix = '.jpg', test_name = 'test',
                                    num_workers=4)

fn = PATH+data.val_ds.fnames[0]; fn
# 'data/dogbreed/train/000bec180eb18c7604dcecc8fe0dba07.jpg' 

 ## why is it going to my train directory maybe I need to redistribute images 
# to suit reconfiguration of data sets??

# to be sure check just one image in val set, an error is generated if I attempt to 
# reference another image
data.val_ds.fnames[1]

# ---------------------------------------------------------------------------
# IndexError                                Traceback (most recent call last)
# <ipython-input-12-be174aefb09c> in <module>()
# ----> 1 data.val_ds.fnames[1]
# IndexError: index 1 is out of bounds for axis 0 with size 1

fn = PATH+data.test_ds.fnames[0]; fn
'data/dogbreed/test/fd1a7be32f10493735555e62913c0841.jpg'

fn = PATH+data.trn_ds.fnames[0]; fn
# 'data/dogbreed/train/001513dfcb2ffafc82cccf4d8bbaba97.jpg'

def get_data(sz, bs):
    tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)
    data = ImageClassifierData.from_csv(PATH, 'train', f'{PATH}labels.csv', test_name='test',
                                        val_idxs=val_idxs, suffix='.jpg', tfms=tfms, bs=bs)
    return data if sz>300 else data.resize(340, 'tmp')

sz, bs
# (224, 64)

data = get_data(sz, bs)

learn = ConvLearner.pretrained(arch, data, precompute=True, ps=0.5)

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-47-0708a7145fb8> in <module>()
----> 1 learn = ConvLearner.pretrained(arch, data, precompute=True, ps=0.5)

~/fastai/courses/dl1/fastai/conv_learner.py in pretrained(cls, f, data, ps, xtra_fc, xtra_cut, **kwargs)
     92     def pretrained(cls, f, data, ps=None, xtra_fc=None, xtra_cut=0, **kwargs):
     93         models = ConvnetBuilder(f, data.c, data.is_multi, data.is_reg, ps=ps, xtra_fc=xtra_fc, xtra_cut=xtra_cut)
---> 94         return cls(data, models, **kwargs)
     95 
     96     @property

~/fastai/courses/dl1/fastai/conv_learner.py in __init__(self, data, models, precompute, **kwargs)
     85         elif self.metrics is None:
     86             self.metrics = [accuracy_multi] if self.data.is_multi else [accuracy]
---> 87         if precompute: self.save_fc1()
     88         self.freeze()
     89         self.precompute = precompute

~/fastai/courses/dl1/fastai/conv_learner.py in save_fc1(self)
    132         self.fc_data = ImageClassifierData.from_arrays(self.data.path,
    133                 (act, self.data.trn_y), (val_act, self.data.val_y), self.data.bs, classes=self.data.classes,
--> 134                 test = test_act if self.data.test_dl else None, num_workers=8)
    135 
    136     def freeze(self):

~/fastai/courses/dl1/fastai/dataset.py in from_arrays(cls, path, trn, val, bs, tfms, classes, num_workers, test)
    296             ImageClassifierData
    297         """
--> 298         datasets = cls.get_ds(ArraysIndexDataset, trn, val, tfms, test=test)
    299         return cls(path, datasets, bs, num_workers, classes=classes)
    300 

~/fastai/courses/dl1/fastai/dataset.py in get_ds(fn, trn, val, tfms, test, **kwargs)
    264     def get_ds(fn, trn, val, tfms, test=None, **kwargs):
    265         res = [
--> 266             fn(trn[0], trn[1], tfms[0], **kwargs), # train
    267             fn(val[0], val[1], tfms[1], **kwargs), # val
    268             fn(trn[0], trn[1], tfms[1], **kwargs), # fix

~/fastai/courses/dl1/fastai/dataset.py in __init__(self, x, y, transform)
    160     def __init__(self, x, y, transform):
    161         self.x,self.y=x,y
--> 162         assert(len(x)==len(y))
    163         super().__init__(transform)
    164     def get_x(self, i): return self.x[i]

AssertionError:  

(James Requa) #203

Yes I do personally use it in multiple stages just to make sure my learning rate is still looking good based on the weight updates. Other “transitions” like you mention with resizing the images can also be a good time to use it.

BTW, I highly recommend going through the notebook that Jeremy recently created called cifar10 as it should provide you with a good intuition about how best to use lr_finder, cycle_len, cycle_mult and resizing.


(Kevin Bird) #204

Try deleting your tmp directory. If that doesn’t work then it’s possible you have found a bug and maybe you could dig into it to see what is happening in get_ds. I would start with just deleting your tmp directory though.


(James Requa) #205

Also if you wanted a line by line example, Jeremy did post the code that could be used to create a submission file specifically for dog breed at the top of the Lesson 3 Wiki.


(Chris Palmer) #206

I don’t seem to have a tmp directory!


(Phani Srikanth) #207

The tmp & models directory are inside your data/dogbreeds/ folder. Could you check there?


(Chris Palmer) #208

Hi @KevinB

I am new to Jupyter Notebook and the Ubuntu environment. How can I debug into a called module using these tools?


(Vishnu N) #209

That makes sense, thanks! So data.test_dl.dataset.fnames should return a list of file IDs to direct output values, or does it just contain file index? If the former, then I just need to match up the csv ids to the fnames result, save the csv, and submit it I think.


(sergii makarevych) #210

18.81s/it per 64 images of size 350. Indeed really slow )


(Chris Palmer) #211

Thanks

There are 4 models in that directory, but they are all a week old and dated similarly, so I guess they come from the wor I did last week!


(Chris Palmer) #212

@binga let me know that I should look for tmp under my data/dogbreed folder, and yes there is a tmp there. I removed all of the files, and bingo - it works now!

Thanks :slight_smile:

I am trying the trick of training with all of the data, but as you say its a bit odd to have no validation and it feels like cheating, because I am training with what might be tested against.

But anyway, when I come to validation time under this scenario there is something I need to do - about labels - is this something you are aware of?

log_preds, y = learn.TTA()
probs = np.exp(log_preds)
accuracy(log_preds, y), metrics.log_loss(y, probs)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-30-47692796ae62> in <module>()
      1 log_preds, y = learn.TTA()
      2 probs = np.exp(log_preds)
----> 3 accuracy(log_preds, y), metrics.log_loss(y, probs)

~/src/anaconda3/envs/fastai/lib/python3.6/site-packages/sklearn/metrics/classification.py in log_loss(y_true, y_pred, eps, normalize, sample_weight, labels)
   1652             raise ValueError('y_true contains only one label ({0}). Please '
   1653                              'provide the true labels explicitly through the '
-> 1654                              'labels argument.'.format(lb.classes_[0]))
   1655         else:
   1656             raise ValueError('The labels array needs to contain at least two '

ValueError: y_true contains only one label (19). Please provide the true labels explicitly through the labels argument.

(Phani Srikanth) #213

The syntax for sklearn.metrics.accuracy is accuracy(y_true, y_pred) and not the other way around. Could you try flipping log_preds and y?

Edit: Also, for accuracy, you’ll have to feed the labels not the probabilities!


(Antti) #214

Just wanted to thank everyone for your contributions to this “dog breed identification” thread!

I have today made my first-ever Kaggle submissions, and immediately got to #21 on the leaderboard :slight_smile: . Would have been impossible without your help.

What I did with my three submissions:
(1) follow lesson instructions with model_1
(2) follow lesson instructions with model_2
(3) averaged the predictions from those two

Thanks again!


(James Requa) #215

If you are training with all of the data (including validation set) there isn’t much point to doing any prediction on the validation set which is what learn.TTA() is doing - because you effectively no longer have a validation set :slight_smile: . In this case you should just go directly to learn.TTA(test=True) to predict on the test set.

I assume you have already trained on your training set only while setting aside a validation set (i.e. 20%) and tuned your parameters based on the training/validation losses you were seeing? So now you are just going back and repeating the same steps but now without a validation set?


(Rikiya Yamashita) #216

Congrats @sermakarevich on sub-0.14 :wink:


(Rikiya Yamashita) #217

Sorry for my confusing code @Chris_Palmer.
Mine is just an alternative to val_idxs = [0].


(James Requa) #218

Yes it returns a list of paths to each file which includes the fnames id. So you can just extract the fnames from those and use them directly for your id column of the submission file since your predictions are already following the same ordering as those fnames and luckily the classes are already in the correct order / alphabetized.


(Chris Palmer) #219

Yes, that’s what I am doing. But the way you ask the question makes me wonder if I am going about this the right way. I have for sure got the best I can with a validation set, so I used the same approach to train a brand new model with no validation set. Is this correct, or should I take the existing model I have and retrain it in the same manner with the entire set?


(Wayne Nixalo) #220

Anyone else getting a learning rate scheduler plot looking like this? Doesn’t seem right. Right at the beginning after initializing a learner – using resnext50.

screenshot


(sergii makarevych) #221

What kind of data set it is? Any chance you have few images?