Kaggle ‘Intel & MobileODT Cervical Cancer Screening’ competition


(Ravi Teja Gutta) #62

Not necessarily, Each model will be specialized to detecting some features.So when you averaging them you are trying to use the expertise of all models.But there maybe some models that are bad and might affect the average. Averaging the predictions is just a navie technique in ensembling. There are better methods like model-stacking , rank-averaging etc.

Vgg and resnet combined seems to give better performance than individual ones

Check out https://mlwave.com/kaggle-ensembling-guide/


#63

That’s a pretty good score!


(segovia) #64

Maybe you can use even more models, say GoogleNet, AlexNet, etc.? Would averaging these results improve the score further?


(Ravi Teja Gutta) #65

Hi @shushi2000, I tried googlenet, it does not seem to give as good performance as others.We can try other stuff though FCN, resnet-style FCN, inception-style FCN etc.Someone in the kaggle forums posted that clustering the train and test has given them better scores.I need to look in that as well


#66

I think the clustering was especially meant for using the additional data, because in there you have multiple photos of the same person, so you can make sure that a photo of the same person does not end up in both train and valid and/or test. Are you using the additional data?


(Ravi Teja Gutta) #67

yes @rashudo. for me, having them both in train and valid seems to reflect the LB.It seems public LB also has a lot of same person images.


(segovia) #68

I used fc_model and trained it with slightly different droupout rates, and scored 0.969. However, I found something really odd: the best score is from an obviously not-well trained model after 8 epoch, like on my validation:

Epoch 8/8 1185/1185 [==============================] - 1s - loss: 0.7559 - acc: 0.6709 - val_loss: 0.5041 - val_acc: 0.8280`

The LB score is lower when I used more epoch and got better val_acc:

Epoch 20/20 1185/1185 [==============================] - 1s - loss: 0.2133 - acc: 0.9249 - val_loss: 0.2902 - val_acc: 0.9123

How could this happen?:cold_sweat:


#69

Your validation set is too small? Or you have the same image in your validation and test sets?


(Ravi Teja Gutta) #70

Hi @rashudo, there is a significant leak between test set and additional_train. For me, using 20% of additional_train as validation , seems to correctly represent LB .


(segovia) #71

Well this is embarrassing but yes the val images are included in the train. Now I am fixing this and see if I can get a better score. Thank you, @rashudo.


(segovia) #72

Hi @rteja1113, have you tried to submit those 5 sets of predictions separately? I am just wondering if anyone of them itself can get you a better LB score than the average does.


(Ravi Teja Gutta) #73

No @shushi2000, I didn’t submit individual submissions.I didn’t want to waste submissions.But usually averaging does improve more often than not.


(segovia) #74

Hi @rteja1113, For now the VGG16 with customized fc layers gave me best score of 0.80 and I want to use ResNet next.
Could you please let me know how you recreate the ResNet model, as Jerome did with VGG16 in the lecture.
Here’s what I have done and how I got stuck:

%matplotlib inline
import utils; reload(utils)
from utils import *
from __future__ import division, print_function
#from resnet50 import *
from keras.applications.resnet50 import ResNet50

from keras.models import Model
from keras.layers import Flatten, Dense, Dropout
from keras.layers.normalization import BatchNormalization

resnet_model=ResNet50()

layers = resnet_model.layers

last_conv_idx = [index for index, layer in enumerate(layers) 
                 if type(layer) is Convolution2D][-1]

last_conv_idx #the output is 177

conv_layers=layers[:last_conv_idx+1]

conv_model = Sequential(conv_layers) #Stuck here...

Error messages like:

TypeError                                 Traceback (most recent call last)
<ipython-input-9-34b747f79c6f> in <module>()
     21 conv_layers=layers[:last_conv_idx+1]
     22 
---> 23 conv_model = Sequential(conv_layers)

/home/shi/anaconda2/lib/python2.7/site-packages/keras/models.pyc in __init__(self, layers, name)
    271         if layers:
    272             for layer in layers:
--> 273                 self.add(layer)
    274 
    275     def add(self, layer):

/home/shi/anaconda2/lib/python2.7/site-packages/keras/models.pyc in add(self, layer)
    330                  output_shapes=[self.outputs[0]._keras_shape])
    331         else:
--> 332             output_tensor = layer(self.outputs[0])
    333             if isinstance(output_tensor, list):
    334                 raise TypeError('All layers in a Sequential model '

/home/shi/anaconda2/lib/python2.7/site-packages/keras/engine/topology.pyc in __call__(self, inputs, mask)
   1430         if not isinstance(inputs, list):
   1431             raise TypeError('Merge can only be called on a list of tensors, '
-> 1432                             'not a single tensor. Received: ' + str(inputs))
   1433         if self.built:
   1434             raise RuntimeError('A Merge layer cannot be used more than once, '

TypeError: Merge can only be called on a list of tensors, not a single tensor. Received: if{}.0

I highly appreciate it!


(Ravi Teja Gutta) #75

Hi @shushi2000, I did
size=(448, 448)
res448 = Resnet50(size=size, include_top=False).model

I don’t think you can use Sequential() for resnet because the architecture is not sequential unlike vgg16.There are branches in resnet like explained in Lesson7


(segovia) #76

Oh, I see. Thank you so much! I will try the include_top=False method now.

BTW, I submitted an averaged results today, the LB score is better than any individual submission. :grin: Just in case someone else wants to know.


(Eric Perbos-Brinck) #77

Hello @rteja1113,

Could you share how/where you setup 'image size 448x448" during the precomputing of vgg16bn, while it’s set as 224x224 standard ?

And why 448x448 and not more, since most pictures in MobileODT seem to have a minimum of 2448 either in width or height ?

That is according to this EDA post on Kaggle.

https://www.kaggle.com/philschmidt/cervix-eda-model-selection

Many thanks,

Eric


(Ravi Teja Gutta) #78

Hi @EricPB, I just did Vgg16BN(size=(448,448), include_top=False)

No particular reason why I picked 448.Someone in the forums mentioned that they were using 448.Surprisingly, I was getting comparable results when I use 128 even.


(Eric Perbos-Brinck) #79

Many thanks !

Unfortunately my structure might be different than yours, I use the one from Statefarm Full of @Jeremy

When I enter your input such as:

Import our class

import vgg16bn_p3; reload(vgg16bn_p3)
from vgg16bn_p3 import Vgg16BN

Grab VGG16 and find the last convolutional layer

vgg = Vgg16BN(size=(512,512), include_top=False)
model=vgg.model
last_conv_idx = [i for i,l in enumerate(model.layers) if type(l) is Convolution2D][-1]
conv_layers = model.layers[:last_conv_idx+1]

Build a new model that includes everything up to that last convolutional layer

conv_model = Sequential(conv_layers)

Predict the outputs of that model by calculating the activations of that last convolutional layer

conv_feat = conv_model.predict_generator(batches, int(np.ceil(batches.samples/batch_size)), workers=3)

It triggers a ValueError:

ValueError: Error when checking : expected lambda_3_input to have shape (None, 3, 512, 512) but got array with shape (64, 3, 224, 224)

E.


(Ravi Teja Gutta) #80

Hi @EricPB, you need to mention the size in get_batches() as well.
batches = get_batches(your_directory, target_size=(512, 512))


(Eric Perbos-Brinck) #81

@rteja1113,

Indeed setting image input to 512x512 didn’t improve the score but worsen it from 1.45 to 1.85 :disappointed:

@shushi2000 / @Christina / @rashudo / @1729 : did it improve the score for you ?

Here’s my full notebook’s code with context:

Eric