Kaggle NLP Competition - Toxic Comment Classification Challenge

hiromi · February 16, 2018, 2:54pm

Nice work for both of you!! I’ve been looking at this competition and am leaning towards doing binary classification for each category rather than trying to do everything at once. What did you guys do?

sermakarevich · February 16, 2018, 4:09pm

Hi @hiromi. Glad to know you are there as well. I am complete newbie at NLP so I just try to learn and implement from scratch everything guys recommend to do on the forum:

tfidf on words
tfidfs on chars
naive bayes features
LSTM
GRU
fastText

Sklearn pipelines help a lot to make your code clearer and automise lots of stuff. You can basically wrap anything into sklearn Estimator.

hiromi · February 16, 2018, 5:28pm

Thanks for the tips! Wow, it’s impressive you tried all that. Can’t wait to hear all about what kind of findings you made once the competition is over

Bodhi94 · February 19, 2018, 5:56am

Hi everyone,
Thanks for the tips guys. I just hit 0.9835 using Bi-Directional GRU and Glove word Embeddings…

Anyone interested in form a team from fast.ai community ?

bruno16 · February 19, 2018, 6:24pm

Hi Bohdan,
I’m interested in forming a Team
my kaggle user is bruno16
Rgds
Bruno

treyko · February 20, 2018, 1:36am

I’m struggling with figuring out how to use the language model I trained to make predictions on multiple labels. I’m having two problems I’ve spent a couple days on.

Creating dataset splits that feed in multiple labels to torchtext. I created a custom dataset that takes in dataframes and creates a different field for each label (similar to this post: Creating a ModelData object without torchtext splits?). Is this on the right track? Or should I be feeding in a list of six numbers directly to the label field for each example? Am I on the right track? I’d post code, but I’m not sure if that’s allowed because this is a Kaggle competition.
Modifying the model decoder to output 6 predictions instead of one. As per this thread (Question on labeling text for sentiment analysis) I modified PoolingLinearClassifier to output the sigmoid of a 6 output units. Is this on the right track? I’m still not sure how the model will know what type of loss to use or which of the fields from the splits will be treated as labels.

Anyway, any help on this would be much appreciated! Is this way simpler than I’m making it? I feel like I’m missing something here!

sermakarevich · February 21, 2018, 3:27pm

How do you call py file where you keep DL stuff?

hiromi · February 25, 2018, 2:37am

You are on the right track!! Keep going

sermakarevich · February 25, 2018, 3:50pm

Very simple example of words polarity analysis based on Logit Regression coefficients.:

https://www.kaggle.com/sermakarevich/words-polarity-based-on-lr-weights

hiromi · February 25, 2018, 7:30pm

Here are attempts by classmates to load up the dataset with multiple labels (towards the bottom of the thread) if you find it helpful.

treyko · February 25, 2018, 9:20pm

Thank you so much for the help. I’m going to check out that discussion!

balajib26 · February 26, 2018, 6:28pm

I am training a Bidirectional LSTM with pretrained GLOVE embedding using Crestle GPU. It is taking 1 hour to train per epoch. Is it normal ?
When I had trained CNN with pretrained GLOVE embedding it took only 1 minute per epoch.

sermakarevich · February 26, 2018, 8:04pm

CuDNNLSTM 1 epoch takes 2-3 minutes to run on GTX 1080 Ti with 300-x embeddings.

balajib26 · February 27, 2018, 5:06am

How much time did it train to train CNN ? Was it significantly less ?

sermakarevich · February 27, 2018, 5:13am

Maybe 4-6 minutes for same 1 bidirectional layers LSTM with single FC layer of size 128.

balajib26 · February 27, 2018, 5:20am

Is anything wrong with this code ?

#MAIN Create LSTM model
model=Sequential()
model.add(Embedding(vocab_size, 300, weights=[embedding_matrix],
input_length=500, trainable=False))
model.add((Bidirectional(LSTM(50,dropout=0.2,recurrent_dropout=0.2 ))))
model.add(RepeatVector(500))
model.add((Bidirectional(LSTM(50,return_sequences=True ))))
model.add(GlobalMaxPool1D())
model.add(Dense(50, activation=“relu”))
model.add(Dropout(0.1))
model.add(Dense(6, activation=“sigmoid”))

compile the model

Adam_opt = Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
model.compile(optimizer=Adam_opt, loss=‘binary_crossentropy’, metrics=[‘acc’])

early_stopping = EarlyStopping(monitor=‘val_loss’, patience=5, mode=‘min’)
save_best = ModelCheckpoint(’/home/nbuser/toxic.hdf’, save_best_only=True,
monitor=‘val_loss’, mode=‘min’)

history = model.fit(X_train, y_train, validation_data=(X_eval, y_eval),
epochs=1, verbose=1,callbacks=[early_stopping,save_best])

Bodhi94 · February 27, 2018, 5:35am

Looks okay
I tried with 2 bi-directional LSTM layers but the model did not improve as much as I expected.

balajib26 · February 27, 2018, 5:43am

I used CuDNNLSTM instead of LSTM, that helped. Brought down the training time

sermakarevich · February 27, 2018, 6:04am

Well, it is just deeper and wider. I use input_length 100-250, you use 500. I use 1 LSTM, you use 2 LSTMs. I do not use RepeatVector which, I assume, make output of LSTM1 500 times deeper. +LSTM instead of CuDNNLSTM. This might be the difference.

balajib26 · February 27, 2018, 6:10am

Yes reducing 1 LSTM and using using CuDNNLSTM made it better.
Is 1 LSTM enough to learn a good representation ?