Increasing validation set accuracy

Hi

My validation accuracy is stuck at 3% and I need some help… Appreciate any pointers.

What I am trying to do: Face recognition.

What am I stuck with:
After about 50 epochs my training accuracy is at 99.17% but my validation is stuck at 3%

Epoch 19/20
5770/5770 [==============================] - 2s - loss: 0.0490 - acc: 0.9931 - val_loss: 8.2996 - val_acc: 0.0239
Epoch 20/20
5770/5770 [==============================] - 2s - loss: 0.0464 - acc: 0.9917 - val_loss: 8.4025 - val_acc: 0.0301

A little bit about my Data Set

I am using this data set.

This dataset contained 13k+ images of over 2000 people with some people having 1, 2, 3 4 or more images.
I trimmed my data set to working on only 610 people with 5770 images so all classes have 4 or more images.

Out of these I moved 10% of images to the validation set. For some classes it meant a 3:1 split leaving only one image in the validation set.

My Architecture:
At first I tried to create the model myself as follows but that was slow (very slow) and it only got to 70% and 35% accuracy after a lot of training so I changed my approach.

def conv1(batches):
    model = Sequential([
            BatchNormalization(axis=1, input_shape=(3,176,176)),
            Convolution2D(64,3,3, activation='relu'),
            BatchNormalization(axis=1),
            MaxPooling2D(),
            Convolution2D(128,3,3, activation='relu'),
            BatchNormalization(axis=1),
            MaxPooling2D(),
            Convolution2D(256,3,3, activation='relu'),
            BatchNormalization(axis=1),
            MaxPooling2D(),
            Flatten(),
            Dense(100, activation='relu'),
            BatchNormalization(),
            Dropout(0.0),
            Dense(100, activation='relu'),
            BatchNormalization(),
            Dropout(0.0),
            Dense(610, activation='softmax')
        ])

    return model

I used the technique we learned in class where I took the convolution layers from vgg16 and then added batch norm layers.

My code/gist: is available here.

Appreciate any help and pointers on what i might be doing wrong…

Thx

Ah yes I see the problem :slight_smile: It’s a mistake we’ve all made before! Your validation batches are being shuffled, and then those shuffled batches are being used to create your convolutional features. You’re then trying to make predictions with those shuffled batches and are comparing them to the unshuffled validation labels. Of course, the two sets are totally unrelated!

You should add ‘shuffle=False’ to both of your gen_batches to fix this problem. The fit() call will handle the shuffling for you automatically.

2 Likes

Wow!!! @jeremy that was an amazing tip!

Epoch 19/20
5770/5770 [==============================] - 3s - loss: 0.0964 - acc: 0.9913 - val_loss: 4.2995 - val_acc: 0.2409
Epoch 20/20
5770/5770 [==============================] - 3s - loss: 0.1025 - acc: 0.9879 - val_loss: 4.2990 - val_acc: 0.2420
Out[22]:

My validation accuracy is now jumped up to ~25% but its stuck there now…

Could you show the results of all 20 epochs?

Here:

Train on 5770 samples, validate on 963 samples
Epoch 1/20
5770/5770 [==============================] - 3s - loss: 0.1760 - acc: 0.9801 - val_loss: 4.2602 - val_acc: 0.2430
Epoch 2/20
5770/5770 [==============================] - 3s - loss: 0.1793 - acc: 0.9790 - val_loss: 4.2576 - val_acc: 0.2399
Epoch 3/20
5770/5770 [==============================] - 3s - loss: 0.1675 - acc: 0.9823 - val_loss: 4.2436 - val_acc: 0.2430
Epoch 4/20
5770/5770 [==============================] - 3s - loss: 0.1685 - acc: 0.9795 - val_loss: 4.2683 - val_acc: 0.2430
Epoch 5/20
5770/5770 [==============================] - 3s - loss: 0.1609 - acc: 0.9809 - val_loss: 4.2857 - val_acc: 0.2378
Epoch 6/20
5770/5770 [==============================] - 3s - loss: 0.1472 - acc: 0.9839 - val_loss: 4.2227 - val_acc: 0.2409
Epoch 7/20
5770/5770 [==============================] - 3s - loss: 0.1550 - acc: 0.9835 - val_loss: 4.2842 - val_acc: 0.2347
Epoch 8/20
5770/5770 [==============================] - 3s - loss: 0.1376 - acc: 0.9858 - val_loss: 4.2705 - val_acc: 0.2378
Epoch 9/20
5770/5770 [==============================] - 3s - loss: 0.1330 - acc: 0.9872 - val_loss: 4.2597 - val_acc: 0.2378
Epoch 10/20
5770/5770 [==============================] - 3s - loss: 0.1365 - acc: 0.9834 - val_loss: 4.2871 - val_acc: 0.2482
Epoch 11/20
5770/5770 [==============================] - 3s - loss: 0.1277 - acc: 0.9877 - val_loss: 4.2630 - val_acc: 0.2461
Epoch 12/20
5770/5770 [==============================] - 3s - loss: 0.1297 - acc: 0.9856 - val_loss: 4.2586 - val_acc: 0.2368
Epoch 13/20
5770/5770 [==============================] - 3s - loss: 0.1266 - acc: 0.9851 - val_loss: 4.2636 - val_acc: 0.2471
Epoch 14/20
5770/5770 [==============================] - 3s - loss: 0.1238 - acc: 0.9858 - val_loss: 4.2832 - val_acc: 0.2368
Epoch 15/20
5770/5770 [==============================] - 3s - loss: 0.1206 - acc: 0.9854 - val_loss: 4.3388 - val_acc: 0.2357
Epoch 16/20
5770/5770 [==============================] - 3s - loss: 0.1189 - acc: 0.9853 - val_loss: 4.2861 - val_acc: 0.2388
Epoch 17/20
5770/5770 [==============================] - 3s - loss: 0.1132 - acc: 0.9846 - val_loss: 4.2867 - val_acc: 0.2388
Epoch 18/20
5770/5770 [==============================] - 3s - loss: 0.1061 - acc: 0.9873 - val_loss: 4.2814 - val_acc: 0.2430
Epoch 19/20
5770/5770 [==============================] - 3s - loss: 0.0964 - acc: 0.9913 - val_loss: 4.2995 - val_acc: 0.2409
Epoch 20/20
5770/5770 [==============================] - 3s - loss: 0.1025 - acc: 0.9879 - val_loss: 4.2990 - val_acc: 0.2420

There’s something odd with your validation set. It can’t be a random sample, based on the epoch results you’ve shown here. Did you intend it to be a random sample?

@jeremy I am not sure I completely understand that question.

I am sharing the train and valid sets for Adrien Brody’s images with you on slack (here it doesnt allow to upload zip files).

I basically moved 10% of images from train folder to the valid folder for each person.
This is what i did

  1. Copied all training folder valid

  2. Deleted 90% of the files in the valid folder with the code below.

    matches = []
    for root, dirnames, filenames in os.walk(‘valid’):
    filecount = len(filenames)

     for filename in fnmatch.filter(filenames, '*.jpg'):
         matches.append(os.path.join(root, filename))
     for i in range(int(filecount*0.9)): 
         os.remove(matches[i])
     matches = []
    
  3. If a file is in valid, delete it from training with this code

    import os, fnmatch, shutil
    matches = []

    for root, dirnames, filenames in os.walk(‘valid’):
    filecount = len(filenames)

     for filename in fnmatch.filter(filenames, '*.jpg'):
         matches.append(os.path.join(root, filename))
         check = root.replace("valid", "train")
    

    print root, check

         trainFileName = os.path.join (check,filename)
         if os.path.exists(trainFileName):
           print ('This file should be deleted')
           print trainFileName
           os.remove(trainFileName)
     print matches
    
     matches = []
    

I know its a bit round about way and could be changed to be

  1. From train move 10% image to valid folder.

@jeremy I made 2 changes.

  1. I changed my data set to have atleast 5 or more images per person
    and

  2. I changed the way I was creating the validation set using this code

    for root, dirnames, filenames in os.walk(‘train’):
    filecount = len(filenames)
    for filename in fnmatch.filter(filenames, ‘*.jpg’):
    matches.append(os.path.join(root, filename))
    print filecount

     files2move = math.ceil(filecount*0.2)
     print files2move
    
     dircheck = root.replace("train", "valid")
     if not os.path.exists(dircheck):
         os.makedirs(dircheck)
    
     shuf = np.random.permutation(matches)
     for i in range(int(files2move)):
         filename = shuf[i]
         check = filename.replace("train", "valid")
         shutil.move(shuf[i], dircheck)
         print 'moving file' + shuf[i] + 'to ' +dircheck
     matches = []
    

Now I am just copy 20% of the files from train to valid randomly versus the first few.

I see a minor improvement in my validation set. Its gone up to 32 % but still stuck there.

Train on 4640 samples, validate on 1345 samples
Epoch 1/50
4640/4640 [==============================] - 2s - loss: 0.2061 - acc: 0.9759 - val_loss: 3.6852 - val_acc: 0.3093
Epoch 2/50
4640/4640 [==============================] - 2s - loss: 0.2000 - acc: 0.9746 - val_loss: 3.6874 - val_acc: 0.3071
Epoch 3/50
4640/4640 [==============================] - 2s - loss: 0.1850 - acc: 0.9791 - val_loss: 3.6791 - val_acc: 0.3086
Epoch 4/50
4640/4640 [==============================] - 2s - loss: 0.1697 - acc: 0.9808 - val_loss: 3.6994 - val_acc: 0.3108
Epoch 5/50
4640/4640 [==============================] - 2s - loss: 0.1708 - acc: 0.9823 - val_loss: 3.6979 - val_acc: 0.3041
Epoch 6/50
4640/4640 [==============================] - 2s - loss: 0.1678 - acc: 0.9815 - val_loss: 3.6846 - val_acc: 0.3093
Epoch 7/50
4640/4640 [==============================] - 2s - loss: 0.1624 - acc: 0.9815 - val_loss: 3.6689 - val_acc: 0.3152
Epoch 8/50
4640/4640 [==============================] - 2s - loss: 0.1580 - acc: 0.9806 - val_loss: 3.6748 - val_acc: 0.3115
Epoch 9/50
4640/4640 [==============================] - 2s - loss: 0.1498 - acc: 0.9838 - val_loss: 3.6728 - val_acc: 0.3115
Epoch 10/50
4640/4640 [==============================] - 2s - loss: 0.1376 - acc: 0.9858 - val_loss: 3.6634 - val_acc: 0.3190
Epoch 11/50
4640/4640 [==============================] - 2s - loss: 0.1293 - acc: 0.9862 - val_loss: 3.7182 - val_acc: 0.3063
Epoch 12/50
4640/4640 [==============================] - 2s - loss: 0.1303 - acc: 0.9845 - val_loss: 3.7056 - val_acc: 0.3175
Epoch 13/50
4640/4640 [==============================] - 2s - loss: 0.1240 - acc: 0.9860 - val_loss: 3.6960 - val_acc: 0.3167
Epoch 14/50
4640/4640 [==============================] - 2s - loss: 0.1150 - acc: 0.9884 - val_loss: 3.6558 - val_acc: 0.3167
Epoch 15/50
4640/4640 [==============================] - 2s - loss: 0.1174 - acc: 0.9862 - val_loss: 3.6636 - val_acc: 0.3160
Epoch 16/50
4640/4640 [==============================] - 2s - loss: 0.1201 - acc: 0.9866 - val_loss: 3.7061 - val_acc: 0.3048
Epoch 17/50
4640/4640 [==============================] - 2s - loss: 0.1133 - acc: 0.9860 - val_loss: 3.7257 - val_acc: 0.3071
Epoch 18/50
4640/4640 [==============================] - 2s - loss: 0.1111 - acc: 0.9884 - val_loss: 3.7025 - val_acc: 0.3108
Epoch 19/50
4640/4640 [==============================] - 2s - loss: 0.0985 - acc: 0.9892 - val_loss: 3.7192 - val_acc: 0.3138
Epoch 20/50
4640/4640 [==============================] - 2s - loss: 0.1029 - acc: 0.9877 - val_loss: 3.7259 - val_acc: 0.3160
Epoch 21/50
4640/4640 [==============================] - 2s - loss: 0.1016 - acc: 0.9894 - val_loss: 3.7275 - val_acc: 0.3115
Epoch 22/50
4640/4640 [==============================] - 2s - loss: 0.0889 - acc: 0.9925 - val_loss: 3.6850 - val_acc: 0.3100
Epoch 23/50
4640/4640 [==============================] - 2s - loss: 0.0922 - acc: 0.9873 - val_loss: 3.6679 - val_acc: 0.3138
Epoch 24/50
4640/4640 [==============================] - 2s - loss: 0.0834 - acc: 0.9920 - val_loss: 3.6860 - val_acc: 0.3145
Epoch 25/50
4640/4640 [==============================] - 2s - loss: 0.0947 - acc: 0.9886 - val_loss: 3.6735 - val_acc: 0.3071
Epoch 26/50
4640/4640 [==============================] - 2s - loss: 0.0928 - acc: 0.9892 - val_loss: 3.6786 - val_acc: 0.3197
Epoch 27/50
4640/4640 [==============================] - 2s - loss: 0.0793 - acc: 0.9914 - val_loss: 3.6698 - val_acc: 0.3152
Epoch 28/50
4640/4640 [==============================] - 2s - loss: 0.0791 - acc: 0.9920 - val_loss: 3.7121 - val_acc: 0.3086
Epoch 29/50
4640/4640 [==============================] - 2s - loss: 0.0737 - acc: 0.9938 - val_loss: 3.6995 - val_acc: 0.3078
Epoch 30/50
4640/4640 [==============================] - 2s - loss: 0.0760 - acc: 0.9918 - val_loss: 3.6755 - val_acc: 0.3138
Epoch 31/50
4640/4640 [==============================] - 2s - loss: 0.0689 - acc: 0.9927 - val_loss: 3.6611 - val_acc: 0.3190
Epoch 32/50
4640/4640 [==============================] - 2s - loss: 0.0711 - acc: 0.9922 - val_loss: 3.6857 - val_acc: 0.3160
Epoch 33/50
4640/4640 [==============================] - 2s - loss: 0.0750 - acc: 0.9922 - val_loss: 3.6635 - val_acc: 0.3152
Epoch 34/50
4640/4640 [==============================] - 2s - loss: 0.0671 - acc: 0.9946 - val_loss: 3.6950 - val_acc: 0.3115
Epoch 35/50
4640/4640 [==============================] - 2s - loss: 0.0650 - acc: 0.9935 - val_loss: 3.7160 - val_acc: 0.3093
Epoch 36/50
4640/4640 [==============================] - 2s - loss: 0.0666 - acc: 0.9929 - val_loss: 3.7227 - val_acc: 0.3004
Epoch 37/50
4640/4640 [==============================] - 2s - loss: 0.0659 - acc: 0.9935 - val_loss: 3.7066 - val_acc: 0.3093
Epoch 38/50
4640/4640 [==============================] - 2s - loss: 0.0573 - acc: 0.9938 - val_loss: 3.6489 - val_acc: 0.3123
Epoch 39/50
4640/4640 [==============================] - 2s - loss: 0.0662 - acc: 0.9922 - val_loss: 3.7203 - val_acc: 0.3093
Epoch 40/50
4640/4640 [==============================] - 2s - loss: 0.0604 - acc: 0.9938 - val_loss: 3.7301 - val_acc: 0.3033
Epoch 41/50
4640/4640 [==============================] - 2s - loss: 0.0577 - acc: 0.9940 - val_loss: 3.7423 - val_acc: 0.3071
Epoch 42/50
4640/4640 [==============================] - 2s - loss: 0.0527 - acc: 0.9950 - val_loss: 3.6948 - val_acc: 0.3152
Epoch 43/50
4640/4640 [==============================] - 2s - loss: 0.0544 - acc: 0.9931 - val_loss: 3.6603 - val_acc: 0.3123
Epoch 44/50
4640/4640 [==============================] - 2s - loss: 0.0523 - acc: 0.9946 - val_loss: 3.6904 - val_acc: 0.3086
Epoch 45/50
4640/4640 [==============================] - 2s - loss: 0.0522 - acc: 0.9942 - val_loss: 3.7916 - val_acc: 0.3063
Epoch 46/50
4640/4640 [==============================] - 2s - loss: 0.0507 - acc: 0.9953 - val_loss: 3.7368 - val_acc: 0.3115
Epoch 47/50
4640/4640 [==============================] - 2s - loss: 0.0503 - acc: 0.9946 - val_loss: 3.7316 - val_acc: 0.3086
Epoch 48/50
4640/4640 [==============================] - 2s - loss: 0.0468 - acc: 0.9961 - val_loss: 3.7734 - val_acc: 0.3130
Epoch 49/50
4640/4640 [==============================] - 2s - loss: 0.0551 - acc: 0.9922 - val_loss: 3.7223 - val_acc: 0.3115
Epoch 50/50
4640/4640 [==============================] - 2s - loss: 0.0514 - acc: 0.9942 - val_loss: 3.7266 - val_acc: 0.3204

I am going to see if I can change weight optimizer

Weight regularization didnt help.

But I realized I had the shuffle=False set in a fit statement

bn_model.fit(conv_feat, trn_labels, batch_size=batch_size, nb_epoch=10,
validation_data=(conv_val_feat, val_labels), shuffle=False)

removing that helped me get up to 40 … but still stuck at 40% now.

Appreciate any help!

Looks like you’re making continuous improvements! :slight_smile: Nice job.

How many classes are you predicting? What’s the mean and median number of images per class?

I have 423 classes to predict.

For train I have 4640 images in 423 classes = 10.9 mean
For valid I have 1345 images in 423 classes = 3.17 mean

For train the median is 6
and for Valid the median is 2

One odd thing i noticed is that i have the largest number of files in one folder that has 424 files in train. The next number was 188.
This might skewing up the number.

I can reduce the number of files to 10 and try

Okay, that makes sense. You have a very large number of classes, with few images per class. I would suggest removing classes with less than 20 validation images, if that’s possible.

@jeremy that definitely seems to be the issue. It got better when i trim the data set. I am getting 63% now. :frowning:

Train set file counts
[28, 101, 32, 510, 57, 32, 40, 29, 24, 21, 33, 21, 51, 89, 216, 22, 35, 124, 22]
[ 0 21 21 22 22 24 28 29 32 32 33 35 40 51 57 89 101 124
216 510]

Valid set file counts
[20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20]
[ 0 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20]

Train on 1487 samples, validate on 380 samples
Epoch 1/50
1487/1487 [==============================] - 0s - loss: 0.0109 - acc: 0.9993 - val_loss: 1.2869 - val_acc: 0.6289
Epoch 2/50
1487/1487 [==============================] - 0s - loss: 0.0092 - acc: 1.0000 - val_loss: 1.2778 - val_acc: 0.6263
Epoch 3/50
1487/1487 [==============================] - 0s - loss: 0.0095 - acc: 0.9987 - val_loss: 1.2988 - val_acc: 0.6211
Epoch 4/50
1487/1487 [==============================] - 0s - loss: 0.0113 - acc: 1.0000 - val_loss: 1.2971 - val_acc: 0.6237
Epoch 5/50
1487/1487 [==============================] - 0s - loss: 0.0108 - acc: 0.9993 - val_loss: 1.3061 - val_acc: 0.6132
Epoch 6/50
1487/1487 [==============================] - 0s - loss: 0.0116 - acc: 1.0000 - val_loss: 1.3184 - val_acc: 0.6105
Epoch 7/50
1487/1487 [==============================] - 0s - loss: 0.0131 - acc: 0.9973 - val_loss: 1.3714 - val_acc: 0.6053
Epoch 8/50
1487/1487 [==============================] - 0s - loss: 0.0128 - acc: 0.9980 - val_loss: 1.3709 - val_acc: 0.6026
Epoch 9/50
1487/1487 [==============================] - 0s - loss: 0.0111 - acc: 0.9987 - val_loss: 1.3476 - val_acc: 0.6079
Epoch 10/50
1487/1487 [==============================] - 0s - loss: 0.0099 - acc: 1.0000 - val_loss: 1.3321 - val_acc: 0.6184
Epoch 11/50
1487/1487 [==============================] - 0s - loss: 0.0130 - acc: 0.9980 - val_loss: 1.3309 - val_acc: 0.6184
Epoch 12/50
1487/1487 [==============================] - 0s - loss: 0.0089 - acc: 0.9993 - val_loss: 1.3063 - val_acc: 0.6316
Epoch 13/50
1487/1487 [==============================] - 0s - loss: 0.0096 - acc: 0.9993 - val_loss: 1.3075 - val_acc: 0.6316
Epoch 14/50
1487/1487 [==============================] - 0s - loss: 0.0113 - acc: 0.9987 - val_loss: 1.3091 - val_acc: 0.6368
Epoch 15/50
1487/1487 [==============================] - 0s - loss: 0.0092 - acc: 1.0000 - val_loss: 1.3220 - val_acc: 0.6263
Epoch 16/50
1487/1487 [==============================] - 0s - loss: 0.0078 - acc: 1.0000 - val_loss: 1.3392 - val_acc: 0.6263
Epoch 17/50
1487/1487 [==============================] - 0s - loss: 0.0102 - acc: 1.0000 - val_loss: 1.3656 - val_acc: 0.6132
Epoch 18/50
1487/1487 [==============================] - 0s - loss: 0.0106 - acc: 0.9993 - val_loss: 1.3572 - val_acc: 0.6132
Epoch 19/50
1487/1487 [==============================] - 0s - loss: 0.0115 - acc: 0.9980 - val_loss: 1.3880 - val_acc: 0.6053
Epoch 20/50
1487/1487 [==============================] - 0s - loss: 0.0109 - acc: 0.9987 - val_loss: 1.4080 - val_acc: 0.6026
Epoch 21/50
1487/1487 [==============================] - 0s - loss: 0.0066 - acc: 1.0000 - val_loss: 1.3899 - val_acc: 0.6053
Epoch 22/50
1487/1487 [==============================] - 0s - loss: 0.0087 - acc: 0.9993 - val_loss: 1.3894 - val_acc: 0.6026
Epoch 23/50
1487/1487 [==============================] - 0s - loss: 0.0080 - acc: 1.0000 - val_loss: 1.3714 - val_acc: 0.6079
Epoch 24/50
1487/1487 [==============================] - 0s - loss: 0.0068 - acc: 0.9993 - val_loss: 1.3462 - val_acc: 0.6184
Epoch 25/50
1487/1487 [==============================] - 0s - loss: 0.0119 - acc: 0.9980 - val_loss: 1.3577 - val_acc: 0.6105
Epoch 26/50
1487/1487 [==============================] - 0s - loss: 0.0086 - acc: 0.9993 - val_loss: 1.3517 - val_acc: 0.6132
Epoch 27/50
1487/1487 [==============================] - 0s - loss: 0.0073 - acc: 1.0000 - val_loss: 1.3217 - val_acc: 0.6237
Epoch 28/50
1487/1487 [==============================] - 0s - loss: 0.0072 - acc: 1.0000 - val_loss: 1.3050 - val_acc: 0.6316
Epoch 29/50
1487/1487 [==============================] - 0s - loss: 0.0090 - acc: 0.9980 - val_loss: 1.2976 - val_acc: 0.6342
Epoch 30/50
1487/1487 [==============================] - 0s - loss: 0.0096 - acc: 0.9993 - val_loss: 1.2972 - val_acc: 0.6421
Epoch 31/50
1487/1487 [==============================] - 0s - loss: 0.0066 - acc: 1.0000 - val_loss: 1.2882 - val_acc: 0.6421
Epoch 32/50
1487/1487 [==============================] - 0s - loss: 0.0057 - acc: 1.0000 - val_loss: 1.3006 - val_acc: 0.6342
Epoch 33/50
1487/1487 [==============================] - 0s - loss: 0.0069 - acc: 1.0000 - val_loss: 1.2982 - val_acc: 0.6342
Epoch 34/50
1487/1487 [==============================] - 0s - loss: 0.0066 - acc: 1.0000 - val_loss: 1.2998 - val_acc: 0.6368
Epoch 35/50
1487/1487 [==============================] - 0s - loss: 0.0077 - acc: 1.0000 - val_loss: 1.3163 - val_acc: 0.6316
Epoch 36/50
1487/1487 [==============================] - 0s - loss: 0.0065 - acc: 1.0000 - val_loss: 1.3434 - val_acc: 0.6211
Epoch 37/50
1487/1487 [==============================] - 0s - loss: 0.0062 - acc: 0.9987 - val_loss: 1.3274 - val_acc: 0.6316
Epoch 38/50
1487/1487 [==============================] - 0s - loss: 0.0065 - acc: 1.0000 - val_loss: 1.3381 - val_acc: 0.6211
Epoch 39/50
1487/1487 [==============================] - 0s - loss: 0.0089 - acc: 0.9993 - val_loss: 1.3370 - val_acc: 0.6184
Epoch 40/50
1487/1487 [==============================] - 0s - loss: 0.0069 - acc: 1.0000 - val_loss: 1.3277 - val_acc: 0.6158
Epoch 41/50
1487/1487 [==============================] - 0s - loss: 0.0070 - acc: 0.9993 - val_loss: 1.3389 - val_acc: 0.6158
Epoch 42/50
1487/1487 [==============================] - 0s - loss: 0.0066 - acc: 1.0000 - val_loss: 1.3625 - val_acc: 0.6053
Epoch 43/50
1487/1487 [==============================] - 0s - loss: 0.0068 - acc: 1.0000 - val_loss: 1.3532 - val_acc: 0.6053
Epoch 44/50
1487/1487 [==============================] - 0s - loss: 0.0078 - acc: 0.9987 - val_loss: 1.3679 - val_acc: 0.6105
Epoch 45/50
1487/1487 [==============================] - 0s - loss: 0.0077 - acc: 0.9993 - val_loss: 1.4037 - val_acc: 0.6026
Epoch 46/50
1487/1487 [==============================] - 0s - loss: 0.0063 - acc: 1.0000 - val_loss: 1.3935 - val_acc: 0.6053
Epoch 47/50
1487/1487 [==============================] - 0s - loss: 0.0060 - acc: 1.0000 - val_loss: 1.3667 - val_acc: 0.6158
Epoch 48/50
1487/1487 [==============================] - 0s - loss: 0.0066 - acc: 1.0000 - val_loss: 1.3466 - val_acc: 0.6132
Epoch 49/50
1487/1487 [==============================] - 0s - loss: 0.0064 - acc: 0.9993 - val_loss: 1.3348 - val_acc: 0.6184
Epoch 50/50
1487/1487 [==============================] - 0s - loss: 0.0056 - acc: 0.9993 - val_loss: 1.3448 - val_acc: 0.6211

aaargh… pulling my hair out…

1 Like

Hey there!
I think I have the same problem.
I’m using Wireless Sensor Data Mining (WISDM) dataset.
image
All I’m doing is splitting the data to 60% train 20% test and 20% for validation.
The way I’m doing that is choosing 60%,20%,20% of each user data randomly. This makes me to have a share of each user’s data for training, validation and the test(It will make more sense if you look at the pictures above).
I made 90x3 window of data to feed conv network. I reshape this window to 30x3x3 for conv overtime and feed for LSTM network

This is my network:


and the compile result is:

Layer (type) Output Shape Param #

input_11 (InputLayer) (None, 30, 3, 3, 1) 0


time_distributed_51 (TimeDis (None, 30, 3, 3, 16) 64


time_distributed_52 (TimeDis (None, 30, 3, 3, 16) 784


time_distributed_53 (TimeDis (None, 30, 3, 3, 16) 784


time_distributed_54 (TimeDis (None, 30, 3, 3, 16) 784


time_distributed_55 (TimeDis (None, 30, 144) 0


lstm_21 (LSTM) (None, 30, 32) 22656


lstm_22 (LSTM) (None, 30, 32) 8320


flatten_22 (Flatten) (None, 960) 0


dense_11 (Dense) (None, 6) 5766

Total params: 39,158
Trainable params: 39,158
Non-trainable params: 0


Just take a look at the training

Train on 14651 samples, validate on 4872 samples
Epoch 1/50
learning rate is 0.01
14651/14651 [===========] - 50s 3ms/step - loss: 0.8227 - acc: 0.7086 - val_loss: 0.5487 - val_acc: 0.7929
Epoch 2/50
learning rate is 0.005
14651/14651 [===========] - 42s 3ms/step - loss: 0.4950 - acc: 0.8221 - val_loss: 0.4714 - val_acc: 0.8323
Epoch 3/50
learning rate is 0.005
14651/14651 [==========] - 44s 3ms/step - loss: 0.4418 - acc: 0.8369 - val_loss: 0.5129 - val_acc: 0.7906
Epoch 4/50
learning rate is 0.0025
14651/14651 [=========] - 43s 3ms/step - loss: 0.3635 - acc: 0.8609 - val_loss: 0.4146 - val_acc: 0.8411
Epoch 5/50
learning rate is 0.0025
14651/14651 [=========] - 42s 3ms/step - loss: 0.3223 - acc: 0.8766 - val_loss: 0.3904 - val_acc: 0.8500
Epoch 6/50
learning rate is 0.00125
14651/14651 [========] - 40s 3ms/step - loss: 0.2769 - acc: 0.8962 - val_loss: 0.4019 - val_acc: 0.8475
Epoch 7/50
learning rate is 0.00125
14651/14651 [=======] - 42s 3ms/step - loss: 0.2619 - acc: 0.9053 - val_loss: 0.3912 - val_acc: 0.8537
Epoch 8/50
learning rate is 0.000625
14651/14651 [==========] - 43s 3ms/step - loss: 0.2312 - acc: 0.9155 - val_loss: 0.4164 - val_acc: 0.8432
Epoch 9/50
learning rate is 0.000625
14651/14651 [===========] - 42s 3ms/step - loss: 0.2172 - acc: 0.9227 - val_loss: 0.3999 - val_acc: 0.8495
Epoch 10/50
learning rate is 0.0003125
14651/14651 [===========] - 44s 3ms/step - loss: 0.2053 - acc: 0.9276 - val_loss: 0.4138 - val_acc: 0.8487
Epoch 11/50
learning rate is 0.0003125
14651/14651 [===========] - 42s 3ms/step - loss: 0.2000 - acc: 0.9304 - val_loss: 0.4098 - val_acc: 0.8493
Epoch 12/50
learning rate is 0.00015625
14651/14651 [============] - 43s 3ms/step - loss: 0.1927 - acc: 0.9339 - val_loss: 0.3552 - val_acc: 0.8709
Epoch 13/50
learning rate is 0.00015625
14651/14651 [==========] - 42s 3ms/step - loss: 0.1900 - acc: 0.9338 - val_loss: 0.4046 - val_acc: 0.8508
Epoch 14/50
learning rate is 7.8125e-05
14651/14651 [===========] - 42s 3ms/step - loss: 0.1861 - acc: 0.9363 - val_loss: 0.3971 - val_acc: 0.8541
Epoch 15/50
learning rate is 7.8125e-05
14651/14651 [===========] - 42s 3ms/step - loss: 0.1848 - acc: 0.9358 - val_loss: 0.3850 - val_acc: 0.8592
Epoch 16/50
learning rate is 3.90625e-05
14651/14651 [===========] - 38s 3ms/step - loss: 0.1829 - acc: 0.9369 - val_loss: 0.3830 - val_acc: 0.8610
Epoch 17/50
learning rate is 3.90625e-05
14651/14651 [==========] - 44s 3ms/step - loss: 0.1824 - acc: 0.9375 - val_loss: 0.3844 - val_acc: 0.8598
Epoch 18/50
learning rate is 1.953125e-05
14651/14651 [==============================] - 42s 3ms/step - loss: 0.1813 - acc: 0.9383 - val_loss: 0.3917 - val_acc: 0.8582
Epoch 19/50
learning rate is 1.953125e-05
14651/14651 [=========] - 43s 3ms/step - loss: 0.1809 - acc: 0.9379 - val_loss: 0.3917 - val_acc: 0.8569
Epoch 20/50
learning rate is 9.765625e-06
14651/14651 [=========] - 43s 3ms/step - loss: 0.1805 - acc: 0.9387 - val_loss: 0.3941 - val_acc: 0.8555
Epoch 21/50
learning rate is 9.765625e-06
14651/14651 [===================] - 42s 3ms/step - loss: 0.1803 - acc: 0.9388 - val_loss: 0.3896 - val_acc: 0.8584
Epoch 22/50
learning rate is 4.8828125e-06
14651/14651 [========] - 42s 3ms/step - loss: 0.1800 - acc: 0.9386 - val_loss: 0.3925 - val_acc: 0.8565
Epoch 23/50
learning rate is 4.8828125e-06
14651/14651 [============] - 42s 3ms/step - loss: 0.1799 - acc: 0.9388 - val_loss: 0.3956 - val_acc: 0.8555
Epoch 24/50
learning rate is 2.44140625e-06
14651/14651 [===========] - 42s 3ms/step - loss: 0.1798 - acc: 0.9391 - val_loss: 0.3935 - val_acc: 0.8557
Epoch 25/50
learning rate is 2.44140625e-06
14651/14651 [=========] - 43s 3ms/step - loss: 0.1797 - acc: 0.9388 - val_loss: 0.3934 - val_acc: 0.8559
Epoch 26/50
learning rate is 1.220703125e-06
14651/14651 [=========] - 39s 3ms/step - loss: 0.1796 - acc: 0.9390 - val_loss: 0.3941 - val_acc: 0.8555
Epoch 27/50
learning rate is 1.220703125e-06
14651/14651 [=========] - 42s 3ms/step - loss: 0.1796 - acc: 0.9390 - val_loss: 0.3938 - val_acc: 0.8559
Epoch 28/50
learning rate is 6.103515625e-07
14651/14651 [===========] - 42s 3ms/step - loss: 0.1796 - acc: 0.9390 - val_loss: 0.3940 - val_acc: 0.8557
Epoch 29/50
learning rate is 6.103515625e-07
14651/14651 [==========] - 42s 3ms/step - loss: 0.1796 - acc: 0.9391 - val_loss: 0.3939 - val_acc: 0.8557
Epoch 30/50
learning rate is 3.0517578125e-07
14651/14651 [==========] - 42s 3ms/step - loss: 0.1795 - acc: 0.9390 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 31/50
learning rate is 3.0517578125e-07
14651/14651 [=========] - 42s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3939 - val_acc: 0.8555
Epoch 32/50
learning rate is 1.52587890625e-07
14651/14651 [=========] - 43s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 33/50
learning rate is 1.52587890625e-07
14651/14651 [==========] - 43s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 34/50
learning rate is 7.62939453125e-08
14651/14651 [=========] - 42s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 35/50
learning rate is 7.62939453125e-08
14651/14651 [=======] - 42s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 36/50
learning rate is 3.814697265625e-08
14651/14651 [===========] - 41s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 37/50
learning rate is 3.814697265625e-08
14651/14651 [=========] - 42s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 38/50
learning rate is 1.9073486328125e-08
14651/14651 [==========] - 42s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 39/50
learning rate is 1.9073486328125e-08
14651/14651 [=======] - 44s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 40/50
learning rate is 9.5367431640625e-09
14651/14651 [========] - 43s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 41/50
learning rate is 9.5367431640625e-09
14651/14651 [========] - 43s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 42/50
learning rate is 4.76837158203125e-09
14651/14651 [========] - 42s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 43/50
learning rate is 4.76837158203125e-09
14651/14651 [=========] - 42s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 44/50
learning rate is 2.384185791015625e-09
14651/14651 [=========] - 42s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 45/50
learning rate is 2.384185791015625e-09
14651/14651 [========] - 41s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 46/50
learning rate is 1.1920928955078125e-09
14651/14651 [========] - 42s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 47/50
learning rate is 1.1920928955078125e-09
14651/14651 [===========] - 44s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 48/50
learning rate is 5.960464477539063e-10
14651/14651 [=======] - 43s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 49/50
learning rate is 5.960464477539063e-10
14651/14651 [========] - 42s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555
Epoch 50/50
learning rate is 2.9802322387695313e-10
14651/14651 [=========] - 42s 3ms/step - loss: 0.1795 - acc: 0.9391 - val_loss: 0.3940 - val_acc: 0.8555

I think there is something wring with Val_loss . It’s never decrease.
Is it Overfitting?
Am I wrong with splitting data?