Model starts to lose accuracy mid training

I am training a model with all train data from the Cats vs Dogs dataset.
I am using 1090 validation imges and 22010 training images.
With the sample images to model worked fine, I got a score of around 0.2 and ended around the middle of the competition.
However, when I input all the images, the model starts out great but at some point starts to lose accuracy, sometimes to even 0.5
What can be causing this? I thought I had messed up something in the training data, but I found no problems with that
Thank you in advance for your help

Hmm I also just realized, when I predict with the test dataset, all my outputs are 1
This didn’t happen with the sample dataset