InceptionResNetV2 validation accuracy issue

I am trying to implement InceptionResNetV2 using the code that was previously available on fastai for ResNet50. The code looks like:

train_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)
test_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)

train_generator = train_datagen.flow_from_directory(train_data_dir,batch_size=batch_size)
validation_generator = test_datagen.flow_from_directory(validation_data_dir, shuffle=False, batch_size=batch_size)

base_model = InceptionResNetV2(weights=‘imagenet’, include_top=False)
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(2048, activation=‘relu’)(x)
predictions = Dense(4, activation=‘sigmoid’)(x)

model = Model(inputs=base_model.input, outputs=predictions)
for layer in base_model.layers: layer.trainable = False
model.compile(optimizer=‘SGD’, loss=‘categorical_crossentropy’, metrics=[‘accuracy’])

%%time
model.fit_generator(train_generator, train_generator.n // batch_size, epochs=1, workers=4,
validation_data=validation_generator)

Epoch 1/1
2158/2158 [==============================] - 5267s 2s/step - loss: 0.7547 - acc: 0.6323 - val_loss: nan - val_acc: 0.2491

Training accuracy increases while validation accuracy is either nan or very low. What could be the problem ?