Okay I have managed to fix the issue.
After changing the dtype to np.float32 from np.float64. I noticed @ Mikhail Gurevich had worked on the same dataset. I looked at how he processed the face points coords.
If anyone is reading this and having the same problem and working on the same dataset: Read the thread below
ImagePoints problem with Kaggle Cat Dataset (Facial features recognition)