Is there a way (workaround) for CUDA error: out of memory when running preds = learn.TTA(is_test=True) ?
i restarted kernel couple of times but still get the error. the process finishes first pass gets to 100% but then it shows the error.
I also tried to create another data bunch with smaller size images (224->112-56) also smaller batch size (64->32->16) just for the prediction part but i got the same error… i guess the test images are not resized?
I am using n1-highmem-8 in GCP with P4 GPU.
the training took long time, 1 epoch around 5 hrs but finished OK (no error here)
prediction is fast, as there is a lot less images, but fails with Cuda out of memory error
any chance that by chaning any of the arguments would help?
learn.TTA(beta:float=0.4, scale:float=1.35, is_test:bool=False, with_loss:bool=False)
i moved the prediction to crestle they have K80 GPU and it run fine. but i still would be good to know if there is a way to fix out of memory error on P4 GPU (GCP) without retraining the model with lower res picutes, as training was looong…
from what i can see on nvidia site:
P4 is 8G
K80 is 24 GB, most likely at crestle we have access to 1/2 of memory that is 12 GB (thanks @skottapa)
I’m not sure exactly, but I use Colab and these type of error pops up when I have too many prototypes of my model, in the same notebook. This is to do with your GPU memory being full