Memory Error with Resnext50 model

Hi @everyone,

Upon executing this line:

I am getting this error:

I am using a computer with 12GB Nvidia Titan Xp.

Upon doing Nvidia-smi, I am getting this:

image

Should I reset my GPU or is there any other good option?

Just keep reducing the batch size until it runs. Jeremy also recommends restarting the kernel after a cuda failure.

1 Like

I think the problem is not the batch size I used. The problem is my GPU memory is being used without running many processes. I need to free my space first. I have used a lot of PyTorch cuda processes and they have not freed the memory I guess. I might be wrong but I think we need to free memory after a process has used it in PyTorch.

I am getting this multithreading error too.

@jeremy if you can help here. Thanks!

That’s a big model, so you’ll need a small batchsize. Try bs=24.

1 Like

I am getting memory errors when I try changing batch sizes. The same error comes up whenever I play with bs (tried higher, lower, all resnets). Despite the error, it still seems to complete. Am I setting the batch size wrong?

A snippet of code from planets.

i have a 8GB GTX 1080 - needed to decrease the batch size to 16 to make it work

That’s the thing. It appears to work (currently at bs = 10), it just tosses those two errors initially. As I go up into larger size images it does not trigger again.

Update: Looks like this is just a warning and I can ignore it

@mindtrinket that’s not showing any memory error. It’s just a warning about an unrelated issue - there’s another thread about it if you’re interested in learning more