in test time , Is “torch.no_grad()” automatically called when we call learn.model.eval(); ?
Because in below code , I can’t see any change in run time by using “with torch.no_grad()” or not
learn.model.eval();
import time
start = time.process_time()
with torch.no_grad():
probs_kasre = learn.model(t_x)
print("\n total time is :\t" , time.process_time() - start)
No. See here for the difference in what each does:
thanks for your reply Zachary
I have seen this link , I know the differences
maybe I asked that in a wrong way
I mean , does fast.ai package automatically call torch.no_grad() in val time ?
or there is any other reason that i can’t see any decrease in run time by using
“with torch.no_grad(): …”
All I see is PyTorch code, so that may be where the confusion lies 
learn.model.eval is the same as model.eval from torch because that’s all learn.model is. Does that help some?
There’s nothing inherently special about fastai’s learn.model
helped
Do you always see impressive speed up when you use “with torch.no_grad(): …” (in comparison with not using that ) ?
How much decrease in running time ?
and I think it’s something independent from architecture and Data and… , am i right ?
I’m not sure about the running time, but if I’ve just setup the model and throw some dummy data into it to ensure the output shapes look correct, not using torch.no_grad() stores the gradients, and this takes up GPU memory, overloads my GPU and leads to an out of memory error that can’t be fixed with torch.cuda.empty_cache(), so you gotta restart your notebook kernel. All of this is when the model is not in eval mode.
And also, re speed up, bigger the architecture, more gradients it needs to store, which can lead to a longer runtime.
Generally if you want to push efficiency from a PyTorch model you’d want to push that trained model to torch script or ONNX (if possible)
Is ONNX always a better choice? For inference on CPU, yes, there’s a massive difference, but assuming you’re running inference on a GPU, is this accurate:
Torch Script > PyTorch > ONNX
Last I checked, onnxruntime isn’t implemented for GPU inference in Python yet, just C++
It is supported now
See my fastinference ONNX port: https://muellerzr.github.io/fastinference/onnx/
(Not as much as I want on those docs, working on it this weekend)
Here’s some examples, top is ONNX, bottom is my improved fastai:
For a better example, top is ONNX:
