Specifically to the error:
“Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.”
What I understood from Thomas is that pytorch compares the graph from the previous batch with the graph generated by the current batch - if the new graph lacks a node that was in the old graph that’s when you get this error.
That’s why you must repeat calculations that place all the variables that were in the graph on the very first batch, and they can’t be skipped.
I don’t know enough pytorch yet to explain why dynamically skipping a node is considered to be an error. So this is only a circumstantial explanation.
@t-v, please kindly correct me if I my explanation is incorrect or incomplete.