Thank you!
Here are some passages from the book you pointed to:
While our neural network gives impressive performance, that performance is somewhat mysterious. The weights and biases in the network were discovered automatically. And that means we don’t immediately have an explanation of how the network does what it does.
And while that passage is followed by a lot more details, it just descriptive and does not quite add up to “an explanation”.
Admittedly, this subsequent passage is more than mere description:
It does this through a series of many layers, with early layers answering very simple and specific questions about the input image, and later layers building up a hierarchy of ever more complex and abstract concepts.
But that only explains how a fully trained network functions and it does even attempt to explain the processes (ie the training phase) that took us from the untrained network to the trained network.
Later there is a direct reference to the training (“learning”) phase processes,
… use learning algorithms so that the network can automatically learn the weights and biases - and thus, the hierarchy of concepts - from training data.
There is that word , “automatically” again… Very descriptive but very decidedly not an explanation of the training processes. It this context it appears that “automatically” is a synonym for “mysteriously”.
Is there really so little – any? – theoretical progress being made that sheds any light on the effectiveness of deep learning techniques?
I realize this course is focused on “practical”, but it’s good to be up front about the question, “Is there – or is there not – a generally accepted theory or explanation behind all of this?”
It appears that the correct take away, “There is still no generally accepted explanation for it,” or, “It remains a mystery”?
Please contribute pointers to “the latest” on this subject if you are aware of anything. Thanks.
