1 million neurons and curve fitting

In a ReLU network each neuron can only create one bend in a curve.
If you have a conventional ReLU network with 1 million neurons in a layer, each neuron must have 1 million weight parameters linking back to the prior layer.
That is 1 million parameters to get one bend in a curve. That is intuitively a bit excessive. How much is a good number then? 1000 parameters per curve bend, 100? Is the optimal actually right at the other extreme at 1 or 2 parameters per curve bend, if you can arrange such a thing?
Backpropagation version:

1 Like

I think I get something of what you are saying. But if the prior layer has N neurons, won’t there be N (not 1 million) weight parameters connecting each neuron to the previous layer?

Yes, sure. For simplicity I assumed uniformity of layer design.
I used 1 million as an extreme example to highlight the issue.
Maybe Ankit Patel is starting to inch toward some understanding in this video: