I’m trying to make sense of the colorful dimension.
I understand that weights can “explode” and diverge.
However then they get reset back to zero.
Why is that? Is there a rule somewhere that says that if a weight is NaN then set it to 0?
I’m trying to make sense of the colorful dimension.
I understand that weights can “explode” and diverge.
However then they get reset back to zero.
Why is that? Is there a rule somewhere that says that if a weight is NaN then set it to 0?