Ah so it is! I guess my point was that even this simple function is enough, you don't need any 'magic' beyond that. In particular, this continuous function can be used (infinitely) to represent discontinuous functions - the step function being the usual example. That is the more interesting and relevant mathematical fact I think.
Well, yes. The 'magic' is the nonlinearity. It's because the composition of linear (actually also affine) functions is still just linear (affine). You don't get any additional power by combining many of them - which is also the reason why linear functions are so well understood and easy to work with.
You give sprinkle in a tiny nonlinearity (e.g. x^2 instead of x) and suddenly you can get infinite complexity by weighted composition - which is also the reason why we're so helpless with nonlinear functions and reach for linear approximations immediately (cf. gradient descent).
>It's because the composition of linear (actually also affine) functions is still just linear (affine).
Except you can "compose" affine functions using Horner's schema to get any polynomial... No need to sprinkle tiny non linearities. It's a buffet. Grab as much as you need.
