>> They used a drumstick because it provided a consistent way to produce a sound.
More to the point, that bit.
The article makes a big todo about how humans use sound to learn about their environment and so on, but imagine if we needed to get a drumstick to make sounds consistent enough to learn to recognise them.
Supervised and unsupervised learning is not the real challenge. The real challenge is to get to the point where algorithms can build a model without the benefit of an insanely expensive data pre-processing pipeline. Deep learning's big promise is exactly that, but it's not always delivered (for instance, there's a paper by Hinton and I forget who else, where they report that training LSTM RNNs on raw characters does not give best performance) (so we're stuck with tokenisation and the implicit assumptions they impose on your data- my corollary).
"The real challenge is to get to the point where algorithms can build a model without the benefit of an insanely expensive data pre-processing pipeline."
Arguably, that is equivalent to saying the real problem is still unsupervised/semi-supervised learning. IE, being able to just throw a bunch of raw data and maybe just a bit of hand configuration at an algorithm and have it do complicated things for you. The success of Deep learning is to scale to tons of data and build really complicated models, but as it is used today that data is still hand labeled for supervised learning in an insanely expensive data pre-processing step. Good unsupervised or semi-supervised learning could hopefully let us get out of this, but I don't think anyone really knows how to get there yet. Co-training is an older example of semi-supervised learning, and more recently there were Ladder Networks, but I don't think any algorithm has been shown to work really well and become the norm in the way LSTM RNNs or CNNs have.
>> Arguably, that is equivalent to saying the real problem is still unsupervised/semi-supervised learning.
I don't agree because data pre-processing and labeling are two distinct parts of the pipeline and you can totally have one without the other.
And there's more to it than that. Currently we have to provide the context for an algorithm to learn. We do this by selecting training examples. Whether these examples are labelled or not, they are only a small part of the world we wish the algorithm to learn about.
You don't even have to go as far as the wider physical world to see this in action. In any training context, if your training set is missing a category of entities, Y, then your algorithm will never model Y. It doesn't make any difference if your model is trained in a supervised manner or not. What matters is that there is a part of the world that it hasn't seen.
I guess you can say that humans don't have a way to learn this way either, but human learning has a big advantage: we need very little data and very little training to incorporate new knowledge and our context of a world is very broad to begin with. It's at once broad, specialised, robust and flexible. We're a bit scary if you think about it.
Which leads me to believe that the limitation of our machine learning algorithms is not in the labeling, or even in the data pre-processing but in some fundamental aspect of building a context from examples only. There's something missing and it's not something we know about (hah!). The missing part means that you can learn from examples until the heat death of the universe and there will still be an infinity of things you don't know anything about- and that are potentially part of your immediate environment.
Obviously, removing the need for pre-processing will make things much cheaper and there will be progress, ditto removing the need for supervision. But it won't get us anywhere nearer human learning, despite people's best wishes, because we're missing a part of the puzzle that's a whole other ball game.
(and which I obviously don't claim to have any idea about)
We are making progress with embeddings. Today we can embed everything2vec and then apply RNN's on top to do reasoning. Another field of interest is building large ontologies which represent factual data and relationships between concepts at encyclopedic scale.
I'd say the hottest area now is not unsupervised, but reinforcement learning, which is in the middle ground between supervised and unsupervised.
More to the point, that bit.
The article makes a big todo about how humans use sound to learn about their environment and so on, but imagine if we needed to get a drumstick to make sounds consistent enough to learn to recognise them.
Supervised and unsupervised learning is not the real challenge. The real challenge is to get to the point where algorithms can build a model without the benefit of an insanely expensive data pre-processing pipeline. Deep learning's big promise is exactly that, but it's not always delivered (for instance, there's a paper by Hinton and I forget who else, where they report that training LSTM RNNs on raw characters does not give best performance) (so we're stuck with tokenisation and the implicit assumptions they impose on your data- my corollary).
Also, there are ways to avoid the expenses of hand-labelling, for instance co-training: https://en.wikipedia.org/wiki/Co-training