Time and time again, this blog does not fail to impress. I especially liked her piece on Diffusion models from earlier this year; It was a very nice, simplified version of a complex topic that named some of the most important papers and contributions over the last few years. All the while, the blog wasn't overly simplified like other blogs seem to do all to often (not providing key derivations of formulas, discussing topics at a glance, reading more like a PR piece than an actual informational blog.)
Agree. She had a very informative tutorial session yesterday on self-supervised learning at NeurIPS-2021. While I don't think the recording is publicly available [1], the slides are [2].
Interesting, I also initially read it with a negative impression, e.g. "this blog constantly fails to impress me", even though that's the opposite of what the sentence says.
Not to derail the topic, but anyone have any insight on why that might me? Pretty sure it's fine, idiomatic English. Am I just primed to expect negative criticism in HN comments? :/
I had to re-read that line as well. I have huge respect for Lillian's work so this line just didn't make any sense to me, forcing me to read it carefully!
As someone who worked with these techniques a lot in the past, I can say that SSL definitely makes sense in theory, but in practice, the gain doesn't pay off the complexity, except in rare cases w/ pseudo-labelling for example, which is very simple. Usually you tune a lot of hyperparams and tricks to make it work and the gain are usually minimal if you have a reasonable amount of labeled data.
It is true when your task if relatively simple and you can create labels to cover most of the data cases. For long tail problems like vision or NLP you can't label everything and semi-supervised learning helps a lot.
A simple way to do SSL with not enough data is Lipschitz learning. It avoids the necessity for pseudo-labeling and computes an absolutely minimizing Lipschitz extension of the labels instead.
I think the part of this that surprised me the most was learning that Self-Teaching actually... works? Not entirely sure why, but my first instinct when I was first getting into AI was that training a model on its own predictions would just... not provide any benefit for some reason. Well, today I learned otherwise! I love being proven wrong about stuff like this.
>Semi-supervised learning seems currently under-appreciated, especially in medicine.
In medicine it would be appreciated more if it were more effective. Many times the right answer to "I don't have enough data to do X" is: don't do X.
I'm not entirely pessimistic on this by the way, I think principled semi-supervised approaches are likely to work much better than some of the hail mary's you see people try in the space with transfer learning and generative models etc. But it's still hard, and often it just isn't going to work with the kind of practical numbers some people want to be able to work with in medicine.
