> I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results for some queries since nobody is fighting about the position until it's widely known.
There is a solution for all this mess and I'm blocking HN and a few different domains until I implement at least the first step after which I can share it here.
That’s not going to happen before Cloudflare is dethroned. See this recent thread for some perspective: https://news.ycombinator.com/item?id=27153603
And even if there’s no Cloudflare, large sites that people want to search will always find ways to block bad bots.
The only thing I can think of that might work is using crowd-sourced data, with all the problems that come with crowdsourcing.