Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

An interesting exercise that most reasonably competent programmers can do now is to setup their own websearch engine with Common Crawl, Elastic MapReduce, JSoup or Gumbo or equivalent HTML parser, and ElasticSearch. It costs on the order of high-hundreds to low-thousands of $$$ to process the ~2B webpages in the Common Crawl corpus (about half the size of the first tier of Google's index) and stand up an ElasticSearch cluster with it for a few days.

You too can build your very own search engine. Unfortunately, the result quality is roughly what AltaVista was like in 1995. That is why people keep going back to Google.



Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: