Used to work on Google Search, used ES extensively for a startup I founded (which was sort of quasi-search...it was built around feed ranking, where the query is constant and a stream of documents is constantly coming in), and have also used Postgres extensively in other companies.
The big problem with all the off-the-shelf search solutions (RDBMS full-text search, ES, Algolia) is that search ranking is a complicated and subtle problem, and frequently depends on signals that are not in the document itself. Google's big insight is that how other people talk about a website is more important than how the website talks about itself, and its ranking algorithm weights accordingly.
ES has the basic building blocks to construct such a ranking algorithm. In terms of fundamental infrastructure I found ES to be just as good as Google, and better in some ways. But its out-of-the-box ranking function sucks. Expect to put a domain expert just on search ranking and evaluation to get decent results, and they're going to have to delve pretty deeply into advanced features of ES to get there.
AFAICT Postgres search only lets you tweak the ranking algorithm by assigning different weights to fields, assuming that the final document score is a linear combination of individual fields. This is usually not what you want - it's pretty common to have non-linear terms from different signals.
So let’s say that you are building a search engine for performance car parts. There are going to be a bunch of technical terms you use there that are not necessarily going to stand out in the document itself but you know them to be important. For example, the amount of boost pressure a turbo can provide or the number of pistons in a brake caliper. Is there some structured way to specify the grammar which is used for such terms and treat those as important such that when a user puts in “19psi turbo” that they don’t get a bunch of results for just “turbo” which isn’t exactly what they want?
This subproblem is called "bigram detection", or more generally "n-gram detection" for phrases of > 2 words. I'm not going to give away Google's algorithms, but you should be able to look up public academic research with that keyword.
When I implemented search for my app, I built a hierarchy of the different attributes weighted by levels and scored search phrases for how closely they matched. This is for mobile phones. For example, searching for "Silver iphone 11 pro" would give a higher ranking to all iphone 11 pros that are Silver, then comes differently coloured 11 Pros, followed by 11 Pro Max and 11s, lastly all silver coloured phones.
>search ranking is a complicated and subtle problem
Absolutely !
I think never before the saying "the devil is in the details"... is more appropriate than here.
Sure most users agree on the extreme's the REALLY bad search result (putting in 'apple' getting out LCD TV's [maybe the tv's are in parent-category 'Electronics' and you have category and popularity boost to high' ?]) and the REALLY good (results that you expected)
Search Results are commonly evaluated with Recall(did ALL the documents that are relevant got returned)
vs Precision (how many of the results are 'correct')
But that is "one" of many-many metrics.
The biggest issues are non-tech ppl (like your boss or manager) walking in and "discussing" his/her pet-peeve-search query, cause in his mind when he put in APPLE iPhone we should be returning ONLY apple-iphones and NOT apple-iPhone accessories) or maybe we should be returning ONLY the "latest iphone" not the model from 2 generations back
I've commented this before, you can usually only shoot for an "average amount of happiness" (sounds like Arthur Schopenhauer ? :P) for most users. Never "ok ppl, search works perfect" for everyone one now
As to the "practical matters", we found that building a "search-test-suite" where you put in the "manager's pet-peeve" as well as any angry-emails about search queries, and whenever do you search-tuning it's easy to see any-query-regression oh and of course this needs to be an automated search-test-suite.
Nice insightful comment. Can you give a bit additional insight related to when it makes sense to use ES vs Postgres? My takeaway from your comment is that unless your product depends on search as a central component that inbuilt search is good enough. Is that an incorrect takeaway?
That's probably a decent takeaway. If full-text search is a "nice to have" (a bonus feature for advanced users, for example) you can probably rely on built-in Postgres search. If search is a fundamental way that users interact with your product, you want to spend the time and money to get it right, and that probably means ElasticSearch and a custom ranking function.
> Google's big insight is that how other people talk about a website is more important than how the website talks about itself, and its ranking algorithm weights accordingly.
I'm having trouble believing that seeing how top results on opinionated keywords are all SEO spam of websites no one visits by themselves.
You don't remember what a game-changer Google was back in 2000 or so. Its results are pretty awful now, probably because of Goodhart's Law, but they were such a huge improvement over Jeeves et al..
An interesting exercise that most reasonably competent programmers can do now is to setup their own websearch engine with Common Crawl, Elastic MapReduce, JSoup or Gumbo or equivalent HTML parser, and ElasticSearch. It costs on the order of high-hundreds to low-thousands of $$$ to process the ~2B webpages in the Common Crawl corpus (about half the size of the first tier of Google's index) and stand up an ElasticSearch cluster with it for a few days.
You too can build your very own search engine.
Unfortunately, the result quality is roughly what AltaVista was like in 1995. That is why people keep going back to Google.
I remember when Google came into being and how amazed I was. It was quite something totally different and I switched and never looked back for a lot of years.
In recent years however I tend to massively agree with your sentiment and experience. Every day I do not find the things that are really helpful on page 1 - 3, sadly.
There is an incredibly large difference between effective scoring for data that is being gamed and data that is not being gamed. Most people's full text data isn't being gamed quite like web search SEO.
There is also an incredibly large difference between effective scoring for data that has deep relationships and data that does not. Most people's full text data doesn't have deep relationships like web pages do with inbound links.
Might you or anyone else have some recommendations for books or other resource on large scale search architecture that you think are are worthy reads on the subject?
I would also be interested in hearing if you or anyone else might ave any similar resources you could recommend on the subject of "search ranking"?
"Managing gigabytes" is one of the standard introductory textbooks. It's quite dated now (published 1994, before Google was started, and hence why it says "gigabytes" rather than "exabytes"), but a lot of the fundamental concepts are the same. At least it'll give you a vocabulary and conceptual framework for thinking about search.
I’ve never really had a good idea of what Percolator is. I understand it to be a “streaming” indexer that replaced a “batch” indexer based on MapReduce. ES allows indexing individual documents (or small batches) vs. rebuilding the entire index; is that the connection you’re making?
The big problem with all the off-the-shelf search solutions (RDBMS full-text search, ES, Algolia) is that search ranking is a complicated and subtle problem, and frequently depends on signals that are not in the document itself. Google's big insight is that how other people talk about a website is more important than how the website talks about itself, and its ranking algorithm weights accordingly.
ES has the basic building blocks to construct such a ranking algorithm. In terms of fundamental infrastructure I found ES to be just as good as Google, and better in some ways. But its out-of-the-box ranking function sucks. Expect to put a domain expert just on search ranking and evaluation to get decent results, and they're going to have to delve pretty deeply into advanced features of ES to get there.
AFAICT Postgres search only lets you tweak the ranking algorithm by assigning different weights to fields, assuming that the final document score is a linear combination of individual fields. This is usually not what you want - it's pretty common to have non-linear terms from different signals.