Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

There currently isn't a way to filter docs alongside a KNN query, and the dimension support is limited to 1024 (a Lucene limitation) and OpenAI embeddings are 1536 dimensions - also indexing performance is not comparable. Wishing this changes, as they're a good stack for the reasons you state


True though I do think 2k dims is coming it 8.8


Are they forking Lucene or somehow getting the Lucene devs to increase that limit? Because this PR has been open for over a year now: https://github.com/apache/lucene/issues/11507


No - they just did something in Elasticsearch to make their own FieldType https://github.com/elastic/elasticsearch/pull/95257




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: