Searchkick VS zombodb

I run a large scale production application that does something along these lines. If the data needs to be close to real-time, I'd say use `searchkick` + Elasticsearch, and use `searchkick`'s async feature to "stream" the data from your table to the ES index. Your dashboard will then just query from the ES index via searchkick.

You're right, that's actually what we implemented, application-level hooks, but they needed development and maintenance effort that come for free with the adapter we're using for OpenSearch integration, which also comes with welcome features: synonyms, partial matches, and many others.

Spoiler, the adapter is Searchkick: https://github.com/ankane/searchkick

Searchkick

Normally for Rails applications you would use a gem like searchkick since it greatly reduces the initial Elasticsearch complexity.

We lean heavily on Elasticsearch at CompanyCam. One of it's primary use cases is serving our highly filterable project feed. It is incredibly fast, even when you apply multiple filters to your query and are searching a largish data set. Our primary interface for interacting with Elasticsearch is using the Searchkick gem. Searchkick is a powerhouse and provides so many features out of the box. One place where we bump up against the edges is when trying to reindex a large collection.

Convinced? Ok read on and I’ll show you what switching from Elasticsearch to Meilisearch looked like for a real production app — ScribeHub. We also moved from Ankane’s excellent Searchkick gem to the first party meilisearch-rails gem and I’ll show you the changes there as well.

If you want to use a look-aside pattern (like you might have used with Searchkick + Elasticsearch), you should check out ActiveStash: https://github.com/cipherstash/activestash

Searchkick learns what your users are looking for. As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users. BONUS: it's written and supported by "ankane" who has flawless reputation amongst the Ruby community.

And lots of interesting extensions use it, like

https://github.com/tembo-io/pgmq

https://github.com/zombodb/zombodb

https://github.com/supabase/pg_jsonschema

Point 2 is generally solvable via engineering effort and careful dedicated code. From the existing tools, PGSync is an open source project that aims to specifically solve this problem. ZomboDB is an interesting Postgres extension that tackles point 2 (and I think partially point 3), by controlling and querying Elasticsearch through Postgres. I haven't yet tried either of these two projects, so I can't comment on their trade-offs, but I wanted to mention them.

Curious, did you try zombodb? [https://www.zombodb.com/]

What about zombodb, do you think that quickwit has all the necessary APIs?

No. Haha. Was just the right name for https://github.com/zombodb/zombodb at the time. Software where the only limit is yourself!

> The hardest part of building any search engine is keeping the index up-to-date with changes made to the underlying data store.

This deserves mention, as it solves that problem: https://github.com/zombodb/zombodb

From the README:

> ZomboDB brings powerful text-search and analytics features to Postgres by using Elasticsearch as an index type. Its comprehensive query language and SQL functions enable new and creative ways to query your relational data.

> From a technical perspective, ZomboDB is a 100% native Postgres extension that implements Postgres' Index Access Method API. As a native Postgres index type, ZomboDB allows you to CREATE INDEX ... USING zombodb on your existing Postgres tables. At that point, ZomboDB takes over and fully manages the remote Elasticsearch index and guarantees transactionally-correct text-search query results.

I find other things also hard in search engines: dealing with the plethora of human languages and all the requirements we may have to processing them. A mature solution like ES therefor is almost a must in the more demanding cases.

You can read more about it on our GitHub repo, but basically it brings most of the power of elasticsearch’s searching and analytics abilities straight into Postgres.