Weaviate VS bbolt

In this tutorial, I introduce Weaviate, an open-source vector database, with the thenlper/gte-base embedding model from Alibaba, through Hugging Face's transformers library.

This will be solved in Weaviate https://github.com/weaviate/weaviate/issues/2424

Link to GitHub -->

Hey @rvrs, I work on Weaviate and we are doing some improvements around increasing write throughput:

1. gRPC. Using gRPC to write vectors has had a really nice performance boost. It is released in Weaviate core but here is still some work on do on the clients. Feel free to get in contact if you would like to try it out.

2. Parameter tuning. lowering `efConstruction` can speed up imports.

3. We are also working on async indexing https://github.com/weaviate/weaviate/issues/3463 which will further speed things up.

In comparison with pgvector, Weaviate has more flexible query options such as hybrid search and quantization to save memory on larger datasets.

Highly opinionated as I'm working for Weaviate, so take my comment with a large portion of salt.

My highly opinionated view is that for Elastic, they're not really open source and the dependency on Java of the Lucene ecosystem is a big disadvantage, so as you already said, speed, they're getting better at this, but if you need to scale, this problem scales with you.

So if you already have ELK stack and don't need to scale, sure go for it otherwise, Weaviate offers real open source, so use it for free on your own infrastructure https://github.com/weaviate/weaviate

If you do not wish to store your private data on pinecone you can use open source alternatives like Weaviate where you can spin up your own instance. Other option could be to use Agents. You'll need to find sutaible agent for your database which will allow LLMs to directly query data from your private database.

I tried cromadb but had terrible performance and could not pin down the cause (likely a problem on my end). Weaviate was easy to setup and had excellent performance, this is probably what I will use in the future. Next on my list is txtinstruct, to finetune a model with data that does not change and using a vector db for everything else seems promising.

A Locker is a store on your file system (built on top of the amazing bbolt).

- bbolt for storage on disk. In order to get the smallest db file size possible make sure you insert the keys in order and set:

I'm wanting to store some wasm modules (as []byte) in BoltDB. Right now the modules are <1MB, but eventually, they could be 10-50MB in size. Is this going to reduce the performance of BoltDB all around, if the size of a value is this large? If it makes a difference, I'm using the Storm toolkit for querying.

bbolt - An embedded key/value database for Go.

No transactions, no consistency guarantees, no benchmarks, global locks in the storage implementation, a collection is copied in its entirety on every insertion to it...I realize it's not for the same use case as MySQL or MongoDB, but a more obvious comparison here is e.g. https://github.com/etcd-io/bbolt. So why should someone use this over bbolt?

use go-sqlite3 to work with sqlite3 is one choice.

https://github.com/etcd-io/bbolt is another pure go option.

cznic seems like an alternative to bbolt. nice to have some options.

This is the PR for the freelist improvement from that alibaba article: https://github.com/etcd-io/bbolt/pull/141

There is also an option to have a custom domain like go.etcd.io/bbolt

Remove sqlite or replace with bbolt

I would used some embedded kv store like go-leveldb or bolt. Key is BigEnding timestamp + optional tail to allow duplicate timestamps.