sqlite_protobuf VS dflat

SQLite's virtual table API (https://www.sqlite.org/vtab.html) makes it possible to access other data structures through the query engine. You don't need to know much if anything about how the database engine executes queries, you only need to implement the callbacks it needs to do its job. A few years ago I wrote an extension to let me search through serialized Protobufs which were stored as blobs in a regular database.

https://github.com/rgov/sqlite_protobuf

Some time ago I joined a project that was recording logs in Protobuf format. I demonstrated that we could log the serialized Protobuf structures to a SQLite database and then use a plugin to query across the message fields: https://github.com/rgov/sqlite_protobuf

Learning how to use the SQLite extension API was a little mind-expanding but it worked like a charm.

This is really, IMHO (as someone implements things on top of SQLite too https://dflat.io) pushes SQLite too far as the implementation of cross-db transactions have some big issues: https://www.sqlite.org/limits.html (the number of attached databases cannot exceed 10 or 125 (if you compile your own)) https://www.sqlite.org/wal.html (in WAL mode, there is no transactional guarantee for cross database transactions (atomic per database, but not cross database))

Dflat: SQLite FlatBuffers\ (4 comments)

Flatbuffers would be better as it doesn't need to deserialize at all (I think protobuf supports to partially deserialization nowadays?). I did something similar with https://dflat.io/ too!