-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
I'm using datasette[0] and sqlite-utils[1] on replit[2] for a project, which unfortunately I can't share for the time being. I've been very productive, and the low learning curve on this setup makes it easy for others to use and modify what I've done.
I expect to have time to make a demo datasette project available on replit (and github) 'soon.'
I'm so grateful for @simonw 's work on datasette and sqlite! These packages are beautifully designed and documented, and they work so well! Thank you very much @simonw!
[0] https://datasette.io/
I use it in a network video recorder. https://github.com/scottlamb/moonfire-nvr/blob/master/design...
Data in a flat file would be much less efficient, and a SQL server (eg PostgreSQL) would be more for the administrator to deal with. SQLite's a nice sweet spot.
I've thought about using a key/value database (either a BTree one like SQLite uses under the hood or a LSM one like RocksDB). It'd be faster and less total code. But SQLite's too convenient to give up. The SQL interface is nice for debugging in particular. And it's already plenty fast enough, so my time's better spent adding some glaring missing features and better UI.
I'm using magnetico (https://github.com/boramalper/magnetico), a selfhosted bittorrent DHT scraper. After a few months of running I now have a little bit under 1M magnets in a ~3G database. SQLite is useful not just because I can query it with the full power of a relational database, but because I'm using its FTS capability for searching.
If I had to install a typical database or some search engine I would never have used it. It is more than enough for what I'm using it for.
SQLite is so ubiquitous you'd be hard pressed to find an app or service that doesn't use it.
Look through the licenses for the software included as part of your phone's OS. You'll find SQLite in there.
Search GitHub for sqlite, there are several projects with thousands of stars that use SQLite. Here's one: https://github.com/Tencent/wcdb
It's a really cool feature of sqlite. I use it in a prover benchmark tool [1] where each run produces a new sqlite file that I can rsync elsewhere. The tool can then list these files and connect to multiple files to do joins across them (to compare results from different runs). It's very convenient.
[1] https://github.com/sneeuwballen/benchpress
I think this is an excellent architecture for powerful, respectful, hosted applications. I’ve been thinking about a few extensions of this idea:
First, use advances in privacy technology to create a service-wide data warehouse that has enough information to help you make good decisions without exposing any specific user’s data. Done properly, users will benefit from your improved decision-making without giving up their personal data. Differential Privacy can do this.
Second, give users the opportunity to download their own little database in native format (e.g. SQLite) This is the ultimate in data portability. I think Dolt [0] might be good for this, because its git-like approach gives you push/pull syncing as well as diffing. That would make it easy for users to keep a local copy of the data up to date.
Third, you can start to support self-hosting and perhaps even open-source the primary user-facing application. The hosted service sells convenience and features enabled by the privacy-respecting data warehouse.
The big questions, of course, are many:
- Would users pay for this?
- Does increased development cost and reduced velocity outweigh the privacy benefits?
- Would the open-source component enable clones that undermine your business, or attract new users who may eventually upgrade to your paid service?
I would like to find out the answers!
[0] https://github.com/dolthub/dolt