Apache Calcite
sqlitebrowser
Our great sponsors
Apache Calcite | sqlitebrowser | |
---|---|---|
28 | 279 | |
4,320 | 19,802 | |
2.1% | 1.3% | |
9.0 | 9.0 | |
1 day ago | 2 days ago | |
Java | C++ | |
Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Apache Calcite
-
Data diffs: Algorithms for explaining what changed in a dataset (2022)
> Make diff work on more than just SQLite.
Another way of doing this that I've been wanting to do for a while is to implement the DIFF operator in Apache Calcite[0]. Using Calcite, DIFF could be implemented as rewrite rules to generate the appropriate SQL to be directly executed against the database or the DIFF operator can be implemented outside of the database (which the original paper shows is more efficient).
-
Apache Baremaps: online maps toolkit
Yes, planetiler rocks and the memory mapped collections enabled us to remove our dependency to rocksdb.
From my perspective, planetiler started as an effort to generate vector tiles from the OpenMapTile schema as fast as possible (pbf -> mvt). By contrast, Baremaps started as an effort to create a new schema and style from the ground up. In this regard, having a database (pbf -> db <- mvt) enables to live reload changes made in the configuration files. The database has a cost, but also comes with additional advantages (updates, dynamic data, generation of tiles at zoom levels 16+, etc.).
That being said, I think the two projects overlap and I hope we will find opportunities to collaborate in the future. For instance, whereas PostgreSQL is still required in Baremaps, I recently ported a lot of the ST_ function of Postgis to Apache Calcite with the intent to execute SQL on fast memory mapped collection.
https://github.com/apache/calcite/blob/main/core/src/main/ja...
A planet wide import in Postgis currently takes about 4 hours with the COPY API (easy to parallelize) followed by about 12 hours of simplification in Postgis (not easy to parallelize). I will try to publish a detailed benchmark in the future.
-
How to manipulate SQL string programmatically?
Use a SQL Parser like sqlglot or Apache Calcite to compile user's query into an AST.
- Can SQL be used without an RDBMS?
- Want to contribute more to open source projects.
-
Postgres wire compatible SQLite proxy
Awesome to see work in the DB wire compatible space. On the MySQL side, there was MySQL Proxy (https://github.com/mysql/mysql-proxy), which was scriptable with Lua, with which you could create your own MySQL wire compatible connections. Unfortunately it appears to have been abandoned by Oracle and IIRC doesn't work with 5.7 and beyond. I used it in the past to hack together a MySQL wire adapter for Interana (https://scuba.io/).
I guess these days the best approach for connecting arbitrary data sources to existing drivers, at least for OLAP, is Apache Calcite (https://calcite.apache.org/). Unfortunately that feels a little more involved.
-
Launch HN: Hydra (YC W22) β Query Any Database via Postgres
For anyone interested, Apache Calcite[0] is an open source data management framework which seems to do many of the same things that Hydra claims to do, but taking a different approach. Operating as a Java library, Calcite contains "adapters" to many different data sources from existing JDBC connectors to Elasticsearch to Cassandra. All of these different data sources can be joined together as desired. Calcite also has it's own optimizer which is able to push down relevant parts of the query to the different data sources. However, you get full SQL on data sources which don't support it, with Calcite executing the remaining bits itself.
Unfortunately, I would not be too surprised if Calcite was found to be less performance-optimized than Hydra. That said, there are users of Calcite at Google, Uber, Spotify, and others who have made great use of various parts of the framework.
-
Open Source SQL Parsers
There are multiple projects that maintain parsers for popular open source databases like MySQL and Postgres. For other open source databases, the grammar can be extracted from the open-source project. For commercial databases, the only option is to reverse engineer the complete grammar. There are SQL parser/optimizer platforms like Apache Calcite that help to reduce the effort to implement the SQL dialect of your choice.
-
Does Java have an open source package that can execute SQL on txt/csv?
Yes. Apache Calcite can do that.
-
Memoization in Cost-based Optimizers
You may find a similar design in many production-grade heuristic optimizers. In our previous blog post about Presto, we discussed the Memo class that manages such references. In Apache Calcite, the heuristic optimizer HepPlanner models node references through the class HepRelVertex.
sqlitebrowser
- SQLite-Web: Web-based SQLite database browser written in Python
-
A SQLite extension that brings column-oriented tables to SQLite
Ahhh. We (sqlitebrowser.org) used to do something along those lines in one of our repos too.
We moved to downloading and compiling the latest SQLite release as a step in the unit tests instead, so we automatically always have the latest SQLite release.
It turns out to be really easy to do, because the SQLite download page has hidden tags in it just for this purpose:
https://github.com/sqlitebrowser/sqlitebrowser/blob/7a31ef97...
We have a spread of different GitHub Actions based workflows that do stuff whenever a PR is proposed or merged:
https://github.com/sqlitebrowser/sqlitebrowser/tree/master/....
Most of those are oriented around building packages for various OS's (Linux, macOS, Windows) so people can try the latest code.
While there are some tests, they're more like extremely basic sanity tests and don't rely on Docker.
Those tests rely on whichever version of SQLite was downloaded and compiled into the GUI (as per above code snippet).
---
That being said, that's for the client side GUI application. There's a server side of things too (https://github.com/sqlitebrowser/dbhub.io -> dbhub.io) that does use docker for it's automated tests:
https://github.com/sqlitebrowser/dbhub.io/tree/master/.githu...
Those are integration tests though (eg "make sure we didn't bust communication with our cli", "make sure our go library still works 100% with the server"), and a reasonably decent set of End to End (E2E) tests of the web interface using Cypress.
---
Does that help? :)
-
Sqlite3 Utility on the Browser
Do you know about SQLite DB Browser ? It's a multi platform application that would perfectly fit your use case :).
-
How do I get sql exactly?
Try https://sqlitebrowser.org/ itβs great for local use and training purposes
-
i feel overwhelmed , should i start learning django from tutorials first or begin by building projects.
You can see how the db file looks for the app using https://sqlitebrowser.org/
-
SQLite-Utils
If you want to fiddle with SQLite and don't need all the power herein, I recommend DB Browser for SQLite.
- How to access cards of a deck ( .apkg file) programmatically?
-
What's the easiest way to quickly run SQL queries on a small datasets?
An SQLite database? There are GUI you can use like DB Browser for SQLite which is easy to run.
-
Ask HN: Examples of desktop software with 20+ years of longevity?
DB Browser for SQLite (https://sqlitebrowser.org) was first released to the public domain on 2003-08-19. So, it'll be 20 years in a few months time. :)
What are some alternatives?
sqlitestudio - A free, open source, multi-platform SQLite database manager.
Trino - Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
ANTLR - ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.
Presto - The official home of the Presto distributed SQL query engine for big data
sqlcipher - SQLCipher is a standalone fork of SQLite that adds 256 bit AES encryption of database files and other security features.
JSqlParser - JSqlParser parses an SQL statement and translate it into a hierarchy of Java classes. The generated hierarchy can be navigated using the Visitor Pattern
Apache Spark - Apache Spark - A unified analytics engine for large-scale data processing
Apache Drill - Apache Drill is a distributed MPP query layer for self describing data
zetasql - ZetaSQL - Analyzer Framework for SQL
nocodb - π₯ π₯ π₯ Open Source Airtable Alternative
dbhub.io - A "Cloud" for SQLite databases. Collaborative development for your data. π
beekeeper-studio - Modern and easy to use SQL client for MySQL, Postgres, SQLite, SQL Server, and more. Linux, MacOS, and Windows.