Apache Calcite
sshfs
Our great sponsors
Apache Calcite | sshfs | |
---|---|---|
28 | 59 | |
4,363 | 4,618 | |
2.1% | - | |
9.0 | 3.7 | |
2 days ago | almost 2 years ago | |
Java | C | |
Apache License 2.0 | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Apache Calcite
-
Data diffs: Algorithms for explaining what changed in a dataset (2022)
> Make diff work on more than just SQLite.
Another way of doing this that I've been wanting to do for a while is to implement the DIFF operator in Apache Calcite[0]. Using Calcite, DIFF could be implemented as rewrite rules to generate the appropriate SQL to be directly executed against the database or the DIFF operator can be implemented outside of the database (which the original paper shows is more efficient).
[0] https://calcite.apache.org/
-
Apache Baremaps: online maps toolkit
Yes, planetiler rocks and the memory mapped collections enabled us to remove our dependency to rocksdb.
From my perspective, planetiler started as an effort to generate vector tiles from the OpenMapTile schema as fast as possible (pbf -> mvt). By contrast, Baremaps started as an effort to create a new schema and style from the ground up. In this regard, having a database (pbf -> db <- mvt) enables to live reload changes made in the configuration files. The database has a cost, but also comes with additional advantages (updates, dynamic data, generation of tiles at zoom levels 16+, etc.).
That being said, I think the two projects overlap and I hope we will find opportunities to collaborate in the future. For instance, whereas PostgreSQL is still required in Baremaps, I recently ported a lot of the ST_ function of Postgis to Apache Calcite with the intent to execute SQL on fast memory mapped collection.
https://github.com/apache/calcite/blob/main/core/src/main/ja...
A planet wide import in Postgis currently takes about 4 hours with the COPY API (easy to parallelize) followed by about 12 hours of simplification in Postgis (not easy to parallelize). I will try to publish a detailed benchmark in the future.
-
How to manipulate SQL string programmatically?
Use a SQL Parser like sqlglot or Apache Calcite to compile user's query into an AST.
- Can SQL be used without an RDBMS?
- Apache Calcite
- Want to contribute more to open source projects.
-
CITIC Industrial Cloud — Apache ShardingSphere Enterprise Applications
The SQL Federation engine contains processes such as SQL Parser, SQL Binder, SQL Optimizer, Data Fetcher and Operator Calculator, suitable for dealing with co-related queries and subqueries cross multiple database instances. At the underlying layer, it uses Calcite to implement RBO (Rule Based Optimizer) and CBO (Cost Based Optimizer) based on relational algebra, and query the results through the optimal execution plan.
-
Postgres wire compatible SQLite proxy
Awesome to see work in the DB wire compatible space. On the MySQL side, there was MySQL Proxy (https://github.com/mysql/mysql-proxy), which was scriptable with Lua, with which you could create your own MySQL wire compatible connections. Unfortunately it appears to have been abandoned by Oracle and IIRC doesn't work with 5.7 and beyond. I used it in the past to hack together a MySQL wire adapter for Interana (https://scuba.io/).
I guess these days the best approach for connecting arbitrary data sources to existing drivers, at least for OLAP, is Apache Calcite (https://calcite.apache.org/). Unfortunately that feels a little more involved.
-
Launch HN: Hydra (YC W22) – Query Any Database via Postgres
For anyone interested, Apache Calcite[0] is an open source data management framework which seems to do many of the same things that Hydra claims to do, but taking a different approach. Operating as a Java library, Calcite contains "adapters" to many different data sources from existing JDBC connectors to Elasticsearch to Cassandra. All of these different data sources can be joined together as desired. Calcite also has it's own optimizer which is able to push down relevant parts of the query to the different data sources. However, you get full SQL on data sources which don't support it, with Calcite executing the remaining bits itself.
Unfortunately, I would not be too surprised if Calcite was found to be less performance-optimized than Hydra. That said, there are users of Calcite at Google, Uber, Spotify, and others who have made great use of various parts of the framework.
[0] https://calcite.apache.org/
-
Anyone know of any software that can help in designing then outputting to various database
Abstraction Layer - You can use something like Calcite to abstract out your data storage. https://calcite.apache.org/
sshfs
- Sshfs Still Orphaned?
-
Rclone syncs your files to cloud storage
> It's replaced sshfs for some cases.
I'd been using sshfs for some years until I learned that rclone can mount remotes to the file system, and I've been using that happily since then.
https://rclone.org/commands/rclone_mount/
> at present SSHFS does not have any active, regular contributors, and there are a number of known issues
https://github.com/libfuse/sshfs#development-status
- sshfs is NO longer orphaned
-
A currently maintained fork of SSHFS
Interesting, I alaways assumed sshfs was part of OpenSSH, learn something new every day.
Also, looks like sshfs used in Slackware is abandoned.
https://github.com/libfuse/sshfs
A quote from the link, I wonder if this project will be the 'one':
>If you would like to take over this project, you are welcome to do so. Please fork it and develop the fork for a while. Once there has been 6 months of reasonable activity, please contact [email protected] and I'll be happy to give you ownership of this repository or replace with a pointer to the fork.
I also wonder if it was abandoned due to the RHEL re-orgs like what happened to bluetooth.
-
Free Tech Tools and Resources - PDF Tool, Windows Vulnerability, Linux Cheatsheet & More
SSHFS offers a solution for connecting to SSH servers through a network filesystem client. Enables users to seamlessly mount remote filesystems, without any server-side requirements. Underknowledge appreciates it "for mounting remote machines."
-
How does openrc run commands under another user, even if the user has no shell access? My attempts to do this with sshfs using `su` and `rununser` fail.
However, my setup relies on me using sshfs to "mount" a remote directory (which houses the media that jellyfin uses). For jellyfin to have access to this directory, it has to run the command under its user (based on sshfs manpages).
-
how best to edit remote files?
So I need to work with remote files and wondered how people here go about that. I've looked at sshfs, which seems the most obvious way to go and presumably would work fine (?), but it is an archived project; and tried distant.nvim, but that didn't click too well.
-
Does a solution like this exist?
As far as I am aware, sshfs is no longer actively maintained.
-
Mounting a remote filesystem over ssh - a story on how I finally managed to backup my phone
After getting a bit discouraged since the easy solution failed, I ended up discovering that you can mount a remote filesystem over ssh using sshfs.
- Directory as SSH link ?
What are some alternatives?
Trino - Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
S3 Server - Zenko CloudServer, an open-source Node.js implementation of the Amazon S3 protocol on the front-end and backend storage capabilities to multiple clouds, including Azure and Google.
ANTLR - ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.
crontab-ui - Easy and safe way to manage your crontab file
Presto - The official home of the Presto distributed SQL query engine for big data
solaris - A HTML5 game of strategy, intrigue and galactic conquest.
JSqlParser - JSqlParser parses an SQL statement and translate it into a hierarchy of Java classes. The generated hierarchy can be navigated using the Visitor Pattern
peek.nvim - Markdown preview plugin for Neovim
Apache Spark - Apache Spark - A unified analytics engine for large-scale data processing
rclone - "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
Apache Drill - Apache Drill is a distributed MPP query layer for self describing data
active-forks - Find active github forks of a repo https://git.io/vSnrC