Our great sponsors
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
Apache Arrow
Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
That's it. Pigeon has some more handy features which you can check out on github and it is on crates.io as well.
Pigeon is a command line tool based on the excellent libraries clap, connectorx, and polars.
Finally, I think, a compiled binary is handy for command line tools. Admittedly, you have to compile it first, which will take some time. But then you can put ~/.cargo/bin to your PATH, install the latest version from crates.io via cargo install pigeon-rs, and use the binary from everywhere. In Python you would need an alias first because otherwise you would always have to type python3 path/to/python/file. And might have to worry that the command line arguments are piped to the python executable?
Connectorx is using arrow2 data format for fetching from a database. This data format is optimized for columnar data [1]:
The problem with a crate like postgres is that you have to define the types of the query at compile-time. And if you use the simple query protocol in postgres, you just get a bunch of strings, i.e. no proper typing at all. However, for maximal flexibility arbitrary queries should work in pigeon, without knowing the database schema.