Holy Uck Fast Analysis with DuckDB and Pyarrow

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • SonarQube - Static code analysis for 29 languages.
  • Scout APM - Less time debugging, more time building
  • SaaSHub - Software Alternatives and Reviews
  • pqrs

    Command line tool for inspecting Parquet files

    Shameless plug, my project exposes the common Parquet operations using a Rust CLI tool using the Rust API for Apache Arrow and can be used without any Java/Hadoop/Spark dependencies. Also available a static binary.

    https://github.com/manojkarthick/pqrs

  • duckdb

    DuckDB is an in-process SQL OLAP Database Management System

    We have a Relational API in addition to SQL! Here are some examples for the Python Relational API client:

    https://github.com/duckdb/duckdb/blob/master/examples/python...

    Plus, if you are working in Python, you can use DuckDB as the engine underneath Ibis, Fugue, Siuba, or anything that works with SQLAlchemy (using the DuckDB-engine driver)! In R, you can use dplyr or dbplyr.

    DuckDB's file format is one way to persist data (it uses a single file), but you can also write out to Parquet, or write out to Apache Arrow and then parquet (in a partitioned format I believe).

    Disclaimer - I write docs for DuckDB!

  • SonarQube

    Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

  • raku-Dan

    Top level raku Data ANalysis Module

    Hi @ritchie46 - I have just written [raku Dan](https://github.com/p6steve/raku-Dan) as a way to scratch the "data analytics" itch in a new way -- my next step is to write Dan::Polars as a polars binding via (eg.) raku NativeCall. Can you point me to a good recipe for success? [email protected]

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts