mergestat-lite
spyql
mergestat-lite | spyql | |
---|---|---|
10 | 23 | |
3,419 | 902 | |
0.3% | - | |
6.3 | 0.0 | |
3 days ago | over 1 year ago | |
Go | Jupyter Notebook | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
mergestat-lite
-
SQLite Doesn't Use Git
You can query git with this: https://github.com/mergestat/mergestat if you like the idea.
-
A SQLite extension for reading large files line-by-line
Hey, author here, happy to answer any questions! Also checkout this notebook for a deeper dive into sqlite-lines, along with a slick WASM demonstration and more thoughts on the codebase itself https://observablehq.com/@asg017/introducing-sqlite-lines
I really dig SQLite, and I believe SQLite extensions will push it to another level. I rarely reach for Pandas or other "traditional" tools and query languages, and instead opt for plain ol' SQLite and other extensions. As a shameless plug, I recently started a blog series on SQLite and related tools and extensions if you want to learn more! Next week I'll be publishing more SQLite extensions for parsing HTML + making HTTP requests https://observablehq.com/@asg017/a-new-sqlite-blog-series
A few other SQLite extensions:
- xlite, for reading Excel files, in Rust https://github.com/x2bool/xlite
- sqlean, several small SQLite extensions in C https://github.com/nalgeon/sqlean
- mergestat, several SQLite extensions for developers (mainly Github's API) in Go https://github.com/mergestat/mergestat
- Show HN: Contribution Graph as a Git Command
-
Exploring Git Repos With MergeStat π¬
mergestat is an open-source tool that allows users to run SQL queries on the contents and history of git repositories.
-
The world of PostgreSQL wire compatibility
Thanks for this write up! I've been really interested in postgres compatibility in the context of a tool I maintain (https://github.com/mergestat/mergestat) that uses SQLite. I've been looking for a way to expose the SQLite capabilities over a more commonly used wire-protocol like postgres (or mysql) so that existing BI and visualization tools can access the data.
This project is an interesting one: https://github.com/dolthub/go-mysql-server that provides a MySQL interface (wire and SQL) to arbitrary "backends" implemented in go.
It's really interesting how compatibility with existing protocols has become an important feature of new databases - there's so much existing tooling that already speaks postgres (or mysql), being able to leverage that is a huge advantage IMO
-
Go library for printing human readable, relative time differences π°οΈ
timediff is a Go package for printing human readable, relative time differences. Output is based on ranges defined in the Day.js JavaScript library, and can be customized if needed. It's currently used by the mergestat command-line interface.
- Askgit: Command-line tool for running SQL queries on Git repositories
-
Semantic Git Commit Messages
Assuming committers adhere to it, there could be some interesting use cases when combined with a tool like AskGit (https://github.com/askgitdev/askgit) for understanding what "categories" of work is being done in a codebase.
Maybe even what directories/files tend to see `fix` or `refactor` more frequently (signs of a poorly design or "hot" area?)
-
Git as a NoSql Database
I've been very curious to explore this type of use case with askgit (https://github.com/augmentable-dev/askgit) which was designed for running simple "slice and dice" queries and aggregations on git history (and change stats) for basic analytical purposes. I've been curious about how this could be applied to a small text+git based "db". Say, for a regular json or CSV dumps.
This also reminds me of Dolt: https://github.com/dolthub/dolt which I believe has been on HN a couple times
spyql
-
Fq: Jq for Binary Formats
I prefer a SQL-like format. Itβs not as complete but it cover most of the day-to-day use cases. Take a look at https://github.com/dcmoura/spyql (I am the author). Congrats on fq!
-
Command-line data analytics made easy with SPyQL
SPyQL documentation: spyql.readthedocs.io
-
This Week In Python
spyql β Query data on the command line with SQL-like SELECTs powered by Python expressions
- Command-line data analytics made easy
-
Jc β JSONifies the output of many CLI tools
This is great!
I am the author of SPyQL [1]. Combining JC with SPyQL you can easily query the json output and run python commands on top of it from the command-line :-) You can do aggregations and so forth in a much simpler and intuitive way than with jq.
I just wrote a blogpost [2] that illustrates it. It is more focused on CSV, but the commands would be the same if you were working with JSON.
[1] https://github.com/dcmoura/spyql
- The fastest command-line tools for querying large JSON datasets
-
Working with more than 10gb csv
You can import the data into a PostgreSQL/MySQL/SQLite/... database and then query the database. However, even with the right choice of indexes, it might take a while to run queries on a table with hundreds of millions of records. You can easily import your data to these databases with SpyQL: $ spyql "SELECT * FROM csv TO sql(table=my_table_name) | sqlite3 my.db" (you would need to create the table my_table_name before running the command).
-
ClickHouse Cloud is now in Public Beta
https://github.com/dcmoura/spyql/blob/master/notebooks/json_...
And ClickHouse looks like a normal relational database - there is no need for multiple components for different tiers (like in Druid), no need for manual partitioning into "daily", "hourly" tables (like you do in Spark and Bigquery), no need for lambda architecture... It's refreshing how something can be both simple and fast.
- A SQLite extension for reading large files line-by-line
-
I want to convert a large JSON file into Tabular Format.
I thought this library was pretty nifty for json. It's also relatively fast compared to most json parsers: https://github.com/dcmoura/spyql
What are some alternatives?
git-xargs - git-xargs is a command-line tool (CLI) for making updates across multiple Github repositories with a single command.
prql - PRQL is a modern language for transforming data β a simple, powerful, pipelined SQL replacement
crux - General purpose bitemporal database for SQL, Datalog & graph queries. Backed by @juxt [Moved to: https://github.com/xtdb/xtdb]
malloy - Malloy is an experimental language for describing data relationships and transformations.
flan - A tasty tool that lets you save, load and share postgres snapshots with ease
tresql - Shorthand SQL/JDBC wrapper language, providing nested results as JSON and more
sqlite-plus - The ultimate set of SQLite extensions
Preql - An interpreted relational query language that compiles to SQL.
csv-sql - Command-line tool to load csv and excel (xlsx) files and run sql commands
prosto - Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby
datasette-lite - Datasette running in your browser using WebAssembly and Pyodide
pxi - π§ pxi (pixie) is a small, fast, and magical command-line data processor similar to jq, mlr, and awk.