jupysql VS grai-core

Hey, HN community!

We're stoked to launch JupySQL today! JupySQL is an open-source library that brings a modern SQL experience to Jupyter. JupySQL is compatible with all major databases, such as Snowflake, Redshift, PostgreSQL, MySQL, MariaDB, DuckDB, SQL Server, Clickhouse, Trino, and more!

To get started, check out our tutorial: https://jupysql.ploomber.io/en/latest/quick-start.html

SQL is the defacto language for data analysis; however, analysis often requires a mix of SQL and Python. JupySQL bridges this gap, allowing users to execute SQL queries seamlessly in Jupyter and continue their analysis in Python. Add %%sql to the top of your cell and start writing SQL.

Here are some of JupySQL's main features:

- Syntax highlighting

Please show your support with a 🌟: https://github.com/ploomber/jupysql

If anyone is looking for something like this in Python/Jupyter, check out JupySQL: https://github.com/ploomber/jupysql

How are people managing the existence of data frame APIs like pandas/polars with SQL engines like BigQuery, Snowflake, and DuckDB?

Most of my notebooks are a mix of SQL and Python: SQL for most processing, dump the results as a pandas dataframe (via https://github.com/ploomber/jupysql) and then use Python for operations that are difficult to express with SQL (or that I don't know how to do it), so I end up with 80% SQL, 20% Python.

Unsure if this is the best workflow but it's the most efficient one I've come up with.

Disclaimer: my team develops JupySQL.

Elastic v2 if one is interested in such things: https://github.com/grai-io/grai-core/blob/v0.1.33/LICENSE

I’m not sure if this is precisely what you’re looking for but Grai might serve your needs. The backend data model allows you to push any arbitrary metadata you want / need onto the lineage graph and retrieve it either through the rest or graph API. I’m one of the authors so happy to answer any questions you might have.

We've been looking at building something like this for Grai specifically to support Airflow but haven't yet prioritized it.

We were frustrated because although we had tests in our data warehouse, they only notified us after an outage occurred. What we needed was a way to detect changes during CI/CD, so we could fix things before they impacted production. So we developed Grai, as an open-source data lineage toolkit pre-built integrations for the most common data stores and designed to work with CI tools, like Github Actions.