Supabase Wrappers: A Framework for Building Postgres Foreign Data Wrappers

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
  • wrappers

    Postgres Foreign Data Wrapper development framework in Rust.

    > How do you mock it? How do you test it?

    Stripe and Firebase both offer test endpoints for their services. The foreign data wrappers for each allow subbing in a user defined endpoint in their `options` so thats how I'd recommend testing the two that have released. Some of the pre-release integrations can be spun up locally in docker e.g. clickhouse. The FDWs for those similarly take an arbitrary connection string making it pretty straightforward to write tests. Here's an example from the repo https://github.com/supabase/wrappers/blob/main/wrappers/src/...

    > Add custom logic?

    Views!

    > tasks that the database is not designed to do

    The C API for foreign data wrappers is baked right into Postgres proper. They've been around since 2013 and are pretty battle hardened at this point. Supabase Wrappers exposes that API in rust so users can write FDWs without worrying about accidentally tanking the Postgres process. Its more about making a great Postgres feature more accessible than tricking the database into doing anything new.

  • Multicorn

    Data Access Library

    Our release today is a framework which extends this functionality to other databases/systems. If you’re familiar with Multicorn[1] or Steampipe[2], then it’s very similar. The framework is written in Rust, using the excellent pgx[3].

    We have developed FDWs for Stripe, Firebase, BigQuery, Clickhouse, and Airtable (all in various “pre-release” states). The plan is to focus on the tools we’re using internally while we stabalize the framework.

    There’s a lot in the blog post into our goals for this release. It’s early, but one of the things I’m most excited about.

    [0] Postgres FDW: [https://www.postgresql.org/docs/current/sql-createforeigndat...

    [1] Multicorn: https://multicorn.org/

    [2] Steampipe: https://steampipe.io/

    [2] pgx: [https://github.com/tcdi/pgx](https://github.com/tcdi/pgx)

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • pgx

    Discontinued Build Postgres Extensions with Rust! [Moved to: https://github.com/tcdi/pgrx] (by tcdi)

    Our release today is a framework which extends this functionality to other databases/systems. If you’re familiar with Multicorn[1] or Steampipe[2], then it’s very similar. The framework is written in Rust, using the excellent pgx[3].

    We have developed FDWs for Stripe, Firebase, BigQuery, Clickhouse, and Airtable (all in various “pre-release” states). The plan is to focus on the tools we’re using internally while we stabalize the framework.

    There’s a lot in the blog post into our goals for this release. It’s early, but one of the things I’m most excited about.

    [0] Postgres FDW: [https://www.postgresql.org/docs/current/sql-createforeigndat...

    [1] Multicorn: https://multicorn.org/

    [2] Steampipe: https://steampipe.io/

    [2] pgx: [https://github.com/tcdi/pgx](https://github.com/tcdi/pgx)

  • steampipe

    Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

    Our release today is a framework which extends this functionality to other databases/systems. If you’re familiar with Multicorn[1] or Steampipe[2], then it’s very similar. The framework is written in Rust, using the excellent pgx[3].

    We have developed FDWs for Stripe, Firebase, BigQuery, Clickhouse, and Airtable (all in various “pre-release” states). The plan is to focus on the tools we’re using internally while we stabalize the framework.

    There’s a lot in the blog post into our goals for this release. It’s early, but one of the things I’m most excited about.

    [0] Postgres FDW: [https://www.postgresql.org/docs/current/sql-createforeigndat...

    [1] Multicorn: https://multicorn.org/

    [2] Steampipe: https://steampipe.io/

    [2] pgx: [https://github.com/tcdi/pgx](https://github.com/tcdi/pgx)

  • pg_net

    A PostgreSQL extension that enables asynchronous (non-blocking) HTTP/HTTPS requests with SQL

    > speaks a particular API over the network

    it's a interesting idea, and one of the things that we were toying with in our pg_net extension (https://github.com/supabase/pg_net). This is a "generic" async network extension, so you can fetch/put/post. It works well for APIs.

    I think the generic approach works for some things where the data is less "fixed" - for example, an OpenAI API endpoint.

    But for "fixed" data (data warehouses), the wrapper usually needs some custom work for security, protocols, and "push down". I'll be interested to get HN's take on this - they might have some suggestions for us for this framework

  • pgsql-http

    HTTP client for PostgreSQL, retrieve a web page from inside the database.

  • wundergraph

    WunderGraph is a Backend for Frontend Framework to optimize frontend, fullstack and backend developer workflows through API Composition.

    We (https://wundergraph.com) do exactly this, API integration with GraphQL as the integration layer. Integrating APIs is messy. It could require custom middleware, etc... Glueing something together is the easy part. How do you mock it? How do you test it? Add custom logic? Doing it at the API layer allows you to seamlessly scale your serverless workloads. Doing all this work in the database means that we're eating up resources for tasks that the database is not designed to do. Just because FDW exists doesn't mean we should use it for this use case.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • sqlx

    🧰 The Rust SQL Toolkit. An async, pure Rust SQL crate featuring compile-time checked queries without a DSL. Supports PostgreSQL, MySQL, and SQLite. (by launchbadge)

    > I wonder why this wasn't built on top of, or an enhancement to, the existing (excellent) multicorn[1] project

    Have to agree with you there, multicorn is extremely cool. I'm a big sqlalchemy fan so their default SQLA wrapper was a killer feature to give up (although maybe we could do something similar with launchbadge/sqlx[1]). We investigated using multicorn early this year and had a few hiccups. Activity on the original repo[2] quieted way down ~3 years ago. For example, pg14 support hasn't landed and the newest supported python version is EOL in 2022. There is new fork[3] with pg14 support (15 in the pipe) that might pick up in adoption but thats still TBD.

    Supabase aims to support new major Postgres versions within 2-3 months so we have to be very careful taking on dependencies that might slow that process.

    > I'd love to understand more about the technical rationale that drove this.

    Architecturally, multicorn has postgres communicate with a separate python process on the host that does all the hard work. That's convenient, but it can bloat over time and/or be memory hungry for larger result sets. The rust implementation runs in-process and is generally a lot lighter.

    Currently I'd say supabase/wrappers is a safer/easier version of the C API vs a direct analog to multicorn. Over time I think we'll see that comparison become more appropriate. There's a lot of excitement around the concept internally and we've already been floating some ideas wrt `auto-mapping` tables for common SQL dialects etc. Stay tuned!

    Again,

    [1]https://github.com/launchbadge/sqlx

  • multicorn2

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts