steampipe-plugin-sdk VS cloudquery

🔧 The CLI is written in Go, as well as the Steampipe Plugins connecting to various APIs like AWS, GitHub, Zoom, Reddit, and 130+ more. https://hub.steampipe.io/plugins

It's fully open source with a thriving community of users and contributors. As of today, there are 111 plugins (including AWS & Azure). We also have thousands of open source queries, security benchmarks, resource graphs and dashboards available out of the box as mods. See the Steampipe Hub for full docs and details.

Steampipe is free, open source. It's a CLI used to SQL query & report across all your cloud APIs & data sources (e.g. AWS, GitHub, Slack, Kubernetes, Terraform, etc) along with a dashboards-as-code capability to run automated security & compliance benchmarks. Allows you to simply query & report without the overhead for log ingestion, ETL or a DB.

For some background, the Steampipe CLI provides an open source, common SQL interface to query join and report from your cloud APIs (e.g. Azure, AWS, GitHub, AzureAD, M365, etc).

Not the author, but a big fan of steampipe. You can use this example as a starting point to write integrations with a number of other services that are supported by steampipe plugins.

Since Steampipe provides a common SQL interface across plugins, there could be interesting 'joins' that you can do for querying & reporting -- any interesting joins you would consider across SPO, M365 or other plugins?

The CLI is written primarily in Go, as well as its plugins (e.g. AWS, GitHub, Slack, Zoom, Reddit, etc): https://hub.steampipe.io/plugins. You can extend an existing plugin by adding new 'tables', or you can write your own plugin for your favorite cloud service.

Check out CloudQuery - https://github.com/cloudquery/cloudquery for an easy cloud asset inventory.

There both do something like what you're looking for.... https://github.com/cloudquery/cloudquery https://github.com/openraven/magpie

Unified API is a holly grail but as many said quite difficult to abstract every use case in a scalable way that won't break. At CloudQuery (https://github.com/cloudquery/cloudquery) we focus solely on the ELT use-case(Founder/Maintainer here).

Congrats!! How does it compare to the ELT space and the modern data stack where you have ingestion/storage/visualization layers decoupled?

Asking as the founder of CloudQuery (https://github.com/cloudquery/cloudquery), Saw Datasette quite a few times around data exploration but curious to hear about the most popular use-cases of Datasette!

Congrats!! We also focus on performance at CloudQuery (https://github.com/cloudquery/cloudquery) by using Golang, gRPC and still trying to be abstract enough to support different databases :)

In any case good luck!

CloudQuery for ETL

Another ELT framework that's an alternative to Airbyte

Another alternate ELT

Check out CloudQuery, we have PostgreSQL source connectors and S3 destination that supports parquet (Disclaimer: Maintainer and founder here)

Hello! Im Yevgeny, Founder & maintainer at CloudQuery . We've built CloudQuery as an open source high performance ELT framework so you should get pretty good results syncing all your cloud assets from high number of accounts (we have users syncing more than 10K Azure subscription and thousands of AWS accounts concurrently).