cloudquery VS aws

Check out CloudQuery - https://github.com/cloudquery/cloudquery for an easy cloud asset inventory.

There both do something like what you're looking for.... https://github.com/cloudquery/cloudquery https://github.com/openraven/magpie

Unified API is a holly grail but as many said quite difficult to abstract every use case in a scalable way that won't break. At CloudQuery (https://github.com/cloudquery/cloudquery) we focus solely on the ELT use-case(Founder/Maintainer here).

Congrats!! How does it compare to the ELT space and the modern data stack where you have ingestion/storage/visualization layers decoupled?

Asking as the founder of CloudQuery (https://github.com/cloudquery/cloudquery), Saw Datasette quite a few times around data exploration but curious to hear about the most popular use-cases of Datasette!

Congrats!! We also focus on performance at CloudQuery (https://github.com/cloudquery/cloudquery) by using Golang, gRPC and still trying to be abstract enough to support different databases :)

In any case good luck!

CloudQuery for ETL

Another ELT framework that's an alternative to Airbyte

Another alternate ELT

Check out CloudQuery, we have PostgreSQL source connectors and S3 destination that supports parquet (Disclaimer: Maintainer and founder here)

Hello! Im Yevgeny, Founder & maintainer at CloudQuery . We've built CloudQuery as an open source high performance ELT framework so you should get pretty good results syncing all your cloud assets from high number of accounts (we have users syncing more than 10K Azure subscription and thousands of AWS accounts concurrently).

Do you have a policy that you want to codify, or that you’ve been running with python or bash scripts? You are welcome to try codifying it with CloudQuery Policies (See our github and docs for how to develop one). Feel free to visit our discord or GitHub to get help - we’ll also be happy to share your policy on CloudQuery Hub.

CloudQuery is an open-source cloud asset inventory powered by SQL, so a policy to the tool is simply an HCL configuration file that references SQL queries. How this works is by CloudQuery ingesting the data from your cloud provider, such as AWS or Azure, and then the policy executes the SQL statements against that data as if it were a test suite. Our co-founder Yevgeny recently wrote an amazing piece on running the PCI DSS policy against AWS. If you are curious about the multitude of policies currently available you can check them out at https://hub.cloudquery.io/ which is our centrally available and searchable source for publicly available policies.