cloudquery VS cartography

Check out CloudQuery - https://github.com/cloudquery/cloudquery for an easy cloud asset inventory.

There both do something like what you're looking for.... https://github.com/cloudquery/cloudquery https://github.com/openraven/magpie

Unified API is a holly grail but as many said quite difficult to abstract every use case in a scalable way that won't break. At CloudQuery (https://github.com/cloudquery/cloudquery) we focus solely on the ELT use-case(Founder/Maintainer here).

Congrats!! How does it compare to the ELT space and the modern data stack where you have ingestion/storage/visualization layers decoupled?

Asking as the founder of CloudQuery (https://github.com/cloudquery/cloudquery), Saw Datasette quite a few times around data exploration but curious to hear about the most popular use-cases of Datasette!

Congrats!! We also focus on performance at CloudQuery (https://github.com/cloudquery/cloudquery) by using Golang, gRPC and still trying to be abstract enough to support different databases :)

In any case good luck!

CloudQuery for ETL

Another ELT framework that's an alternative to Airbyte

Another alternate ELT

Check out CloudQuery, we have PostgreSQL source connectors and S3 destination that supports parquet (Disclaimer: Maintainer and founder here)

Hello! Im Yevgeny, Founder & maintainer at CloudQuery . We've built CloudQuery as an open source high performance ELT framework so you should get pretty good results syncing all your cloud assets from high number of accounts (we have users syncing more than 10K Azure subscription and thousands of AWS accounts concurrently).

Cartography (Python application) will get you part of the way there, in that it will discover the state of the environment, record node information and relationships in a graph database, and can be easily extended. You can also then easily write queries about the relationships of various resources to answer questions like “what can I reach from VPC X”. Drawing diagrams though is very hard to do well without extra input.

Build an asset inventory of the cloud environment. (using an OSS project called cartography)

The project is still early, so we’d love your feedback! We’ve based our cloud asset inventory on another great OSS project called cartography. So far, we’ve added misconfiguration checks and common identity-based attack paths. Up next on our roadmap are network/access graph visualizations, vulnerability scanning, and secret scanning!

The first step involved in threat modelling is to build an architecture diagram of the system you are protecting. This also involves building a list of assets in the system. You cannot protect something which you don’t have visibility over. We can use tools such as CloudMapper from the previous step for building an architecture diagram of an AWS cloud environment. There are similar tools for threat modeling for other cloud service providers like Cartography.

Lyft made a tool called Cartography, which scrapes your AWS Account(s) and inserts representations of many of your cloud resources into a Neo4j (graph) database. This makes it easy to query and discover how things are connected in your account.

Recently, we at Cloudanix made open source contributions to the Cartography project by Lyft. We contributed to the pre-existing AWS module by adding support for KMS and API gateway. Since the Cartography project did not have support for Microsoft Azure, we started from scratch to add that capability to Cartography. We added support for Azure services like CosmosDB, SQL, Storage, and Virtual Machines.

[2] https://github.com/lyft/cartography