cloud-custodian
cloudquery
Our great sponsors
cloud-custodian | cloudquery | |
---|---|---|
32 | 102 | |
5,180 | 5,524 | |
1.4% | 2.6% | |
9.5 | 10.0 | |
5 days ago | 7 days ago | |
Python | Go | |
Apache License 2.0 | Mozilla Public License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
cloud-custodian
-
Cutting down AWS cost by $150k per year simply by shutting things off
> The best optimization is simply shutting things off
This is the way.
A similar idea has been bouncing around in my mind for a while now. An ideal, turnkey system would do the following:
- Execute via Lambda (serverless).
- Support automated startup and shutdown of various AWS resources on a schedule influenced by specially formatted tags.
- Enable resources to be brought back up out of schedule when demand dictates.
- Operate as a TCP/HTTP proxy that can delay clients so that a given service can be started when it is dormant or, even better, the service isn't serverless but you want it to be. This can't work for everything, but perhaps enough things such that the need to run always on services is reduced.
Cloud Custodian [1] can purportedly do some of this, but I've been reluctant to learn yet another YAML-based DSL to use it.
So this is my "make things designed to be always-on serverless instead" project and the work AWS has done to make Java apps function on Lambda keeps me thinking about the potential to take things that 1) have a relatively long startup time and 2) are designed to be long running service loops, and find a way to force them into the serverless execution model.
-
Optimizing cost on an app which is not used 24/7
Use a tool like this https://cloudcustodian.io/ to manage instance on/off hours or go fargate.
-
EC2 start and stop via Lambda
I'd use a combination of Cloudcustodian for start/stop scheduling and Apprise for notifications.
- Enforce tagging on everything that can be tagged
-
cloud-custodian VS cloudquery - a user suggested alternative
2 projects | 2 Feb 2022
-
Common avenues for reducing waste in AWS (Specifically EC2)
You can try Cloudcustodian. Very good tool to help you make a list of all underutilized instances. This also helps you do a lot more than that. https://cloudcustodian.io/
-
Favorite Resources of 2021
Cloud Custodian; rules engine for cloud security, cost optimization, and governance, DSL in yaml for policies to query, filter, and take actions on resources
-
I added AWS CIS 1.2 compliance checks to GraphQL API for AWS!
Another alternative that I personally would use is CloudCustodian. It is a widely used tool for continuous cloud governance, detection and remediation. There is a CIS pack for it.
-
Implementing Cloud Governance as a Code using Cloud Custodian
Note: Cloud Custodian kubernetes resources still work in progress. We can check the status of the plugin here.
cloudquery
-
Cloud asset tracking
There both do something like what you're looking for.... https://github.com/cloudquery/cloudquery https://github.com/openraven/magpie
-
Show HN: Nango – Open unified API for product integrations
Unified API is a holly grail but as many said quite difficult to abstract every use case in a scalable way that won't break. At CloudQuery (https://github.com/cloudquery/cloudquery) we focus solely on the ELT use-case(Founder/Maintainer here).
-
Welcome to Datasette Cloud
Congrats!! How does it compare to the ELT space and the modern data stack where you have ingestion/storage/visualization layers decoupled?
Asking as the founder of CloudQuery (https://github.com/cloudquery/cloudquery), Saw Datasette quite a few times around data exploration but curious to hear about the most popular use-cases of Datasette!
-
Launch HN: PeerDB (YC S23) – Fast, Native ETL/ELT for Postgres
Congrats!! We also focus on performance at CloudQuery (https://github.com/cloudquery/cloudquery) by using Golang, gRPC and still trying to be abstract enough to support different databases :)
In any case good luck!
-
airbyte VS cloudquery - a user suggested alternative
2 projects | 2 Jun 2023
CloudQuery for ETL
2 projects | 2 Jun 2023Another ELT framework that's an alternative to Airbyte
-
meltano VS cloudquery - a user suggested alternative
2 projects | 2 Jun 2023
Another alternate ELT
-
Cloudquery, Resoto, Steampipe, or Airbyte?
Cloudquery: https://cloudquery.io/
Hello! Im Yevgeny, Founder & maintainer at CloudQuery . We've built CloudQuery as an open source high performance ELT framework so you should get pretty good results syncing all your cloud assets from high number of accounts (we have users syncing more than 10K Azure subscription and thousands of AWS accounts concurrently).
-
Test On 4 Concurrent Jobs Using Python-Polars 0.17.11 to GroupBy Billion Rows
CloudQuery supports a lot of APIs https://github.com/cloudquery/cloudquery
What are some alternatives?
steampipe - Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
terraform - Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
steampipe-mod-aws-compliance - Run individual controls or full compliance benchmarks for CIS, PCI, NIST, HIPAA and more across all of your AWS accounts using Powerpipe and Steampipe.
gatekeeper - 🐊 Gatekeeper - Policy Controller for Kubernetes
ScoutSuite - Multi-Cloud Security Auditing Tool
fixinventory - Fix Inventory consolidates user, resource, and configuration data from your cloud environments into a unified, graph-based asset inventory.
cloudsploit - Cloud Security Posture Management (CSPM)
checkov - Prevent cloud misconfigurations and find vulnerabilities during build-time in infrastructure as code, container images and open source packages with Checkov by Bridgecrew.
cartography - Cartography is a Python tool that consolidates infrastructure assets and the relationships between them in an intuitive graph view powered by a Neo4j database.
opencspm - Open Cloud Security Posture Management Engine
cloud-guardrails - Rapidly apply hundreds of security controls in Azure