cloud-custodian
steampipe
Our great sponsors
cloud-custodian | steampipe | |
---|---|---|
32 | 144 | |
5,180 | 6,325 | |
1.4% | 4.2% | |
9.5 | 9.7 | |
4 days ago | 6 days ago | |
Python | Go | |
Apache License 2.0 | GNU Affero General Public License v3.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
cloud-custodian
-
Cutting down AWS cost by $150k per year simply by shutting things off
> The best optimization is simply shutting things off
This is the way.
A similar idea has been bouncing around in my mind for a while now. An ideal, turnkey system would do the following:
- Execute via Lambda (serverless).
- Support automated startup and shutdown of various AWS resources on a schedule influenced by specially formatted tags.
- Enable resources to be brought back up out of schedule when demand dictates.
- Operate as a TCP/HTTP proxy that can delay clients so that a given service can be started when it is dormant or, even better, the service isn't serverless but you want it to be. This can't work for everything, but perhaps enough things such that the need to run always on services is reduced.
Cloud Custodian [1] can purportedly do some of this, but I've been reluctant to learn yet another YAML-based DSL to use it.
So this is my "make things designed to be always-on serverless instead" project and the work AWS has done to make Java apps function on Lambda keeps me thinking about the potential to take things that 1) have a relatively long startup time and 2) are designed to be long running service loops, and find a way to force them into the serverless execution model.
-
Optimizing cost on an app which is not used 24/7
Use a tool like this https://cloudcustodian.io/ to manage instance on/off hours or go fargate.
-
EC2 start and stop via Lambda
I'd use a combination of Cloudcustodian for start/stop scheduling and Apprise for notifications.
- Enforce tagging on everything that can be tagged
-
cloud-custodian VS cloudquery - a user suggested alternative
2 projects | 2 Feb 2022
-
Common avenues for reducing waste in AWS (Specifically EC2)
You can try Cloudcustodian. Very good tool to help you make a list of all underutilized instances. This also helps you do a lot more than that. https://cloudcustodian.io/
-
Favorite Resources of 2021
Cloud Custodian; rules engine for cloud security, cost optimization, and governance, DSL in yaml for policies to query, filter, and take actions on resources
-
I added AWS CIS 1.2 compliance checks to GraphQL API for AWS!
Another alternative that I personally would use is CloudCustodian. It is a widely used tool for continuous cloud governance, detection and remediation. There is a CIS pack for it.
-
Implementing Cloud Governance as a Code using Cloud Custodian
Note: Cloud Custodian kubernetes resources still work in progress. We can check the status of the plugin here.
steampipe
-
Show HN: Query Your Sheets with SheetSQL
Readers may also enjoy Steampipe [1], an open source CLI to live query Google Sheets [2] and 140+ other services with SQL (e.g. AWS, GitHub, etc). It uses Postgres Foreign Data Wrappers under the hood and supports joins etc across the services. (Disclaimer - I'm a lead on the project.)
-
Osquery: An sqlite3 virtual table exposing operating system data to SQL
be mindful of its AGPLv3 https://github.com/turbot/steampipe/blob/v0.21.8/LICENSE (AFAIK v0.4.3 is the last MIT release https://github.com/turbot/steampipe/blob/v0.4.3/LICENSE ) and the actual providers are Apache 2 <https://github.com/turbot/steampipe-plugin-aws/blob/v0.131.0...> (but I don't know if provider drift makes them compatible with 0.4 or not)
iasql seems to be AWS only, but good for them for taking this on:
Few projects in the same realm that you should also checkout -
[1] Steampipe (https://steampipe.io/)
[2] InfraSQL (https://iasql.com/)
-
How to run an AWS CIS v3.0 assessment in CloudShell
In a prior post I showed how to install Steampipe in AWS CloudShell to instantly query over 460+ resource types from your AWS APIs using SQL, and another post on how to use the Steampipe AWS Compliance mod to assess over 25+ security benchmarks across your AWS accounts.
- Git Query Language
-
Cutting down AWS cost by $150k per year simply by shutting things off
Readers may find Steampipe's [1] AWS Thrifty Mod [2] useful. It will automatically scan multiple accounts and regions for 50 cost saving opportunities - many of which are looking for over-provisioned or unused resources. For example, it's crazy how much you can save by doing things like just converting your EBS volumes to the newer gp3 type. Combine with Flowpipe [3] to automate checks and actions. It's all open source and extensible.
- FLaNK Weekly 08 Jan 2024
-
Zero-ETL for Postgres: Live-query cloud APIs with 100 open source FDWs
Steampipe [1] is an open source project [2] that includes an embedded Postgres to instantly query cloud, code & more with SQL. This release expands our plugin ecosystem [3] to be a full Zero-ETL platform. Steampipe plugins can now run natively in your own Postgres as Foreign Data Wrappers [4], as SQLite extensions [5] or as simple data export tools [6]. Please give it a try, we'd love your feedback and contributions!
-
5 Open Source tools written in Golang that you should know about
Steampipe is a tool that simplifies data extraction from APIs and services, eliminating the need for ETL (Extract, Transform, Load) processes. It includes several components: the Steampipe CLI for querying APIs, ensuring compliance, and creating dashboards; Steampipe Postgres FDWs (Foreign Data Wrappers) which turn APIs into foreign tables in Postgres; Steampipe SQLite extensions that convert APIs into SQLite virtual tables; and standalone export tools for directly exporting data from APIs without requiring a database. Additionally, Turbot Pipes, a part of Steampipe, offers capabilities for querying, checking, and visualizing data, tailored for DevOps teams with a focus on intelligence, automation, and security.
- Pipe Dreams: The life and times of Yahoo Pipes
What are some alternatives?
cloudquery - The open source high performance data integration platform built for developers.
terraform - Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
ScoutSuite - Multi-Cloud Security Auditing Tool
gatekeeper - 🐊 Gatekeeper - Policy Controller for Kubernetes
fixinventory - Fix Inventory consolidates user, resource, and configuration data from your cloud environments into a unified, graph-based asset inventory.
checkov - Prevent cloud misconfigurations and find vulnerabilities during build-time in infrastructure as code, container images and open source packages with Checkov by Bridgecrew.
metriql - The metrics layer for your data. Join us at https://metriql.com/slack
cloud-guardrails - Rapidly apply hundreds of security controls in Azure
inspec-aws - InSpec AWS Resource Pack https://www.inspec.io/
steampipe-mod-github-sherlock - Interrogate your GitHub resources with the help of the world's greatest detectives: Powerpipe + Steampipe + Sherlock.
AWS-Tag-Auditor - The AWS Tag Auditor is an automated process that scans through Amazon resources for missing tags