skyplane
cloud-custodian
skyplane | cloud-custodian | |
---|---|---|
5 | 32 | |
973 | 5,229 | |
1.4% | 0.8% | |
6.9 | 9.5 | |
2 months ago | 5 days ago | |
Python | Python | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
skyplane
- Moving cloud storage
-
UC Berkeley Launches SkyPilot to Help Navigate Soaring Cloud Costs
I'm one of the creators of Skyplane. Skyplane can migrate large datasets between cloud regions at 10s of Gbps while compressing data to reduce egress fees. Happy to chime in!
https://github.com/skyplane-project/skyplane
-
IaSQL and Skyplane: Open Source Projects to Boost Your AWS Deployments
Skyplane, is an open source tool developed at UC Berkeley to optimize the transfer of large datasets between cloud providers, reducing transfer times and costs. It creates an overlay network on top of the cloud providers so it can route around congested cloud network links.
-
Those using AWS, have you ever tried to use AWS Transfer Family to transfer files into an S3 bucket? Can I use python to make these uploads, and if so how do I set it up in aws?
In case you want to move very large data between clouds, I recommend taking a look at https://github.com/skyplane-project/skyplane/
- Skyplane: 110x faster data transfers on any cloud
cloud-custodian
-
Cutting down AWS cost by $150k per year simply by shutting things off
> The best optimization is simply shutting things off
This is the way.
A similar idea has been bouncing around in my mind for a while now. An ideal, turnkey system would do the following:
- Execute via Lambda (serverless).
- Support automated startup and shutdown of various AWS resources on a schedule influenced by specially formatted tags.
- Enable resources to be brought back up out of schedule when demand dictates.
- Operate as a TCP/HTTP proxy that can delay clients so that a given service can be started when it is dormant or, even better, the service isn't serverless but you want it to be. This can't work for everything, but perhaps enough things such that the need to run always on services is reduced.
Cloud Custodian [1] can purportedly do some of this, but I've been reluctant to learn yet another YAML-based DSL to use it.
So this is my "make things designed to be always-on serverless instead" project and the work AWS has done to make Java apps function on Lambda keeps me thinking about the potential to take things that 1) have a relatively long startup time and 2) are designed to be long running service loops, and find a way to force them into the serverless execution model.
[1] https://cloudcustodian.io/
-
When have you screwed up, bad?
Goal was to clear up anything old and set us up to rotate keys in use. Why did I do it in the end of December? It was a quarterly goal and I learned to push those across the line if I wanted a good review. Great incentive, that one. I used Cloud Custodian for this. It has a terrible bug where the code says you'll be acting on days since the key was used but actually is reading days since it was created.
-
Open-Source tools for monitoring ML/AI usage- Recommendations?
What is wrong with https://github.com/cloud-custodian/cloud-custodian?
-
Automate deletion of aws ebs snapshots older than year
You can start reading about it here.
-
Optimizing cost on an app which is not used 24/7
Use a tool like this https://cloudcustodian.io/ to manage instance on/off hours or go fargate.
-
Going for the CCP with a Compliance background. Any insight on what direction to pursue in AWS?
Certs aside, there are some great compliance tools out there that you might want to become familiar with. Here is one that comes to mind (is open-source): https://cloudcustodian.io/
- What are some of the community's best recommendations and use cases for Cost Optimization and FinOps
-
EC2 start and stop via Lambda
I'd use a combination of Cloudcustodian for start/stop scheduling and Apprise for notifications.
-
Tag Enforcement
Cloud custodian is a good utility if config rules doesn’t satisfy your needs. It’s also cross platform.
-
26 AWS Security Best Practices to Adopt in Production
AWS Security with open source – Cloud Custodian is a Cloud Security Posture Management (CSPM) tool. CSPM tools evaluate your cloud configuration and identify common configuration mistakes. They also monitor cloud logs to detect threats and configuration changes.
What are some alternatives?
skypilot - SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution—all with a simple interface.
terraform - Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
cloudpods - A cloud-native open-source unified multi-cloud and hybrid-cloud platform. 开源、云原生的多云管理及混合云融合平台
ScoutSuite - Multi-Cloud Security Auditing Tool
prowler - Prowler is an Open Source Security tool for AWS, Azure, GCP and Kubernetes to do security assessments, audits, incident response, compliance, continuous monitoring, hardening and forensics readiness. Includes CIS, NIST 800, NIST CSF, CISA, FedRAMP, PCI-DSS, GDPR, HIPAA, FFIEC, SOC2, GXP, Well-Architected Security, ENS and more
steampipe - Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.
s5cmd - Parallel S3 and local filesystem execution tool.
gatekeeper - 🐊 Gatekeeper - Policy Controller for Kubernetes
docker-android - Android in docker solution with noVNC supported and video recording
fixinventory - Fix Inventory consolidates user, resource, and configuration data from your cloud environments into a unified, graph-based asset inventory.
cloud_sizes - Track the sizes of different cloud providers
cloudquery - The open source high performance ELT framework powered by Apache Arrow