vscode-infracost VS cloud-custodian

Hi, we are Ali, Hassan, and Alistair, co-founders of Infracost (https://www.infracost.io/). Infracost helps engineers see the cost of each Terraform change before launching resources. When changes are made, it posts a comment with the cloud cost impact. For example, “you’ve added 2 instances and volumes, and change an instance type from medium to large, your bill will increase by 25% next month, from $1000 to $1250 per month”.

We launched in February 2021 (https://news.ycombinator.com/item?id=26064588), and Infracost is now being actively used by over 3,000 companies. However, there is a shift happening in the cloud cost management space. New teams, called FinOps teams (a combination of "Finance" and "DevOps"), are being formed within companies to manage cloud costs.

One of the first tasks assigned to these teams is to determine "who is using what" - that is, which teams, business units, products, etc. are spending the most on cloud. To accomplish this, they use tags. Tags are labels that all cloud resources should have and are key-value pairs. For example, a server could be tagged with: product=HackerNews; environment=production; team=blueTeam. So if resources are not tagged properly, then you can’t tell who is using what.

However, FinOps teams face challenges because their tools are reactive. These tools begin by analyzing cloud bills and providing visibility of tags from there. This means that they are looking at resources that are already running in production and costing money. A customer recently shared, “I want all resources to be properly tagged. But if they are not, I would rather a resource not be tagged at all than be tagged incorrectly.”

My "aha" moment! FinOps teams can define a tagging policy that can be validated in CI/CD before resources are launched. This is important because if code is shipped with the wrong tags, FinOps teams will have to fight for sprint time to fix them. Even if you shut down an untagged resource directly in the cloud, the next time Terraform runs, the resource will launch again with no tag. You need to fix the issue at its root.

I’d love your feedback on our solution to the tagging problem. You define your tag key-value policy in our SaaS product, and Infracost checks all Terraform resources per change. If anything fails the policy, it posts a comment with the details of which resources need tags, and what the allowed values are. Once fixed, it will let the code be shipped to production.

Try it out by going to https://dashboard.infracost.io/, setting up with the GitHub app or GitLab app, and defining your tagging policy. It will then scan your repository and inform you of any missing tags and their file and line number. You can use the free trial, but if you need more time, please message me and I’ll extend it for you.

I would also love to hear how others ensure that the correct tag keys and values are applied to all resources, and whether this is done proactively or reactively. Additionally, I would be interested in hearing about any lessons learned in the process.

Cheers

Great toolchain, including Infracost or tfsec.

Infracost is an open-source tool for estimating the cost of cloud infrastructure using Infrastructure as Code templates.

Throughout this post, you will see me mention the cost of running this using an estimate. I have been using for a while, a tool called infracost which is an open source (with subscription based additions) cost estimator tool - https://www.infracost.io/. For this demonstration, using the sample code listed above, it would cost an estimated $76.65/month - so if you don't want rack up a bill, only deploy when you want to test, and use Terraform to destroy the services when you are done.

Infracost : Infracost is an open-source tool that allows users to see the cost of running their infrastructure, such as AWS resources, in near real-time. It uses the AWS Price List API to determine the costs of resources, and can be integrated into CI/CD pipelines to provide cost feedback during the development process. This allows developers to make informed decisions about their infrastructure and optimize costs. Additionally, Infracost can be used to create alerts based on cost thresholds, so you can be notified when your infrastructure costs exceed a certain amount. This can be especially useful for teams that operate on a tight budget or need to manage costs closely.

An Infracost API key. You can get one by signing up for free at Infracost.io.

Since Wing compiles to Terraform, you should be able to manually run Infracost (https://www.infracost.io) to get an idea of costs.

Having that capability baked into the language/compiler would be a great addition.

> The best optimization is simply shutting things off

This is the way.

A similar idea has been bouncing around in my mind for a while now. An ideal, turnkey system would do the following:

- Execute via Lambda (serverless).

- Support automated startup and shutdown of various AWS resources on a schedule influenced by specially formatted tags.

- Enable resources to be brought back up out of schedule when demand dictates.

- Operate as a TCP/HTTP proxy that can delay clients so that a given service can be started when it is dormant or, even better, the service isn't serverless but you want it to be. This can't work for everything, but perhaps enough things such that the need to run always on services is reduced.

Cloud Custodian [1] can purportedly do some of this, but I've been reluctant to learn yet another YAML-based DSL to use it.

So this is my "make things designed to be always-on serverless instead" project and the work AWS has done to make Java apps function on Lambda keeps me thinking about the potential to take things that 1) have a relatively long startup time and 2) are designed to be long running service loops, and find a way to force them into the serverless execution model.

[1] https://cloudcustodian.io/

Goal was to clear up anything old and set us up to rotate keys in use. Why did I do it in the end of December? It was a quarterly goal and I learned to push those across the line if I wanted a good review. Great incentive, that one. I used Cloud Custodian for this. It has a terrible bug where the code says you'll be acting on days since the key was used but actually is reading days since it was created.

What is wrong with https://github.com/cloud-custodian/cloud-custodian?

You can start reading about it here.

Use a tool like this https://cloudcustodian.io/ to manage instance on/off hours or go fargate.

Certs aside, there are some great compliance tools out there that you might want to become familiar with. Here is one that comes to mind (is open-source): https://cloudcustodian.io/

I'd use a combination of Cloudcustodian for start/stop scheduling and Apprise for notifications.

Cloud custodian is a good utility if config rules doesn’t satisfy your needs. It’s also cross platform.

AWS Security with open source – Cloud Custodian is a Cloud Security Posture Management (CSPM) tool. CSPM tools evaluate your cloud configuration and identify common configuration mistakes. They also monitor cloud logs to detect threats and configuration changes.