boto3 VS cloudformation-guard

Former AWS here.

My literal job for the last part of my time at AWS was "help triage bugs in the AWS SDK." This is by far the best repro I've ever seen for such an in-depth event.

Most of the tickets you get in open ticket trackers are incomplete [ https://github.com/boto/boto3/issues/4011 ] nonsensical [ https://github.com/boto/boto3/issues/4018 ] or weird [ https://github.com/boto/boto3/issues/358 ].

Recently, my colleague brought up the difficulty of using the AWS SDK for Python (Boto3) while working with DynamoDB, especially the cumbersome mapping of AttributeValue objects on the Table operations. One of the easiest ways to get around this difficulty is to switch from the clients interface to the resources interface.

A majority of software in the modern world is built upon various third party packages. These packages help offload work that would otherwise be rather tedious. This includes interacting with cloud APIs, developing scientific applications, or even creating web applications. As you gain experience in python you'll be using more and more of these packages developed by others to power your own code. In this example I've decided to expand our math functionality with NumPy. pdm add is what's used to add dependencies like this to our project:

The updates could be parsed from the github repo's CHANGELOG files (ex: javascript, java, python). I'm picturing an RSS feed generated for a specific language and module (ex: python s3, javascript s3, java sqs)

This can be quite annoying because it makes you wonder why the high-level API isn't able to deal with these common data types. Part of the reason for this is most likely that floats in Python can be counter-intuitive, so Decimal is a better data type if you want numbers to behave as non-computer-scientists expect it. To learn more about these complexities, check out this discussion on GitHub about implementing float support in boto3 and the Python documentation on the subject. Additionally, DynamoDB has no native DateTime data type, so there is no straightforward mapping.

AWS Data Wrangler is a Python library that simplifies the process of interacting with various AWS services, built on top of some useful data tools and open-source projects such as Pandas, Apache Arrow and Boto3. It offers streamlined functions to connect to, retrieve, transform, and load data from AWS services, with a strong focus on Amazon S3.

Alternatively, you could create a Python script using either Boto3 or her asynchronous sister, aioBoto3 that will spin through the contents of the origin bucket and move it over to the destination.

Once my site was stood up, I needed to build out the user count API. Through the console, I set up a DynamoDB table and created a user count item. Getting my lambda to interface with AWS resources was a breeze with the Boto3 SDK. You can see my Python code that increments the user count whenever someone visits the site here. The key is the usage of the update_item method that comes from Boto3.

If you want to get a feel for what kind of logging and how much logging is done in projects, boto3 is a very widely used SDK created by Amazon: https://github.com/boto/boto3

If you now use these services to fix the infrastructure findings, a drift occurs that is not always easy to fix. It is better to check for possible problems before the actual deployment. This approach is called “Shift-Left”. This can be done with the package cdk-validator-cfnguard. It's based on the CloudFormation Guard package.

AWS Config rules allow you to determine if a resource is compliant or not. Previously when you wanted to do custom checks you needed to write AWS Lambda functions to validate the configuration of a resource. Since Aug 2, 2022 you have the ability to use cfn-guard rules to achieve the same.

In my previous blog, How do you prove that your infrastructure is compliant. I explained how you can prove your infrastructure is compliant using CloudFormation Guard. But, how do you write those rules? And even more important, how do you test your rules? If you look at the repository CloudFormation Guard. You will notice that the project itself offers a testing framework. Alright! Let’s build a ruleset and write some tests for it!

When you use CloudFormation Guard in combination with CodeBuild Reports it makes it easier to see what rules have failed and keeps a history. When you have a solid set of compliance rules. It gives you a report that you can use to prove that the build of the infrastructure was compliant. You are also able to prevent non-compliant code rollout in production.

cloudformation-guard.

AWS CloudFormation: can help with deploying compliant stacks. You can make sure that a stack is compliant by using AWS CloudFormation guard.

See https://github.com/aws-cloudformation/cloudformation-guard

Currently, we're also exploring the brand new AWS Config rules backed by guard. Now you can write rules using guard which is a policy-as-code language. Here is some example of a Guard Rule which we are testing.

https://github.com/aws-cloudformation/cloudformation-guard is also very useful, but more so when you want to keep your templates consistent to standards.