aws-blog.de-projects VS boto3

With this theory out of the way, let‘s see how we can implement it. I have once again created a repository with a SAM-based app. The app deploys the required Cognito user pool and app client for authentication and grants the Frontend Lambda permission to describe the User Pool and retrieve some secrets at runtime so that they don‘t have to be hardcoded. Almost all the rest happens in the Lambda function.

You can deploy the solution to your own account, as explained in the Github repository. It's basically a combination of sam build and sam deploy --guided because we need SAM to create an ECR repository for the docker image. After the deployment, you need to add credentials to the Parameter Store parameter; otherwise, you won't be able to access the webapp. I didn't include default credentials on purpose.

The code for this setup is available on Github again. I'm not going to walk through all of the authorizer code here, feel free to check out the full implementation, it should be relatively easy to read.

The build_app function creates a Dash app that displays some text, an image, and some dynamic functionality. I've omitted some of the code for brevity, you can find the full code on Github. I wrapped the instantiation in a function that allows me to pass additional arguments to the apps' constructor.

In order to fix these shortcomings, we can extend boto3 to teach it how to deal with more data types than it can on its own. The code, including some tests for it is available on GitHub, and you can jump straight into that if you're impatient, but we're also going to walk through it and talk about some background info that may interest you.

This works well for situations where only one set of credentials can be active at any point in time. There's only one problem here. There is no built-in hook in the lifecycle to decommission or deactivate the old credentials after a grace period, so we have to build our own. Fortunately, the architecture for that isn't too complicated. We extend the four-step lifecycle with a fifth deletePreviousSecret step implemented through custom logic. A self-contained example implementation in Terraform and Python is available here. Let's go through each step in the sequence and see what's happening.

By playing around with the two different implementations, I also learned a few more performance details that surprised me. Let's talk a bit about my experiment setup. The complete code is also available on GitHub if you want to follow along.

Pictures are cheap. Let's look at some code that you can also find on GitHub. You need the AWS SDK for Python and a standard SQS queue to run this yourself. We begin with code that can send a few dummy messages to our SQS queue.

This lambda function sleeps for a random number of seconds between one and ten. It fails in about 15% of all invocations, simulating a broken configuration. In all other cases, it returns the message that processing was successful. Let's look at the implementation now. You can find the complete code on Github if you're interested.

aws-blog.de

Former AWS here.

My literal job for the last part of my time at AWS was "help triage bugs in the AWS SDK." This is by far the best repro I've ever seen for such an in-depth event.

Most of the tickets you get in open ticket trackers are incomplete [ https://github.com/boto/boto3/issues/4011 ] nonsensical [ https://github.com/boto/boto3/issues/4018 ] or weird [ https://github.com/boto/boto3/issues/358 ].

Recently, my colleague brought up the difficulty of using the AWS SDK for Python (Boto3) while working with DynamoDB, especially the cumbersome mapping of AttributeValue objects on the Table operations. One of the easiest ways to get around this difficulty is to switch from the clients interface to the resources interface.

A majority of software in the modern world is built upon various third party packages. These packages help offload work that would otherwise be rather tedious. This includes interacting with cloud APIs, developing scientific applications, or even creating web applications. As you gain experience in python you'll be using more and more of these packages developed by others to power your own code. In this example I've decided to expand our math functionality with NumPy. pdm add is what's used to add dependencies like this to our project:

The updates could be parsed from the github repo's CHANGELOG files (ex: javascript, java, python). I'm picturing an RSS feed generated for a specific language and module (ex: python s3, javascript s3, java sqs)

This can be quite annoying because it makes you wonder why the high-level API isn't able to deal with these common data types. Part of the reason for this is most likely that floats in Python can be counter-intuitive, so Decimal is a better data type if you want numbers to behave as non-computer-scientists expect it. To learn more about these complexities, check out this discussion on GitHub about implementing float support in boto3 and the Python documentation on the subject. Additionally, DynamoDB has no native DateTime data type, so there is no straightforward mapping.

AWS Data Wrangler is a Python library that simplifies the process of interacting with various AWS services, built on top of some useful data tools and open-source projects such as Pandas, Apache Arrow and Boto3. It offers streamlined functions to connect to, retrieve, transform, and load data from AWS services, with a strong focus on Amazon S3.

Alternatively, you could create a Python script using either Boto3 or her asynchronous sister, aioBoto3 that will spin through the contents of the origin bucket and move it over to the destination.

Once my site was stood up, I needed to build out the user count API. Through the console, I set up a DynamoDB table and created a user count item. Getting my lambda to interface with AWS resources was a breeze with the Boto3 SDK. You can see my Python code that increments the user count whenever someone visits the site here. The key is the usage of the update_item method that comes from Boto3.

If you want to get a feel for what kind of logging and how much logging is done in projects, boto3 is a very widely used SDK created by Amazon: https://github.com/boto/boto3