aws-lambda-handler-cookbook VS aws-lambda-power-tuning

In case you didn’t know, the Cookbook is a template project that allows you to get started with serverless with three clicks, and it has all the best practices and utilities that a production-grade serverless service requires.

In this two-post series, you will learn to monitor serverless services with CloudWatch by building dashboards, widgets and alarms with AWS CDK. In this post, part two of the series, we will use AWS CDK to monitor a sample serverless service with CloudWatch dashboards according to the principles presented in the first post in the series. In the first post, you will learn what it means to monitor a serverless service, why it is essential, and how to build CloudWatch dashboards to monitor your serverless service with widgets. The widgets display information from CloudWatch logs, metrics, custom metrics, and define CloudWatch alarms as part of a proactive approach. This blog post was originally published on my website, “Ran The Builder.” Introduction Utilizing AWS CloudWatch dashboards enables centralized monitoring of API Gateway, Lambda functions, and DynamoDB, providing real-time insights into their performance and operational health. By aggregating metrics, logs, and alarms, CloudWatch facilitates swift issue diagnosis and analysis across your serverless applications. Additionally, setting up alarms ensures immediate reaction to anomalous activities. In this post, we will write CDK code that builds CloudWatch dashboards that monitor logs and metrics and create alarms for a sample serverless service. We will build a dashboard that monitors a sample serverless service, the ‘orders’ service. These are the resources we will build with AWS CDK. Sample Serverless Service Architecture The ‘orders’ service allows users to order products. Let’s build a monitoring dashboard for this service that implements the concepts introduced in the first part of the series. We aim to monitor the service API gateway, Lambda function, and DynamoDB tables and ensure everything is in order. In addition, we want to visualize service KPI metrics. We will build two CloudWatch dashboards, a high-level summary and a low-level summary, each serving a different persona. The dashboards will display widgets of CloudWatch logs (error logs for our Lambda functions) and CloudWatch metrics of various resources. In addition, we will define CloudWatch alarms to notify us of critical performance degradations and errors. If you wish to understand the reasoning behind this approach and why monitoring is essential, read the first part of this series. You can find the code and the service code here. The code is part of my AWS Lambda Handler cookbook template project. This repository provides a working, deployable, open-source, serverless service template with an AWS Lambda function and AWS CDK Python code with all the best practices and a complete CI/CD pipeline. You can start a serverless service in 3 clicks! CDK Code We will use an open-source library: cdk-monitoring-constructs. The library provides “easy-to-use CDK constructs for monitoring your AWS infrastructure with Amazon CloudWatch.” It abstracts the CW widget creation and simplifies it with out of the box support for many AWS services such as API Gateway, Lambda, DynamoDB and more. The library utilizes the concept of factory classes. You have factory classes for creating a widget based on log group or based on CW metrics for a large number of commonly used AWS services. You can also create CW alarms with ease and monitor custom CW metrics (KPIs). You can use your factory classes to define custom colors, font sizes, sizes, default alarm settings, and more. We will use the library to monitor the ‘orders’ service resources: Lambda function, Api Gateway, and DynamoDB with ease. Below is an L3 CDK construct that builds all the monitoring resources. Let’s review what it creates and deep dive into each section. We will review three different functions that build all the resources. ‘_build_topic’ — builds the SNS topic that alarms will send alarm details when triggered. ‘_build_high_level_dashboard’ — builds the high-level dashboard. ‘_build_low_level_dashboard’ — builds the low-level dashboard. As input, we receive the API Gateway resource, two DynamoDB tables, and a list of Lambda functions to monitor. Let’s go over each of the functions in lines 14–16. Alarms’ Topic CloudWatch alarms are useless unless they have an action once they trigger. We have configured the alarms to send an SNS notification to a new SNS topic. From there, you can use any subscription — HTTPS/SMS/Email, etc. to notify your teams of the alarm. In lines 24–29, we define the KMS key that will be used to encrypt SNS messages at rest. In lines 30–34, we define the topic and use the key we previously described. In lines 37–44, we set a permissions policy and allow CloudWatch to publish messages to the topic. This will occur once an alarm gets triggered; CW would send an SNS message describing the alarm. Now that we have the SNS topic, we can pass it to the following functions used when building CW alarms. High Level Dashboard This dashboard is designed to be an executive overview of the service. Total API gateway metrics provide information on the performance and error rate of the service. It also includes an alarm on the API Gateway error rate. KPI metrics are included in the bottom part as well. Personas that use this dashboard: SRE, developers, and product teams (KPIs). Lets review the function ‘_build_high_level_dashboard’ that generates this dashboard: In lines 26–34, we build the dashboard facade. It represents a dashboard in CW. This class holds all widgets we create and has multiple factory functions that build widgets and alarms. You can override its default settings and set the default factory for alarms, widgets, and metrics with your custom settings. In lines 29–32, we create an alarm factory and set it so that all alarms produced by the facade will have an action to send to the SNS topic we previously defined once triggered. In line 35, we add a header to the dashboard. In lines 36–39, we add multiple widgets that monitor our API Gateway — the top four widgets. They are provided out of the box as part of the library. In line 38, we add an alarm for an error threshold for the API Gateway (HTPP 4XX or 5XX). The alarm will use the default SNS action defined in the previous lines. In lines 40–51, we create the bottom widgets that monitor the custom CloudWatch metric under the namespace ‘orders_kpi’ named ‘ValidCreateOrderEvents.’ We can add multiple metrics to one widget group (line 50), but in the ‘orders’ service, we have only one. Low Level Dashboard It is aimed at a deep dive into all the service’s resources. Requires an understanding of the service architecture and its moving parts. The dashboard provides the Lambda function’s metrics for latency, errors, throttles, provisioned concurrency, and total invocations. In addition, a CloudWatch logs widget shows only ‘error’ logs from the Lambda function. As for DynamoDB tables, we have the primary database and the idempotency table for usage, operation latency, errors, and throttles. Personas that use this dashboard: developers, SREs. Let’s review the CDK code that builds these widgets: In lines 36–44, we build the dashboard facade. It represents a dashboard in CW. This class holds all widgets we create and has multiple factory functions that build widgets and alarms. In lines 39–42, we create an alarm factory and set it so that all alarms produced by the facade will have an action to send to the SNS topic we previously defined once triggered. In line 45, we add a header to the dashboard. In lines 46–56, we build the top two rows of the dashboard. Two per Lambda function — in this case, we have only one function. We use the built-in widget creation for the Lambda function to monitor all the crucial aspects of the function. In lines 47–49, we define an alarm that monitors the p90 duration of the function. If you want to learn more about percentile metrics, check out my first post. In lines 51–56, we define a widget that displays logs from the Lambda function log group but only ERROR logs. In lines 58 and 59, we use the built-in DynamoDB widget support of the library to monitor the main DB and the idempotency table we have. Full CDK Snippet Here’s all the code together: You can find the updated service code here.

You can find the complete code here.

In this post, I’ll guide you through deploying AWS AppConfig configurations using the AWS Cloud Development Kit (CDK) and the new AppConfig L2 constructs (higher abstraction). We will deploy a JSON configuration for feature flags’ usage for serverless and non-serverless applications alike with Python CDK code. This blog post was originally published on my website, “Ran The Builder.” Quick AppConfig Introduction AWS AppConfig is a self-managed service that stores plain TEXT/YAML/JSON configuration to be consumed by multiple clients. Let’s look at AppConfig’s advantages: FedRAMP High certified Fully Serverless Out-of-the-box support for schema validations that run before a configuration update. Out-of-the-box integration with AWS CloudWatch alarms triggers an automatic configuration revert if a configuration update fails your AWS Lambda functions. Read more about it here. You can define configuration deployment strategies. Deployment strategies define how and when to change a configuration. Read more about it here. It provides a single API that fetches configuration. If you’ve read my previous AppConfig posts, you know that I use AWS Lambda Powertools feature flags utility to fetch and evaluate AppConfig configurations. Our Goal We aim to deploy a JSON freeform configuration that we can use with the AWS Lambda Powertools feature flags utility. The construct I present here will deploy a JSON configuration to AppConfig as a freeform configuration with a zero wait time deployment strategy. For production accounts, you should use a canary deployment profile and configure JSON validations and a CloudWatch alarm to trigger an automatic rollback in case of service errors during configuration deployment. If you want to learn about best practices for feature flags and how to use, build, and test feature flags, check out my two posts about them: AWS Lambda Cookbook — Part 6 — Configuration & Feature Flags Best Practices Manage Your AWS Lambda Feature Flags Like a Boss Let’s go ahead and write some code. Python Dependencies To consume the new L2 construct, which contains better abstractions, add the following statements to your poetry file: Be aware that we are using an alpha construct, which might break or contain bugs. Use it at your discretion. However, after testing it, I can confirm it works as expected at the time of writing, and it’s already part of the AWS Lambda Handler Cookbook project. Configuration Construct Let’s go over my new construct, which uses L2 higher abstractions constructs, thus reducing the overall effort to define AppConfig configurations: Our construct requires several input parameters: id_ (str): The scoped construct ID. It must be unique. Environment (str): AppConfig environment name to create. service_name (str): AppConfig application name to create. configuration_name (str): AppConfig configuration name to create. configuration_str (str): AppConfig configuration content to create In lines 33–40, we create the AppConfig application; in line 40, we set the removal policy to destroy. You will have a similar approach applied to all resources. In lines 41–48, we define the AWS AppConfig environment. In lines 51–61, we define the AppConfig deployment strategy. I chose an immediate deployment (zero minutes for bake and deployment times), but you should use the canary deployment options or define your own for production workloads. Read the docs here. In lines 63–75, we define the configuration to deploy and connect all the resources. Notice how I chose line 69, the freeform type. You must use freeform for usage with the Powertools feature flags utility. In line 68, we provide the configuration string we got as a construct input parameter. Line 75 is a workaround until the removal policy is set and exposed correctly (see the link to the CDK issue). Bottom line: The new L2 constructs are a welcome edition as they reduce the total amount of code and the overall complexity, but I suggest that you create your own construct that wraps these L2 constructs into one or use the example I provided here. For other information and connecting advanced features such as JSON validators, extensions, and CloudWatch alarm, refer to the official documentation below: https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_appconfig_alpha.html

The ‘orders’ service is a serverless service template project I created on GitHub. It helps you get started in the serverless domain with all the best practices, a working CI/CD pipeline and CDK infrastructure code.

Let’s create a new Serverless service from my cookiecutter serverless project, based on my AWS Lambda handler cookbook project.

This AWS CDK code defines the variables of the schema ‘MyHandlerEnvVars’ and sets their values. Look specifically at ‘__add_get_lambda_integration’ function.

You can find all examples at this GitHub repository, including CDK deployment code.

Read more about it here and here.

Sometimes, changing services, like opting for HTTP over REST API Gateway, leveraging tools like Lambda Powertuning to optimize functions, or reducing a CloudWatch log retention and changing log level, can lead to significant savings.

In case of not enabling SnapStart for the Lambda function we observed that increasing memory reduces the warm execution time for our use case especially for p>90. As adding more memory to the Lambda function is also a cost factor, the sweet spot between cold and warm start time and cost is somewhere between 768 and 1204 MB memory setting for the Lambda function for our use case. You can use AWS Lambda Power Tuning for very nice visualisations.

The aws lambda power tuning tool helps optimise the Lambda performance and cost in a data-driven manner. Let's try it out:

For Lambda, a very useful tool to help optimise is the AWS Lambda Power Tuning tool, released by Alex Casalboni, Developer Advocate at AWS: https://github.com/alexcasalboni/aws-lambda-power-tuning

Lambda memory affects not only the CPU performance and and host execution priority, but also network performance. Be wary though as the price scales linearly. You can use a tool like Lambda Power Tuning to find the sweet spot for your application. https://github.com/alexcasalboni/aws-lambda-power-tuning

This tool, which is open source and available here, takes the form of a Step Function that is deployed on your AWS account. The purpose of this Step Function is to run your lambda with different memory configurations several times and output a comparison in the form of a graph (or JSON) to try to find the optimal balance between cost and execution time. There are three possible optimization modes: cost, execution time, or a "balanced" mode where it tries to find a balance between the two.

The AWS Documentation's Memory and Computing Power page is a good starting point. To avoid configuring it manually, it's worth checking out AWS Lambda Power Tuning, which will help you find the sweet spot.

Utilizing tools such as AWS X-Ray, AWS Lambda Power Tuning, and AWS Lambda Powertools tracer utility is recommended. Read more about it here.

Install the AWS SAM CLI in your local environment. Configure your AWS credentials (requires AWS CLI installed): $ aws configure Clone this git repository: $ git clone https://github.com/alexcasalboni/aws-lambda-power-tuning.git Build the Lambda layer and any other dependencies (Docker is required): $ cd ./aws-lambda-power-tuning $ sam build -u sam build -u will run SAM build using a Docker container image that provides an environment similar to that which your function would run in. SAM build in-turn looks at your AWS SAM template file for information about Lambda functions and layers in this project. Once the build has completed you should see output that states Build Succeeded. If not there will be error messages providing guidance on what went wrong. Deploy the application using the SAM deploy "guided" mode: $ sam deploy -g

Use AWS Lambda Power Tuning to balance cost and performance.