PDM VS boto3

# See https://pre-commit.com for more information # See https://pre-commit.com/hooks.html for more hooks repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v3.2.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - id: check-yaml - id: check-toml - id: check-added-large-files - repo: local hooks: - id: tox lint name: tox-validation entry: pdm run tox -e test,lint language: system files: ^src\/.+py$|pyproject.toml|^tests\/.+py$ types_or: [python, toml] pass_filenames: false - id: tox docs name: tox-docs language: system entry: pdm run tox -e docs types_or: [python, rst, toml] files: ^src\/.+py$|pyproject.toml|^docs\/ pass_filenames: false - repo: https://github.com/pdm-project/pdm rev: 2.10.4 # a PDM release exposing the hook hooks: - id: pdm-lock-check - repo: https://github.com/jumanjihouse/pre-commit-hooks rev: 3.0.0 hooks: - id: markdownlint

PDM is a solution that allows for easy creation and management of python projects. Some of the key features that will improve the management of python projects include:

You'd need a proper dependencies management tool like PDM or Poetry to exhaustively resolve and lock down all the transitive dependencies if you want to have anything closed to reproducible build.

Are you working on a Python project that uses pdm for dependency management and dotenv for local environment variable and secrets management? Do you find it frustrating when CLI tools like pgcli don't automatically pick up your .env file, forcing you to resort to npm install -g dotenv-cli? I've got a more convenient solution for you!

I first learned about PDM from a blog post written by one of the PDM contributers. The post was about OOPifying argparse to allow for easy creation/modification of subcommands that exist as their own classes/files, and to avoid maintaining a single long script with an endless number of subparser.add_argument(...) lines.

That's more or less what PEP 582 plans to do, but it's been stalled and mired in discussions for years. The PDM tool went ahead and implemented it though if you want to use it: https://github.com/pdm-project/pdm

pdm – A modern Python package and dependency manager

pdm and PEP 582 enter the chat

I'm liking PDM for a while now. Quicker than Poetry and built according to the Python package spec in mind and not as an afterthought. While it was originally meant to work with PEP 582, it works with virtual environments too (now default).

https://github.com/pdm-project/pdm

Former AWS here.

My literal job for the last part of my time at AWS was "help triage bugs in the AWS SDK." This is by far the best repro I've ever seen for such an in-depth event.

Most of the tickets you get in open ticket trackers are incomplete [ https://github.com/boto/boto3/issues/4011 ] nonsensical [ https://github.com/boto/boto3/issues/4018 ] or weird [ https://github.com/boto/boto3/issues/358 ].

Recently, my colleague brought up the difficulty of using the AWS SDK for Python (Boto3) while working with DynamoDB, especially the cumbersome mapping of AttributeValue objects on the Table operations. One of the easiest ways to get around this difficulty is to switch from the clients interface to the resources interface.

A majority of software in the modern world is built upon various third party packages. These packages help offload work that would otherwise be rather tedious. This includes interacting with cloud APIs, developing scientific applications, or even creating web applications. As you gain experience in python you'll be using more and more of these packages developed by others to power your own code. In this example I've decided to expand our math functionality with NumPy. pdm add is what's used to add dependencies like this to our project:

The updates could be parsed from the github repo's CHANGELOG files (ex: javascript, java, python). I'm picturing an RSS feed generated for a specific language and module (ex: python s3, javascript s3, java sqs)

This can be quite annoying because it makes you wonder why the high-level API isn't able to deal with these common data types. Part of the reason for this is most likely that floats in Python can be counter-intuitive, so Decimal is a better data type if you want numbers to behave as non-computer-scientists expect it. To learn more about these complexities, check out this discussion on GitHub about implementing float support in boto3 and the Python documentation on the subject. Additionally, DynamoDB has no native DateTime data type, so there is no straightforward mapping.

AWS Data Wrangler is a Python library that simplifies the process of interacting with various AWS services, built on top of some useful data tools and open-source projects such as Pandas, Apache Arrow and Boto3. It offers streamlined functions to connect to, retrieve, transform, and load data from AWS services, with a strong focus on Amazon S3.

Alternatively, you could create a Python script using either Boto3 or her asynchronous sister, aioBoto3 that will spin through the contents of the origin bucket and move it over to the destination.

Once my site was stood up, I needed to build out the user count API. Through the console, I set up a DynamoDB table and created a user count item. Getting my lambda to interface with AWS resources was a breeze with the Boto3 SDK. You can see my Python code that increments the user count whenever someone visits the site here. The key is the usage of the update_item method that comes from Boto3.

If you want to get a feel for what kind of logging and how much logging is done in projects, boto3 is a very widely used SDK created by Amazon: https://github.com/boto/boto3