SaaSHub helps you find the best software and product alternatives Learn more →
Top 23 Python S3 Projects
-
airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
-
InfluxDB
InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.
-
awesome-aws
A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.
Inspired by awesome list and donnemartin/awesome-aws.
-
-
s3cmd
Official s3cmd repo -- Command line tool for managing S3 compatible storage services (including Amazon S3 and CloudFront).
-
For me, it's case-by-case. I don't back up the VMs directly, just the date of the stateful applications running on the VMs (or bare metal servers, I do identical stuff for them).
For postgres, I used to just have a systemd timer that would `pg_dumpall` and throw it in s3.
Now I use https://github.com/wal-e/wal-e to backup my postgresql databases.
For other local files, I use borg backup for personal files and services I just run for myself, and I use restic to backup server files to s3.
The operating system's configuration is all stored in git via the magic of NixOS, so I don't have to worry about files in /etc, they all are 100% reproducible from my NixOS configuration.
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
-
-
cloudpathlib
Python pathlib-style classes for cloud storage services such as Amazon S3, Azure Blob Storage, and Google Cloud Storage.
cloudpathlib – pathlib-like interface for cloud storage services
-
s3viewer
Storage Explorer - Publicly open storage viewer (Amazon S3 Bucket, Azure Blob, FTP server, HTTP Index Of/)
-
astro-sdk
Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
-
megabots
🤖 State-of-the-art, production ready LLM apps made mega-easy, so you don't have to build them from scratch 🤯 Create a bot, now 🫵
-
obstore
The simplest, highest-throughput Python interface to S3, GCS & Azure Storage, powered by Rust.
Project mention: Obstore: Efficient streaming object store operations in Python | news.ycombinator.com | 2025-01-16 -
-
-
glacier_deep_archive_backup
Extremely low cost, off-site backup/restore using AWS S3 Glacier Deep Archive
-
amazon-s3-find-and-forget
Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)
-
-
The copy method uses the underlying s3transfer library that ships with boto3 and transparently manages the multipart uploads. I should note here, that despite the name being "multipart upload", we don't need to download the object first before uploading it again, it uses the UploadPartCopy API, which keeps the data internal to S3.
-
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python S3 discussion
Python S3 related posts
-
7 Best Change Data Capture (CDC) Tools in 2025
-
Amazon S3 Tools: Command Line S3 Client and S3 Backup
-
Trouble with s3cmd on M3 Mac
-
A case for moving away from the cloud and embracing local storage solutions
-
browsr 🗂️ a pleasant file explorer on your command line
-
Drupal 9, s3fs, and dynamic node creation
-
How can I handle very large file uploads
-
A note from our sponsor - SaaSHub
www.saashub.com | 16 May 2025
Index
What are some of the best open-source S3 projects in Python? This list will help you:
| # | Project | Stars |
|---|---|---|
| 1 | airbyte | 18,103 |
| 2 | awesome-aws | 12,705 |
| 3 | Moto | 7,863 |
| 4 | s3cmd | 4,707 |
| 5 | wal-e | 3,464 |
| 6 | smart_open | 3,313 |
| 7 | DataEngineeringProject | 1,237 |
| 8 | grafana-backup-tool | 908 |
| 9 | django-s3direct | 665 |
| 10 | cloudpathlib | 542 |
| 11 | s3viewer | 442 |
| 12 | astro-sdk | 369 |
| 13 | megabots | 349 |
| 14 | obstore | 353 |
| 15 | browsr | 327 |
| 16 | dbt-athena | 249 |
| 17 | glacier_deep_archive_backup | 248 |
| 18 | amazon-s3-find-and-forget | 243 |
| 19 | BucketStore | 230 |
| 20 | s3transfer | 221 |
| 21 | s3-credentials | 211 |
| 22 | TileDB-Py | 193 |
| 23 | aioaws | 175 |
Sponsored
InfluxDB – Built for High-Performance Time Series Workloads
InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.
www.influxdata.com