Python S3

Open-source Python projects categorized as S3

Top 23 Python S3 Projects

  1. airbyte

    The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

    Project mention: Personal Picks: Data Product News (April 16, 2025) | dev.to | 2025-04-15
  2. InfluxDB

    InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

    InfluxDB logo
  3. awesome-aws

    A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.

    Project mention: Awesome Kubernetes Resources !!! 🔥 | dev.to | 2025-03-18

    Inspired by awesome list and donnemartin/awesome-aws.

  4. Moto

    A library that allows you to easily mock out tests based on AWS infrastructure.

    Project mention: Testando código que chama serviços da AWS | dev.to | 2025-02-04
  5. s3cmd

    Official s3cmd repo -- Command line tool for managing S3 compatible storage services (including Amazon S3 and CloudFront).

  6. wal-e

    Continuous Archiving for Postgres

    Project mention: Ask HN: What do you use to backup your VMs? | news.ycombinator.com | 2024-09-27

    For me, it's case-by-case. I don't back up the VMs directly, just the date of the stateful applications running on the VMs (or bare metal servers, I do identical stuff for them).

    For postgres, I used to just have a systemd timer that would `pg_dumpall` and throw it in s3.

    Now I use https://github.com/wal-e/wal-e to backup my postgresql databases.

    For other local files, I use borg backup for personal files and services I just run for myself, and I use restic to backup server files to s3.

    The operating system's configuration is all stored in git via the magic of NixOS, so I don't have to worry about files in /etc, they all are 100% reproducible from my NixOS configuration.

  7. smart_open

    Utils for streaming large files (S3, HDFS, gzip, bz2...)

  8. DataEngineeringProject

    Example end to end data engineering project.

  9. SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
  10. grafana-backup-tool

    A Python-based application to backup Grafana settings by using the Grafana API

  11. django-s3direct

    Directly upload files to S3 compatible services with Django.

  12. cloudpathlib

    Python pathlib-style classes for cloud storage services such as Amazon S3, Azure Blob Storage, and Google Cloud Storage.

    Project mention: Thiw Week In Python | dev.to | 2025-03-28

    cloudpathlib – pathlib-like interface for cloud storage services

  13. s3viewer

    Storage Explorer - Publicly open storage viewer (Amazon S3 Bucket, Azure Blob, FTP server, HTTP Index Of/)

  14. astro-sdk

    Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

  15. megabots

    🤖 State-of-the-art, production ready LLM apps made mega-easy, so you don't have to build them from scratch 🤯 Create a bot, now 🫵

  16. obstore

    The simplest, highest-throughput Python interface to S3, GCS & Azure Storage, powered by Rust.

    Project mention: Obstore: Efficient streaming object store operations in Python | news.ycombinator.com | 2025-01-16
  17. browsr

    🗂️ a pleasant file explorer in your terminal supporting all filesystems

  18. dbt-athena

    The athena adapter plugin for dbt (https://getdbt.com)

  19. glacier_deep_archive_backup

    Extremely low cost, off-site backup/restore using AWS S3 Glacier Deep Archive

  20. amazon-s3-find-and-forget

    Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)

  21. BucketStore

    A simple library for interacting with Amazon S3.

  22. s3transfer

    Amazon S3 Transfer Manager for Python

    Project mention: Enabling Apache Airflow to copy large S3 objects | dev.to | 2024-08-26

    The copy method uses the underlying s3transfer library that ships with boto3 and transparently manages the multipart uploads. I should note here, that despite the name being "multipart upload", we don't need to download the object first before uploading it again, it uses the UploadPartCopy API, which keeps the data internal to S3.

  23. s3-credentials

    A tool for creating credentials for accessing S3 buckets

  24. TileDB-Py

    Python interface to the TileDB storage engine

  25. aioaws

    Asyncio compatible SDK for aws services.

  26. SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python S3 discussion

Log in or Post with

Python S3 related posts

  • 7 Best Change Data Capture (CDC) Tools in 2025

    2 projects | dev.to | 11 Apr 2025
  • Amazon S3 Tools: Command Line S3 Client and S3 Backup

    1 project | news.ycombinator.com | 26 Jan 2024
  • Trouble with s3cmd on M3 Mac

    2 projects | /r/aws | 7 Dec 2023
  • A case for moving away from the cloud and embracing local storage solutions

    3 projects | news.ycombinator.com | 3 Dec 2023
  • browsr 🗂️ a pleasant file explorer on your command line

    2 projects | /r/commandline | 30 May 2023
  • Drupal 9, s3fs, and dynamic node creation

    1 project | /r/drupal | 30 May 2023
  • How can I handle very large file uploads

    1 project | /r/djangolearning | 17 May 2023
  • A note from our sponsor - SaaSHub
    www.saashub.com | 16 May 2025
    SaaSHub helps you find the best software and product alternatives Learn more →

Index

What are some of the best open-source S3 projects in Python? This list will help you:

# Project Stars
1 airbyte 18,103
2 awesome-aws 12,705
3 Moto 7,863
4 s3cmd 4,707
5 wal-e 3,464
6 smart_open 3,313
7 DataEngineeringProject 1,237
8 grafana-backup-tool 908
9 django-s3direct 665
10 cloudpathlib 542
11 s3viewer 442
12 astro-sdk 369
13 megabots 349
14 obstore 353
15 browsr 327
16 dbt-athena 249
17 glacier_deep_archive_backup 248
18 amazon-s3-find-and-forget 243
19 BucketStore 230
20 s3transfer 221
21 s3-credentials 211
22 TileDB-Py 193
23 aioaws 175

Sponsored
InfluxDB – Built for High-Performance Time Series Workloads
InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.
www.influxdata.com