Python S3

Open-source Python projects categorized as S3

Top 23 Python S3 Projects

  • airbyte

    The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

    Project mention: How to Build a Chat App with Your Postgres Data using Agent Cloud | dev.to | 2024-05-13

    AgentCloud uses Airbyte to build data pipelines, which allow us to split, chunk, and embed data from over 300 data sources, including Postgres.

  • Scout Monitoring

    Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

    Scout Monitoring logo
  • awesome-aws

    A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.

    Project mention: Top Github repositories for 10+ programming languages | dev.to | 2024-07-16

    Awesome AWS

  • Moto

    A library that allows you to easily mock out tests based on AWS infrastructure.

    Project mention: OpenTF Announces Fork of Terraform | news.ycombinator.com | 2023-08-25

    > OpenMoto

    I dunno if you're trying to play on "hashimoto" but https://github.com/getmoto/moto#readme would be a prime name collision for any such "OpenMoto" name

    But yes, please, to adopting Vault. I don't have a horse in the race about Consul but my suspicion is such an effort would only be worthwhile if trying to adopt Nomad, too, which I gravely doubt

  • s3cmd

    Official s3cmd repo -- Command line tool for managing S3 compatible storage services (including Amazon S3 and CloudFront).

    Project mention: Amazon S3 Tools: Command Line S3 Client and S3 Backup | news.ycombinator.com | 2024-01-26
  • wal-e

    Continuous Archiving for Postgres

    Project mention: Run PostgreSQL. The Kubernetes Way | news.ycombinator.com | 2023-09-22

    See the GitHub: https://github.com/wal-e/wal-e

    Unmaintained would’ve made more sense to say, but the maintainer choose the words “obsolete” so I took those. :)

    Seems to be obsolete due to a lack of interest and contributions.

  • smart_open

    Utils for streaming large files (S3, HDFS, gzip, bz2...)

  • DataEngineeringProject

    Example end to end data engineering project.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • grafana-backup-tool

    A Python-based application to backup Grafana settings by using the Grafana API

  • django-s3direct

    Directly upload files to S3 compatible services with Django.

  • s3viewer

    Storage Explorer - Publicly open storage viewer (Amazon S3 Bucket, Azure Blob, FTP server, HTTP Index Of/)

  • megabots

    🤖 State-of-the-art, production ready LLM apps made mega-easy, so you don't have to build them from scratch 🤯 Create a bot, now 🫵

  • astro-sdk

    Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

  • glacier_deep_archive_backup

    Extremely low cost, off-site backup/restore using AWS S3 Glacier Deep Archive

    Project mention: Duplicity | news.ycombinator.com | 2024-01-24

    If you don't need incremental backups (thus saving space for the signatures) and want to store to S3 Deep Glacier, take a look at https://github.com/mrichtarsky/glacier_deep_archive_backup

  • amazon-s3-find-and-forget

    Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)

  • BucketStore

    A simple library for interacting with Amazon S3.

  • browsr

    🗂️ a pleasant file explorer in your terminal supporting all filesystems

  • dbt-athena

    The athena adapter plugin for dbt (https://getdbt.com) (by dbt-athena)

  • s3-credentials

    A tool for creating credentials for accessing S3 buckets

  • TileDB-Py

    Python interface to the TileDB storage engine

  • pathy

    simple, flexible, offline capable, cloud storage with a Python path-like interface

  • aioaws

    Asyncio compatible SDK for aws services.

  • sumologic-aws-lambda

    A collection of lambda functions to collect data from Cloudwatch, Kinesis, VPC Flow logs, S3, security-hub and AWS Inspector

  • s3fs

    Amazon S3 filesystem for PyFilesystem2 (by PyFilesystem)

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python S3 discussion

Log in or Post with

Python S3 related posts

  • Amazon S3 Tools: Command Line S3 Client and S3 Backup

    1 project | news.ycombinator.com | 26 Jan 2024
  • Trouble with s3cmd on M3 Mac

    2 projects | /r/aws | 7 Dec 2023
  • A case for moving away from the cloud and embracing local storage solutions

    3 projects | news.ycombinator.com | 3 Dec 2023
  • browsr 🗂️ a pleasant file explorer on your command line

    2 projects | /r/commandline | 30 May 2023
  • Drupal 9, s3fs, and dynamic node creation

    1 project | /r/drupal | 30 May 2023
  • How can I handle very large file uploads

    1 project | /r/djangolearning | 17 May 2023
  • browsr 🗂️, a pleasant file explorer in your terminal

    2 projects | /r/Python | 3 May 2023
  • A note from our sponsor - SaaSHub
    www.saashub.com | 21 Jul 2024
    SaaSHub helps you find the best software and product alternatives Learn more →

Index

What are some of the best open-source S3 projects in Python? This list will help you:

Project Stars
1 airbyte 14,916
2 awesome-aws 12,280
3 Moto 7,505
4 s3cmd 4,505
5 wal-e 3,436
6 smart_open 3,123
7 DataEngineeringProject 1,032
8 grafana-backup-tool 813
9 django-s3direct 649
10 s3viewer 427
11 megabots 336
12 astro-sdk 330
13 glacier_deep_archive_backup 239
14 amazon-s3-find-and-forget 236
15 BucketStore 226
16 browsr 226
17 dbt-athena 204
18 s3-credentials 191
19 TileDB-Py 182
20 pathy 171
21 aioaws 168
22 sumologic-aws-lambda 151
23 s3fs 149

Sponsored
Free Django app performance insights with Scout Monitoring
Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.
www.scoutapm.com