Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 23 Python S3 Projects
-
airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
-
awesome-aws
A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
s3cmd
Official s3cmd repo -- Command line tool for managing S3 compatible storage services (including Amazon S3 and CloudFront).
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
s3viewer
Storage Explorer - Publicly open storage viewer (Amazon S3 Bucket, Azure Blob, FTP server, HTTP Index Of/)
-
megabots
🤖 State-of-the-art, production ready LLM apps made mega-easy, so you don't have to build them from scratch 🤯 Create a bot, now 🫵
-
astro-sdk
Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.
-
amazon-s3-find-and-forget
Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)
-
glacier_deep_archive_backup
Extremely low cost, off-site backup/restore using AWS S3 Glacier Deep Archive
-
-
-
-
sumologic-aws-lambda
A collection of lambda functions to collect data from Cloudwatch, Kinesis, VPC Flow logs, S3, security-hub and AWS Inspector
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Project mention: Launch HN: Bracket (YC W22) – Two-Way Sync Between Salesforce and Postgres | news.ycombinator.com | 2023-12-12I'l also give a shout-out to Airbyte (https://airbyte.com/), with which I've had some limited success with integrating Salesforce to a local database. The particular pull for Airbyte is that we can self-host the open source version, rather than pay Fivetran a significant sum to do this for us.
It's an immature tool, so I don't yet know that I can claim we've spent _less_ than Fivetran on the additional engineering and ops time, but it feels like it has potential to do so once stabilized.
> OpenMoto
I dunno if you're trying to play on "hashimoto" but https://github.com/getmoto/moto#readme would be a prime name collision for any such "OpenMoto" name
But yes, please, to adopting Vault. I don't have a horse in the race about Consul but my suspicion is such an effort would only be worthwhile if trying to adopt Nomad, too, which I gravely doubt
Project mention: Amazon S3 Tools: Command Line S3 Client and S3 Backup | news.ycombinator.com | 2024-01-26
See the GitHub: https://github.com/wal-e/wal-e
Unmaintained would’ve made more sense to say, but the maintainer choose the words “obsolete” so I took those. :)
Seems to be obsolete due to a lack of interest and contributions.
There's also the S3direct package which wasn't exactly what I wanted but might work for you.
Project mention: 🤖 Release 0.0.11 in Megabots | Memory and Vectorstores are live! | /r/LLMDevs | 2023-04-26
Project mention: Orchestration: Thoughts on Dagster, Airflow and Prefect? | /r/dataengineering | 2023-06-01Have you tried the Astro SDK? https://github.com/astronomer/astro-sdk
If you don't need incremental backups (thus saving space for the signatures) and want to store to S3 Deep Glacier, take a look at https://github.com/mrichtarsky/glacier_deep_archive_backup
Project mention: browsr 🗂️ a pleasant file explorer on your command line | /r/commandline | 2023-06-03Not yet! But you're not the first person to request this. Give this issue a follow to be notified when I release that feature https://github.com/juftin/browsr/issues/20
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python S3 related posts
- Amazon S3 Tools: Command Line S3 Client and S3 Backup
- Trouble with s3cmd on M3 Mac
- A case for moving away from the cloud and embracing local storage solutions
- browsr 🗂️ a pleasant file explorer on your command line
- Drupal 9, s3fs, and dynamic node creation
- How can I handle very large file uploads
- browsr 🗂️, a pleasant file explorer in your terminal
-
A note from our sponsor - InfluxDB
www.influxdata.com | 25 Apr 2024
Index
What are some of the best open-source S3 projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | airbyte | 13,923 |
| 2 | awesome-aws | 12,147 |
| 3 | Moto | 7,374 |
| 4 | s3cmd | 4,418 |
| 5 | wal-e | 3,423 |
| 6 | smart_open | 3,091 |
| 7 | DataEngineeringProject | 985 |
| 8 | grafana-backup-tool | 793 |
| 9 | django-s3direct | 640 |
| 10 | s3viewer | 419 |
| 11 | megabots | 335 |
| 12 | astro-sdk | 317 |
| 13 | amazon-s3-find-and-forget | 232 |
| 14 | glacier_deep_archive_backup | 229 |
| 15 | BucketStore | 223 |
| 16 | browsr | 204 |
| 17 | s3-credentials | 184 |
| 18 | dbt-athena | 183 |
| 19 | TileDB-Py | 178 |
| 20 | pathy | 170 |
| 21 | aioaws | 167 |
| 22 | s3fs | 148 |
| 23 | sumologic-aws-lambda | 148 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com