amazon-s3-find-and-forget
DataEngineeringProject
| amazon-s3-find-and-forget | DataEngineeringProject | |
|---|---|---|
| 3 | 5 | |
| 232 | 985 | |
| 0.9% | - | |
| 7.3 | 0.0 | |
| 8 days ago | over 1 year ago | |
| Python | Python | |
| Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
amazon-s3-find-and-forget
-
Deleting particular data from S3 External Tables
Take a look at this: https://github.com/awslabs/amazon-s3-find-and-forget We use it for GDPR compliance; it will open a file, delete a row and pack it back. It will modify the file so watch out if you are using Glue job bookmarks. Because you are using external tables, the manifest file will also have to be updated with a proper lenght for the new, updated file. If you have hundreds of tables and thousands of files, and you need to do this on a regular basis this would be the scalable solution, but if you have few files honestly I would do it manually
-
Update S3 Files
Have a look at S3 Find and Forget
-
How to handle GDPR requests for data stored in S3 ?
S3 Find and Forget is probably worth looking into, even if just to get ideas on how to implement a similar solution for yourself
DataEngineeringProject
- What are your favourite GitHub repos that shows how data engineering should be done?
- Is it me or are beginner-friendly ETL pipeline guides that explain from the ground-up how to incorporate the use of various technologies notoriously difficult to find.
-
Starting A Data Engineering Project Series
News RSS Feeds
-
5 Data Sources for Data Engineering Projects
Lastly, the most readily available data source would be data scraped from the internet. To be slightly less vague, I have outlined a project that web-scrapes new online articles every ten minutes to provide all the latest news curated into one place. This project utilizes a wide variety of relevant data engineering tools, which makes it a great project example. The author of this project is Damian Kliś, and he outlines his model architecture below:
-
Can You Recommend Good Data Engineering Projects
Here is my project that got me a few interviews so far: https://github.com/damklis/DataEngineeringProject
What are some alternatives?
isp-data-pollution - ISP Data Pollution to Protect Private Browsing History with Obfuscation
blinkist-scraper - 📚 Python tool to download book summaries and audio from Blinkist.com, and generate some pretty output
awesome-aws - A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.
synapse-s3-storage-provider - Synapse storage provider to fetch and store media in Amazon S3
data-toolset - Upgrade from avro-tools and parquet-tools jars to a more user-friendly Python package.
yaetos - Write data & AI pipelines in (SQL, Spark, Pandas) and deploy to the cloud, simplified
s3-credentials - A tool for creating credentials for accessing S3 buckets
Zillow-Data-Engineering
openwisp-monitoring - Network monitoring system written in Python and Django, designed to be extensible, programmable, scalable and easy to use by end users: once the system is configured, monitoring checks, alerts and metric collection happens automatically.
openverse-catalog - Identifies and collects data on cc-licensed content across web crawl data and public apis.
datajob - Build and deploy a serverless data pipeline on AWS with no effort.
cryptostore - A scalable storage service for cryptocurrency data