Twitter Data Pipeline with Apache Airflow + MinIO (S3 compatible Object Storage)

• airflow_minio_twitter_data_pipeline

  1 3 10.0 Python

  A simple example of Data Pipeline using apache-airflow (Orchestrator) and MinIO(Object Storage like s3)

• Pandas

  163 37,283 10.0 Python

  Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more

  

  Below is the python Task that transforms the tweets list into a Pandas dataframe, then dumps it in our MinIO Object Storage as a CSV file:

• InfluxDB

  www.influxdata.com

  sponsored

  Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

  

• Docker Compose

  145 28,945 9.8 Go

  Define and run multi-container applications with Docker

  

  Then, we’ll use Docker-Compose to easily deploy our code.

• twitter-api-typescript-sdk

  62 753 6.4 TypeScript

  A TypeScript SDK for the Twitter API

  

  To learn more about it, I built a Data Pipeline that uses Apache Airflow to pull Elon Musk tweets using the Twitter API and store the result in a CSV stored in a MinIO (OSS alternative to AWS s3) Object Storage bucket.

• Airflow

  70 29,553 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

  To learn more about it, I built a Data Pipeline that uses Apache Airflow to pull Elon Musk tweets using the Twitter API and store the result in a CSV stored in a MinIO (OSS alternative to AWS s3) Object Storage bucket.

• Sonar

  www.sonarsource.com

  sponsored

  Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.

  

Suggest a related project