Twitter Data Pipeline with Apache Airflow + MinIO (S3 compatible Object Storage)

• airflow_minio_twitter_data_pipeline

  1 3 10.0 Python

  A simple example of Data Pipeline using apache-airflow (Orchestrator) and MinIO(Object Storage like s3)

• Pandas

  393 41,863 10.0 Python

  Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more

  

Below is the python Task that transforms the tweets list into a Pandas dataframe, then dumps it in our MinIO Object Storage as a CSV file:

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Docker Compose

  383 32,312 9.6 Go

  Define and run multi-container applications with Docker

  

Then, we’ll use Docker-Compose to easily deploy our code.

• twitter-api-typescript-sdk

  67 898 0.0 TypeScript

  A TypeScript SDK for the Twitter API

  

To learn more about it, I built a Data Pipeline that uses Apache Airflow to pull Elon Musk tweets using the Twitter API and store the result in a CSV stored in a MinIO (OSS alternative to AWS s3) Object Storage bucket.

• Airflow

  169 34,397 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

To learn more about it, I built a Data Pipeline that uses Apache Airflow to pull Elon Musk tweets using the Twitter API and store the result in a CSV stored in a MinIO (OSS alternative to AWS s3) Object Storage bucket.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project