How to scrape Twitter data and do analysis?

This page summarizes the projects mentioned and recommended in the original post on /r/DataHoarder
Web Content Extracting Osint Twitter Python Scrape

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

  • twitterscraper

    Scrape Twitter for Tweets

    • Twitter data tends to get gigantic, and coupled with the one-week-limit of their API it is tricky to acquire suitable historic data. The API actually allows searches further in the past, e.g. https://twitter.com/search?lang=en&q=(%23metoo)%20until%3A2020-01-01%20since%3A2019-01-01&src=typed_query%20until%3A2020-01-01%20since%3A2019-01-01&src=typed_query) But that will not give you the tweets in a form suitable for processing. Tools like TwitterScraper attempt to do this automatically, but Twitter has gotten more strict in its rate limits, and in my experience it basically does not work any more.

  • TWINT

    Discontinued An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.

    • You need Twint. Get it here: https://github.com/twintproject/twint

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Twitter will be purging accounts with no activity for several years soon. We need to archive as many as we can. Any ideas on Methods

    1 project | /r/Archiveteam | 8 May 2023

  • How Do I Use Twint?

    1 project | /r/learnpython | 1 May 2023

  • NYC's transport authority will no longer post service alerts on Twitter

    1 project | /r/technology | 28 Apr 2023

  • What’s currently the best method to archive a twitter account?

    1 project | /r/DataHoarder | 1 Apr 2023

  • Do I have to pay now for the Twitter API if I want to use it for data analysis?

    1 project | /r/datascience | 17 Mar 2023