TWINT VS newspaper

twint is a project that can scrape twitter data via the webpages rather than the twitter API, which means that it can get more than the last 3200 tweets of an account. Unfortunately it seems that the repo was archived and is no longer in development, so I'm not sure if it even still works. It's also a bit heavy on dependencies and is written in Python, neither of which make it easier to install and use.

The tool doesn't work anymore since Twitter changed its APIs, but a good example is twint. Most people in OSINT are not highly technical and don't know their way around a CLI. On the other hand, a CLI tool is one of the quickest, lowest (dev) cost ways to release a tool to the public, and many developers who build tools for the OSINT community do so for free (open source).

You can try twint which is extensive and should be able to do that. Another is via this twitter downloader but might require multiple runs depending on what you want to archive.

They do have examples that don't specify a username such as number 3 on this page or this one on the main page: "twint -g="48.880048,2.385939,1km" -o file.csv --csv - Scrape Tweets from a radius of 1km around a place in Paris and export them to a csv file."

This will motivate researchers to web scrape to circumvent these restrictions. Twint can scrape tweets and it supports proxies. It can also be multi threaded. A huge hassle and it's prone to breaking when the site changes.

The comment I saw said they used this: https://github.com/twintproject/twint

I think then you're looking for something similar to newspaper3k. Unfortunately it's written in python. https://github.com/codelucas/newspaper

There are also many ready made libraries available for content extraction written in python which is more commonly used in data science, e.g. goose3 (https://github.com/goose3/goose3) and newspaper (https://github.com/codelucas/newspaper).

I have searched the web but the library I have found was for Python only. I need a library written in Java or Kotlin so that I could use it on Android. Is there any library for that? If you know that there is no such Java library, please let me know that so that I could stop searching.

I have tried several freemium API, but they drop some whole paragraphs of a simple blog article. Next I'm considering to try out https://github.com/codelucas/newspaper , which executes NLP processing.

I don't think there is any library that works better with the Notion API than others. I used https://github.com/codelucas/newspaper a few years back for article extractions and it worked great!

just by looking for a sec on GitHub I found this. The docs are here and at first glance this thing seems to still be maintained and absolutely feature packed

These Python scrapers take RAW RSS 2.0 XML data as input and replace their description s with full source articles parsed/fetched via Newspaper (first scraper) or Article-Parser (second scraper).