TWINT VS trafilatura

The tool doesn't work anymore since Twitter changed its APIs, but a good example is twint. Most people in OSINT are not highly technical and don't know their way around a CLI. On the other hand, a CLI tool is one of the quickest, lowest (dev) cost ways to release a tool to the public, and many developers who build tools for the OSINT community do so for free (open source).

Related but archived (scraper): https://github.com/twintproject/twint

In the examples for usage if you scroll down all use usernames. I just skimmed over the code and I find references to twitter search api calls. Not sure what they mean by "without using twitter api". Example: https://github.com/twintproject/twint/blob/master/twint/url.py

They do have examples that don't specify a username such as number 3 on this page or this one on the main page: "twint -g="48.880048,2.385939,1km" -o file.csv --csv - Scrape Tweets from a radius of 1km around a place in Paris and export them to a csv file."

this was 2 years ago, but i used to use twint to scrape 10's of millions of tweets. twint didnt require any acct and i used a VPN.

!git clone --depth=1 https://github.com/twintproject/twint.git

Twint is archived and old.

How is your script compared to twint? https://github.com/twintproject/twint

The feature list answers that question pretty well: https://github.com/adbar/trafilatura#features

Basically: you could implement all of this on top of BeautifulSoup - polite crawling policies, sitemap and feed parsing, URL de-duplication, parallel processing, download queues, heuristics for extracting just the main article content, metadata extraction, language detection... but it would require writing an enormous amount of extra code.

The WebScraper tool uses Trafilatura [1] to scrape and parse HTML—nothing too fancy. "Scraping" a React site would require a totally different approach, probably something more akin to Adept's ACT-1 [2].

I run a local chat app built with Griptape and I use it to give me summaries of web pages or answer specific questions all the time :)

1. https://github.com/adbar/trafilatura/

I've been playing with Trafilatura lately, and it's very good. There are a few very thorough comparisons to other projects and it really shines. It doesn't do anything headless from what I can tell, but it doesn't have to do the scraping itself. Maybe an option could be to use Playwright to scrape, then Trafilatura to parse. Food for thought.

Cool! If you care to explain me further... :) ... I tried parsing a page using: https://github.com/adbar/trafilatura, json stringify it and passing it to https://platform.openai.com/docs/api-reference/embeddings/create. How do I use the response as an input later? <3

The only RSS-reader(not an aggregator like freshrss) with a parser included i know of is RSSowlNix (which still works but it's a bit old/not very leightweight). I wrote myself a little dirty workaround in python to solve this problem (since im a noob in PHP). It reads the sqlite database of freshrss, adds a new column to every article to mark articles, that are already parsed and then i use trafilatura to give fulltext parsing a try for every newly added article. To be on the safe site i use it with a proxy. Not all pages work (some javascript gdpr popups are a nightmare), and you cannot jump over paywalls but it works most of the times.