metascraper VS TWINT

Yep but there is a fallback to metascraper [0] which does check the HTML tags. However the fallback didn't work in case GPT returns a 1970 date -- I just fixed this! [1]

I think you can now remove the date from your post content and it should still work. If you submit your website again it should do a re-scrape if you changed the content text. Thanks for catching this :)

[0] https://metascraper.js.org/#/

[1] https://github.com/lindylearn/aboutideasnow/commit/8b0ea5b46...

200 {description: 'easily scrape metadata from an article on the web.', publisher: null, title: 'metascraper, easily scrape metadata from an article on the web.', url: 'https://metascraper.js.org'} [[Prototype]]: Object

I am using this library https://github.com/microlinkhq/metascraper but it doesn't catch it.

First of all, we'll use the got npm package to fetch the website content (feel free to use any other fetching library), and the metascraper npm package to extract the metadata:

> After that, pricing starts at $25 per month for up to 15,000 requests.

This is very expensive for any decent usage. I have used tools like metascraper for this purpose and it worked pretty well. Setup just requires throwing a tiny nodejs app on a raspberry pi or $5 server and that can handle tons of requests.

https://github.com/microlinkhq/metascraper

twint is a project that can scrape twitter data via the webpages rather than the twitter API, which means that it can get more than the last 3200 tweets of an account. Unfortunately it seems that the repo was archived and is no longer in development, so I'm not sure if it even still works. It's also a bit heavy on dependencies and is written in Python, neither of which make it easier to install and use.

The tool doesn't work anymore since Twitter changed its APIs, but a good example is twint. Most people in OSINT are not highly technical and don't know their way around a CLI. On the other hand, a CLI tool is one of the quickest, lowest (dev) cost ways to release a tool to the public, and many developers who build tools for the OSINT community do so for free (open source).

You can try twint which is extensive and should be able to do that. Another is via this twitter downloader but might require multiple runs depending on what you want to archive.

They do have examples that don't specify a username such as number 3 on this page or this one on the main page: "twint -g="48.880048,2.385939,1km" -o file.csv --csv - Scrape Tweets from a radius of 1km around a place in Paris and export them to a csv file."

This will motivate researchers to web scrape to circumvent these restrictions. Twint can scrape tweets and it supports proxies. It can also be multi threaded. A huge hassle and it's prone to breaking when the site changes.

The comment I saw said they used this: https://github.com/twintproject/twint