trafilatura VS libreddit

The feature list answers that question pretty well: https://github.com/adbar/trafilatura#features

Basically: you could implement all of this on top of BeautifulSoup - polite crawling policies, sitemap and feed parsing, URL de-duplication, parallel processing, download queues, heuristics for extracting just the main article content, metadata extraction, language detection... but it would require writing an enormous amount of extra code.

The WebScraper tool uses Trafilatura [1] to scrape and parse HTML—nothing too fancy. "Scraping" a React site would require a totally different approach, probably something more akin to Adept's ACT-1 [2].

I run a local chat app built with Griptape and I use it to give me summaries of web pages or answer specific questions all the time :)

1. https://github.com/adbar/trafilatura/

I've been playing with Trafilatura lately, and it's very good. There are a few very thorough comparisons to other projects and it really shines. It doesn't do anything headless from what I can tell, but it doesn't have to do the scraping itself. Maybe an option could be to use Playwright to scrape, then Trafilatura to parse. Food for thought.

Cool! If you care to explain me further... :) ... I tried parsing a page using: https://github.com/adbar/trafilatura, json stringify it and passing it to https://platform.openai.com/docs/api-reference/embeddings/create. How do I use the response as an input later? <3

I want to test how easy it is to install a package plus special extra dependencies to run a certain script in that package: https://github.com/adbar/trafilatura

Don't use Youtube without going through a proxy like Invidious [1] or Newpipe

Don't use {site} Search without going through a proxy like SearxNG [2]

Don't use TwiXXer without going through a proxy like Nitter - this has gotten more difficult lately but it still works as long as you feed the daemon some registered accounts. Video does not work at the moment but that seems to be fixable.

Don't use Reddit without going through a proxy like libreddit [4]

Start noticing the pattern? Maybe it is time to start producing promotional posters:

The only thing to come between you and ADS could be a proxy / ADS. I'ts just not worth the risk

ADS / New rules for a sane net / Sane net protects you, your partner and your community

A proxy here and a filter there, ADS nowhere

The more you tighten your grip, ${site}, the more viewers will slip through your fingers

[1] https://github.com/iv-org/invidious

[2] https://github.com/searxng/searxng

[3] https://github.com/zedeus/nitter

[4] https://github.com/libreddit/libreddit

This is a particular teddit instance shutting down, not teddit the software project.

Stuff is moving a lot currently and there is a bumpy road ahead, but there are several courses of action already being mapped out (scraping html, r-e the private gql api, a setting for operators to put some auth token, ..).

https://github.com/libreddit/libreddit/issues/836

https://codeberg.org/teddit/teddit/issues/400

Sadly the quick refresh is no longer working for me, and going through all of the instances on Libredirect, they all seem to be constantly down now.

It appears that they are trying to deal with this issue here:

https://github.com/libreddit/libreddit/issues/836

Yes, the official Reddit app uses a different endpoint for authentication. There has been great progress in reverse-engineering the private API: https://github.com/libreddit/libreddit/issues/818

There's a fork of Libreddit that uses the API key leaked from the official Reddit app: https://github.com/libreddit/libreddit/pull/819