python-readability
trafilatura
Our great sponsors
python-readability | trafilatura | |
---|---|---|
4 | 13 | |
2,457 | 2,656 | |
- | - | |
3.4 | 8.4 | |
9 months ago | about 24 hours ago | |
Python | Python | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
python-readability
We haven't tracked posts mentioning python-readability yet.
Tracking mentions began in Dec 2020.
trafilatura
-
Trafilatura: Python tool to gather text on the Web
The feature list answers that question pretty well: https://github.com/adbar/trafilatura#features
Basically: you could implement all of this on top of BeautifulSoup - polite crawling policies, sitemap and feed parsing, URL de-duplication, parallel processing, download queues, heuristics for extracting just the main article content, metadata extraction, language detection... but it would require writing an enormous amount of extra code.
-
Show HN: Build AI Dags with Memory; Run and Validate LLM Tools in Containers
The WebScraper tool uses Trafilatura [1] to scrape and parse HTML—nothing too fancy. "Scraping" a React site would require a totally different approach, probably something more akin to Adept's ACT-1 [2].
I run a local chat app built with Griptape and I use it to give me summaries of web pages or answer specific questions all the time :)
-
Powerful and free scraper with a headless browser under the hood and Readability for parsing
I've been playing with Trafilatura lately, and it's very good. There are a few very thorough comparisons to other projects and it really shines. It doesn't do anything headless from what I can tell, but it doesn't have to do the scraping itself. Maybe an option could be to use Playwright to scrape, then Trafilatura to parse. Food for thought.
-
I made a Chrome Extension that lets you ask any question about the page you are on (bluf.ai)
Cool! If you care to explain me further... :) ... I tried parsing a page using: https://github.com/adbar/trafilatura, json stringify it and passing it to https://platform.openai.com/docs/api-reference/embeddings/create. How do I use the response as an input later? <3
- Issue with sklearn
- How does Firefox's Reader View work?
-
What's something self hosted everyone needs to run ?
The only RSS-reader(not an aggregator like freshrss) with a parser included i know of is RSSowlNix (which still works but it's a bit old/not very leightweight). I wrote myself a little dirty workaround in python to solve this problem (since im a noob in PHP). It reads the sqlite database of freshrss, adds a new column to every article to mark articles, that are already parsed and then i use trafilatura to give fulltext parsing a try for every newly added article. To be on the safe site i use it with a proxy. Not all pages work (some javascript gdpr popups are a nightmare), and you cannot jump over paywalls but it works most of the times.
What are some alternatives?
newspaper - newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
python-goose - Html Content / Article Extractor, web scrapping lib in Python
TWINT - An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
html2text - Convert HTML to Markdown-formatted text.
textract - extract text from any document. no muss. no fuss.
sumy - Module for automatic summarization of text documents and HTML pages.
Goose3 - A Python 3 compatible version of goose http://goose3.readthedocs.io/en/latest/index.html
micawber - a small library for extracting rich content from urls
opengraph - A python module to parse the Open Graph Protocol