Top 18 Web Content Extracting Open-Source Projects

• newspaper

  13 13,737 0.0 Python

  newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

• python-goose

  0 3,942 0.0 HTML

  Html Content / Article Extractor, web scrapping lib in Python

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• textract

  4 3,784 3.5 HTML

  extract text from any document. no muss. no fuss.

• toapi

  0 3,462 0.0 Python

  Every web site provides APIs.

• sumy

  1 3,419 6.7 Python

  Module for automatic summarization of text documents and HTML pages.

• trafilatura

  13 2,853 8.7 Python

  Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

Project mention: Trafilatura: Python tool to gather text on the Web | news.ycombinator.com | 2023-08-14

The feature list answers that question pretty well: https://github.com/adbar/trafilatura#features

Basically: you could implement all of this on top of BeautifulSoup - polite crawling policies, sitemap and feed parsing, URL de-duplication, parallel processing, download queues, heuristics for extracting just the main article content, metadata extraction, language detection... but it would require writing an enormous amount of extra code.

• python-readability

  5 2,568 3.4 Python

  fast python port of arc90's readability tool, updated to match latest readability.js!

Project mention: Show HN: I made a tool to clean and convert any webpage to Markdown | news.ycombinator.com | 2024-04-14

One of the cases when AI not needed. There is very good working algorithm to extract content from the pages, one of implementations: https://github.com/buriy/python-readability

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• html2text

  0 1,664 6.1 Python

  Convert HTML to Markdown-formatted text. (by Alir3z4)

• Goose3

  2 765 6.4 HTML

  A Python 3 compatible version of goose http://goose3.readthedocs.io/en/latest/index.html

  

• micawber

  0 622 4.8 Python

  a small library for extracting rich content from urls

• lassie

  1 600 0.0 HTML

  Web Content Retrieval for Humans™

• inscriptis

  0 233 8.5 Python

  A python based HTML to text conversion library, command line client and Web service.

  

• opengraph

  165 224 0.0 Python

  A python module to parse the Open Graph Protocol

Project mention: Add Thumbnails to your project links for better SEO | dev.to | 2024-05-01

OpenGraph docs

• Haul

  0 157 0.0 Python

  An Extensible Image Crawler (by vinta)

• htmldate

  1 107 7.6 Python

  Fast and robust date extraction from web pages, with Python or on the command-line

• sanitize

  0 64 0.0 Python

  Bringing sanity to world of messed-up data

• JSONPATH

  0 37 5.7 Python

  A query expression for extracting data from JSON. (by linw1995)

• Data Extractor

  0 27 6.0 Python

  Combine XPath, CSS Selectors and JSONPath for Web data extracting.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Web Content Extracting related posts

• Add Thumbnails to your project links for better SEO

  2 projects | dev.to | 1 May 2024

• Como customizar o preview de links em redes sociais no Next.js

  1 project | dev.to | 20 Mar 2024

• Building an SEO-friendly responsive i18n website using Vite-SSG + Vuetify3

  1 project | dev.to | 17 Mar 2024

• Java virtual threads caused a deadlock in TPC-C for PostgreSQL

  4 projects | news.ycombinator.com | 15 Jan 2024

• Is there a reason why cover art is not showing up?

  1 project | /r/AO3 | 8 Dec 2023

• What is an open graph? You must know this feature in web development.

  1 project | dev.to | 23 Oct 2023

• Making Dynamic Website Thumbnail

  4 projects | dev.to | 21 Sep 2023
• A note from our sponsor - InfluxDB
  www.influxdata.com | 4 May 2024

  Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com