Python Web Content Extracting

Open-source Python projects categorized as Web Content Extracting
Edit details
Topics: Python Crawler NLP HTML News
The modern identity platform for B2B SaaS
The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
workos.com
Sponsored

Top 14 Python Web Content Extracting Projects

  • newspaper

    newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

  • toapi

    Every web site provides APIs.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • sumy

    Module for automatic summarization of text documents and HTML pages.

  • trafilatura

    Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

    • Project mention: Trafilatura: Python tool to gather text on the Web | news.ycombinator.com | 2023-08-14

    The feature list answers that question pretty well: https://github.com/adbar/trafilatura#features

    Basically: you could implement all of this on top of BeautifulSoup - polite crawling policies, sitemap and feed parsing, URL de-duplication, parallel processing, download queues, heuristics for extracting just the main article content, metadata extraction, language detection... but it would require writing an enormous amount of extra code.

  • python-readability

    fast python port of arc90's readability tool, updated to match latest readability.js!

    • Project mention: Show HN: I made a tool to clean and convert any webpage to Markdown | news.ycombinator.com | 2024-04-14

    One of the cases when AI not needed. There is very good working algorithm to extract content from the pages, one of implementations: https://github.com/buriy/python-readability

  • html2text

    Convert HTML to Markdown-formatted text. (by Alir3z4)

  • micawber

    a small library for extracting rich content from urls

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo

  • inscriptis

    A python based HTML to text conversion library, command line client and Web service.

  • opengraph

    A python module to parse the Open Graph Protocol

    • Project mention: Como customizar o preview de links em redes sociais no Next.js | dev.to | 2024-03-20

  • Haul

    An Extensible Image Crawler (by vinta)

  • htmldate

    Fast and robust date extraction from web pages, with Python or on the command-line

  • sanitize

    Bringing sanity to world of messed-up data

  • JSONPATH

    A query expression for extracting data from JSON. (by linw1995)

  • Data Extractor

    Combine XPath, CSS Selectors and JSONPath for Web data extracting.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python Web Content Extracting related posts

Index

What are some of the best open-source Web Content Extracting projects in Python? This list will help you:

Project Stars
1 newspaper 13,720
2 toapi 3,462
3 sumy 3,417
4 trafilatura 2,778
5 python-readability 2,563
6 html2text 1,655
7 micawber 622
8 inscriptis 233
9 opengraph 224
10 Haul 157
11 htmldate 106
12 sanitize 64
13 JSONPATH 37
14 Data Extractor 27

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com