|2 months ago||8 months ago|
|MIT License||MIT License|
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
GitHub – GSA/code-gov: An informative repo for all Code.gov repos
12 projects | news.ycombinator.com | 9 Sep 2023
(Apache Nutch is a Java-based web crawler which supports e.g. CommonCrawl (which backs various foundational LLMs)) https://en.wikipedia.org/wiki/Apache_Nutch#Search_engines_bu... . But extruct extracts more types of metadata and data than Nutch AFAIU: https://github.com/scrapinghub/extruct )
datasette-graphql adds a GraphQL HTTP API to a SQLite database:
8 Most Popular Python HTML Web Scraping Packages with Benchmarks
4 projects | dev.to | 1 Feb 2023
The State of Web Scraping in 2021
Lazyweb link: https://github.com/rushter/selectolax
although I don't follow the need to have what appears to be two completely separate HTML parsing C libraries as dependencies; seeing this in the readme for Modest gives me the shivers because lxml has _seen some shit_
> Modest is a fast HTML renderer implemented as a pure C99 library with no outside dependencies.
although its other dep seems much more cognizant about the HTML5 standard, for whatever that's worth: https://github.com/lexbor/lexbor#lexbor
> It looks like the author of the article just googled some libraries for each language and didn't research the topic
Heh, oh, new to the Internet, are you? :-D
will requests-html library work as selenium
5 projects | /r/Python | 13 Feb 2023
Last I checked, pyppeteer wasn't a thing anymore, and I haven't tried Playwright but if it has a headless mode, thats what you want so you don't have a browser open.
What have you automated with python?
2 projects | /r/Python | 28 Jan 2023
Note, the first time you ever run the render() method, it will download Chromium into your home directory (e.g. ~/.pyppeteer/). This only happens once.
4 projects | /r/programmingcirclejerk | 28 Jul 2022
How to start Web scraping with python?
2 projects | /r/learnpython | 22 Nov 2021
PyAutoGUI with CSS Selector
2 projects | /r/learnpython | 28 Oct 2021
If you're talking about CSS I reckon you want to click/input things on a website inside your browser. In this case you would use a web driver which can automate a web browser like Chrome or Firefox. Something like Helium, Selenium or pyppeteer.
The State of Web Scraping in 2021
In my own experience puppeteer is much better/capable than selenium but the problem is that puppeteer requires nodejs. its python-wrapper https://github.com/pyppeteer/pyppeteer was not as good as selenium when you like to use python.
Pyppetteer is feature complete and worth noting: https://github.com/pyppeteer/pyppeteer
Scraping data from interative web charts python
3 projects | /r/learnpython | 29 Jul 2021
Scrape Google Ad Results with Python
2 projects | dev.to | 18 May 2021
using headless browser or browser automation frameworks, such as * selenium or pyppeteer.
Web Scraping 101 with Python
5 projects | news.ycombinator.com | 10 Feb 2021
What are some alternatives?
puppeteer - Node.js API for Chrome
Scrapy - Scrapy, a fast high-level web crawling & scraping framework for Python.
Playwright - Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
lxml - The lxml XML toolkit for Python
playwright-python - Python version of the Playwright testing and automation library.
selenium-python-helium - Selenium-python but lighter: Helium is the best Python library for web automation.
requests - A simple, yet elegant, HTTP library.
lexbor - Lexbor is development of an open source HTML Renderer library. https://lexbor.com
html5lib - Standards-compliant library for parsing and serializing HTML documents and fragments in Python
pyquery - A jquery-like library for python
bleach - Bleach is an allowed-list-based HTML sanitizing library that escapes or strips markup and attributes
gazpacho - 🥫 The simple, fast, and modern web scraping library