utls
selectolax
Our great sponsors
utls | selectolax | |
---|---|---|
6 | 6 | |
1,466 | 944 | |
3.8% | - | |
8.1 | 7.7 | |
18 days ago | 18 days ago | |
Go | Cython | |
BSD 3-clause "New" or "Revised" License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
utls
-
Show HN: Golang HTTP Clients and Servers and Alternative Networking
Does this offer any control at the TLS level? I have been looking for alternate TLS stacks for Go, and the only one I really found was uTLS:
-
Easiest way to MITM my browser requests/responses using Go
Something that you'll run into for smarter anti-bot is that your JA3 tls signature will be checked. Easiest way to get around this is probably to use refraction-networking/utls. Especially true as go's tls signature is fairly unique.
-
GET request works only from the browser
Considering that the rocket league api isn't one of tracker.gg's listed apis, they probably enabled a lot of cloudflare'e antibot. If you really care to get around it, you can probably get fairly far with making sure you put the same headers as the chrome version you're trying to mimic. If that still doesn't work, then theres a chance that tracker.gg enabled ja3 fingerprint so you'd need to use a library like utls to fake your clienthello to look like chrome's.
- The State of Web Scraping in 2021
selectolax
-
GitHub – GSA/code-gov: An informative repo for all Code.gov repos
https://github.com/rushter/selectolax#simple-benchmark )
(Apache Nutch is a Java-based web crawler which supports e.g. CommonCrawl (which backs various foundational LLMs)) https://en.wikipedia.org/wiki/Apache_Nutch#Search_engines_bu... . But extruct extracts more types of metadata and data than Nutch AFAIU: https://github.com/scrapinghub/extruct )
datasette-graphql adds a GraphQL HTTP API to a SQLite database:
-
8 Most Popular Python HTML Web Scraping Packages with Benchmarks
selectolax
-
The State of Web Scraping in 2021
Lazyweb link: https://github.com/rushter/selectolax
although I don't follow the need to have what appears to be two completely separate HTML parsing C libraries as dependencies; seeing this in the readme for Modest gives me the shivers because lxml has _seen some shit_
> Modest is a fast HTML renderer implemented as a pure C99 library with no outside dependencies.
although its other dep seems much more cognizant about the HTML5 standard, for whatever that's worth: https://github.com/lexbor/lexbor#lexbor
---
> It looks like the author of the article just googled some libraries for each language and didn't research the topic
Heh, oh, new to the Internet, are you? :-D
What are some alternatives?
lxml - The lxml XML toolkit for Python
lexbor - Lexbor is development of an open source HTML Renderer library. https://lexbor.com
pyppeteer - Headless chrome/chromium automation library (unofficial port of puppeteer)
html5lib - Standards-compliant library for parsing and serializing HTML documents and fragments in Python
go-cloudflare-scraper - A golang http.Transport layer that uses Otto to solve Cloudflare challenges
pyquery - A jquery-like library for python
trojan-go - Go实现的Trojan代理,支持多路复用/路由功能/CDN中转/Shadowsocks混淆插件,多平台,无依赖。A Trojan proxy written in Go. An unidentifiable mechanism that helps you bypass GFW. https://p4gefau1t.github.io/trojan-go/
gazpacho - 🥫 The simple, fast, and modern web scraping library
bleach - Bleach is an allowed-list-based HTML sanitizing library that escapes or strips markup and attributes
xmltodict - Python module that makes working with XML feel like you are working with JSON
xhtml2pdf - A library for converting HTML into PDFs using ReportLab
xmldataset - xmldataset: xml parsing made easy 🗃️