utls
lambdasoup
Our great sponsors
| utls | lambdasoup | |
|---|---|---|
| 6 | 3 | |
| 1,495 | 375 | |
| 3.6% | - | |
| 8.1 | 2.4 | |
| 13 days ago | 7 days ago | |
| Go | OCaml | |
| BSD 3-clause "New" or "Revised" License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
utls
- uTLS – Go TLS fork with low-level access to ClientHello for mimicry purposes
-
Show HN: Golang HTTP Clients and Servers and Alternative Networking
Does this offer any control at the TLS level? I have been looking for alternate TLS stacks for Go, and the only one I really found was uTLS:
https://github.com/refraction-networking/utls
-
Easiest way to MITM my browser requests/responses using Go
Something that you'll run into for smarter anti-bot is that your JA3 tls signature will be checked. Easiest way to get around this is probably to use refraction-networking/utls. Especially true as go's tls signature is fairly unique.
-
GET request works only from the browser
Considering that the rocket league api isn't one of tracker.gg's listed apis, they probably enabled a lot of cloudflare'e antibot. If you really care to get around it, you can probably get fairly far with making sure you put the same headers as the chrome version you're trying to mimic. If that still doesn't work, then theres a chance that tracker.gg enabled ja3 fingerprint so you'd need to use a library like utls to fake your clienthello to look like chrome's.
- The State of Web Scraping in 2021
-
ClientHello mimicking in Rust
do we have something like https://github.com/refraction-networking/utls but for rust?
lambdasoup
-
Soupault: A static website management tool
I'm using soupault right now to make a simple company wiki (under a dozen pages). I like how it's HTML-first and easily customizable, compared to other static site generators that come with too many bells and whistles. Although now I have to make my own image compression script...
Also, fun fact: soupault is written in OCaml, which apparently has a really nice library for HTML manipulation: https://github.com/aantron/lambdasoup
-
The State of Web Scraping in 2021
OCaml’s Lambda Soup (https://aantron.github.io/lambdasoup/) is a amazing library/, especially for those that prefer functional programming
-
Soupault (soup-oh) is a tool that helps you create and manage static websites
It's used for sorting "widgets" (page processing steps) according to dependency lists that users can specify in the config (like `after = ["foo", "bar"]`).
Other than that, one thing I really like about OCaml is that the compiler team and most library maintainers are considerate towards downstream users with respect to compatibility.
The Lua interpreter [3] that soupault uses for its plugin API is a revived 20 year old research project. It only needed minor modifications to build with recent compiler versions.
[1] https://github.com/aantron/lambdasoup
What are some alternatives?
go-cloudflare-scraper - A golang http.Transport layer that uses Otto to solve Cloudflare challenges
soupault - Static website generator based on HTML element tree rewriting
trojan-go - Go实现的Trojan代理，支持多路复用/路由功能/CDN中转/Shadowsocks混淆插件，多平台，无依赖。A Trojan proxy written in Go. An unidentifiable mechanism that helps you bypass GFW. https://p4gefau1t.github.io/trojan-go/
otoml - TOML parsing, manipulation, and pretty-printing library for OCaml (fully 1.0.0-compliant)
goproxy - An HTTP proxy library for Go
ocaml-tsort - Easy to use and user-friendly topological sort module for OCaml
pyppeteer - Headless chrome/chromium automation library (unofficial port of puppeteer)
soupault.app - The source code of the soupault.app website
colly - Elegant Scraper and Crawler Framework for Golang
selectolax - Python binding to Modest and Lexbor engines (fast HTML5 parser with CSS selectors).
lexbor - Lexbor is development of an open source HTML Renderer library. https://lexbor.com