lol-html
PyO3
Our great sponsors
lol-html | PyO3 | |
---|---|---|
8 | 147 | |
1,390 | 10,997 | |
1.9% | 4.4% | |
5.7 | 9.8 | |
about 1 month ago | 4 days ago | |
Rust | Rust | |
BSD 3-clause "New" or "Revised" License | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
lol-html
-
Ask HN: A fast, Rust HTML parser that works?
So I'm doing some web scraping in Rust, and so I will need to parse HTML. [scraper](https://docs.rs/scraper/latest/scraper/) (which uses [html5ever](https://github.com/servo/html5ever)) is doing fine except that it's the bottleneck of my application.
So I need a faster parser. I've tried [tl](https://docs.rs/tl/latest/tl/) which would've been perfect except that it doesn't actually work on the HTML I have. When I try to `query_selector` the elements I need, it returns nothing.
[Kuchiki](https://docs.rs/kuchiki/latest/kuchiki/) is abandonded.
I couldn't figure out how to get [lol-html](https://github.com/cloudflare/lol-html) to work for me (it's designed for re-writing HTML, whatever that means). It doesn't seem to have an API to extract the inner text of an element.
[html5gum](https://github.com/untitaker/html5gum) seems to be just an HTML tokenizer, or otherwise just too low-level. I have not yet tried [quick-xml](https://github.com/tafia/quick-xml/) but judging from the README, it's pretty low-level too. I mean, if these are the only options left then I will try them. Otherwise, I would love to use a parser that's faster but as ergonomic as `scraper` or `tl`.
At this point, I would be happy with an Lxml bridge/port of some sort. I don't need to mutate HTML, just parse and read data from it.
-
How much Rust work is actually going on at Cloudflare?
I'm also in the Workers org but I have had a bit of interaction with Rust. There's some Rust in the Workers runtime using lol-html for HTMLRewriter as well as some tooling and there's the full blown workers-rs framework that I work on, but that's about it for the Rust I work on regularly.
- Is there a library for manipulating HTML?
- pup: Parsing HTML at the Command Line
-
Texting Robots: Taming robots.txt with Rust and 34 million tests
Thanks again and happy to answer any questions! My current unreleased Rust projects include a web crawler that uses Tokio + Tokio Console + Reqwest with this crate for robots.txt and a fast text extraction library using lol-html that I am planning to sprinkle with some minimal ML to get Readability.js style intelligent extraction (with training in Python). See Fathom for an example of the ML approach I'll likely take.
-
Like JQ, but for HTML
I’d like to see a tool using lol-html [0] and their CSS selector API as a streaming HTML editor.
[0] https://github.com/cloudflare/lol-html
- Things you can’t do in Rust (and what to do instead)
-
Problems with building a backend app in Rust in 2020
Cloudflare has open sourced lol-html, a "Low output latency streaming HTML parser/rewriter with CSS selector-based API". Is that what you are looking for?
PyO3
-
Encapsulation in Rust and Python
Integrating Rust into Python, Edward Wright, 2021-04-12 Examples for making rustpython run actual python code Calling Rust from Python using PyO3 Writing Python inside your Rust code — Part 1, 2020-04-17 RustPython, RustPython Rust for Python developers: Using Rust to optimize your Python code PyO3 (Rust bindings for Python) Musing About Pythonic Design Patterns In Rust, Teddy Rendahl, 2023-07-14
- Rust Bindings for the Python Interpreter
- Polars – A bird's eye view of Polars
-
In Rust for Python: A Match from Heaven
This story unfolds as a captivating journey where the agile Flounder, representing the Python programming language, navigates the vast seas of coding under the wise guidance of Sebastian, symbolizing Rust. Central to their adventure are three powerful tridents: cargo, PyO3, and maturin.
- Segunda linguagem
-
Calling Rust from Python
I would not recommend FFI + ctypes. Maintaining the bindings is tedious and error-prone. Also, Rust FFI/unsafe can be tricky even for experienced Rust devs.
Instead PyO3 [1] lets you "write a native Python module in Rust", and it works great. A much better choice IMO.
[1] https://github.com/PyO3/pyo3
-
Python 3.12
Same w/ Rust and Python, this is really neat because now each thread could have a GIL without doing exactly what you said. The pyO3 commit to allow subinterpreters was merged 21 days ago, so this might "just work" today: https://github.com/PyO3/pyo3/pull/3446
-
Removing Garbage Collection from the Rust Language (2013)
I expected someone to write a rust-based scripting language which tightly integrated with rust itself.
In reality, it seems like the python developers and toolchain are embracing rust enough to reduce the benefits to a new alternative.
https://github.com/PyO3/pyo3
-
Bytewax: Stream processing library built using Python and Rust
Hey HN! I am one of the people working on Bytewax. Bytewax came out of our experience working with ML infrastructure at GitHub. We wanted to use Python because we could move fast, the team was very fluent in it, and the rest of our tooling was Python-native already. We didn't want to introduce JVM-based solutions into our stack because of the lack of experience and the friction we had trying to get Python-centric tooling working with existing solutions like Flink.
In our research, we found Timely Dataflow (https://timelydataflow.github.io/timely-dataflow/, https://news.ycombinator.com/item?id=24837031) and the Naiad project (https://www.microsoft.com/en-us/research/project/naiad/) as well as PyO3 (https://github.com/PyO3/pyo3) and we thought we found a match made in heaven :). Bytewax leverages both of these projects and builds on them to provide a clean API (at least we think so) and table stakes features like connectors, state recovery, and cloud-native scaling. It has been really cool to learn about the dataflow computation model, Rust, and how to wrangle the GIL with Rust and Python :P.
Would love to get your feedback :).
`pip install bytewax` to get started. We have a page of guides (https://www.bytewax.io/guides) with ready-to-run examples.
-
Tell HN: Rust Is the Superglue
You can practice your Rust skills by writing performant and/or gluey extensions for higher-level language such as NodeJS (checkout napi-rs) and Python or complementing JS in the browser if you target Webassembly.
For instance, checkout Llama-node https://github.com/Atome-FE/llama-node for an involved Rust-based NodeJS extension. Python has PyO3, a Rust-Python extension toolset: https://github.com/PyO3/pyo3.
They can help you leverage your Rust for writing cool new stuff.
What are some alternatives?
actor-rust-scraper - Experimental scraper in Rust suited for running locally or on the Apify platform. Inspired by Apify SDK.
rust-cpython - Rust <-> Python bindings
tq - Perform a lookup by CSS selector on an HTML input
pybind11 - Seamless operability between C++11 and Python
yq - Command-line YAML, XML, TOML processor - jq wrapper for YAML/XML/TOML documents
RustPython - A Python Interpreter written in Rust
tools - all-in collection of productivity scripts, CLI tools, utility libraries, fuse filesystems, and also some stuff
milksnake - A setuptools/wheel/cffi extension to embed a binary data in wheels
hq - lightweight command line HTML processor using CSS and XPath selectors
bincode - A binary encoder / decoder implementation in Rust.
cargo-expand - Subcommand to show result of macro expansion
uniffi-rs - a multi-language bindings generator for rust