requests-html VS zpy

requests-html

Ya, thanks for advice. When using requests_html library, I am trying to lower down the speed using response.html.render(timeout=1000), but it raise Runtime error instead on Google Colab: https://github.com/psf/requests-html/issues/517.

For dynamic js, prefer requests-html with xpath selection.

Example: requests-html which has a rather exhaustive README.md, but their dedicated page is not that helpful, if I remember correctly, and currently the domain is suspended.

When it comes to web scraping, the usual people recommend is beautifulsoup, lxml, or selenium. But I highly recommend people check out requests-html also. Its a library that is a happy medium between ease of use as in beautifulsoup and also good enough to be used for dynamic, javascript data where it would be overkill to use a browser emulator like selenium.

Quite possible. If you look at requests-html source code, it is simply one single python file that acts as a wrapper around a bunch of other packages, like requests, chromium, parse, lxml, etc., plus a couple convenience functions. So it could easily be some sort of bad dependency resolution.

What I do is try to see if I can use requests_html first before trying selenium. requests_html is usually enough if I dont need to interact with browser widgets or if the authentication isnt too difficult to reverse engineer.

If your interactive shell is zsh, you could give my project zpy a try, particularly the function pipz that it provides, which is a lightweight pipx clone with great completions and good speed.

I can immediately see some things rye is doing differently, like keeping the venvs themselves free of pip and pip-tools. I wonder in your explorations if you've tried rtx for managing python installations, or my own zpy wrapper of pip-tools+venv (which can also replace pipx).

My patterns in this regard aren't exactly mainstream, as I use flit+pip-tools+zpy (the latter being my own Zsh interface for Python dependency and environment operations), but FWIW here's how I go about nested requirements.

For managing venvs and dependencies and apps, I use my own frontend to pip-tools + venv, zpy. And for running tasks which require an activated venv, I use nox.

If anyone's interested in a pipx clone with excellent tab completion, I would appreciate any feedback on pipz, a function of my zsh plugin for python environment and dependency management: zpy

https://github.com/andydecleyre/zpy

For concise and practical interactive usage of those tools, with excellent tab completion, I made the Zsh frontend zpy.

I also use my own Zsh wrapper functions with it, so for example: https://i.imgur.com/YX8bWy8.png

I'm a big fan of (and small contributor to) pip-tools, but both poetry and pipenv offer management of more stuff, which understandably appeals to folks seeking a simple comprehensible workflow.

Pip-tools is also a bit lower level, offering flexibility and compatibility which I relish, but also requiring more attention from the user to set things up as they wish.

If you or anyone else enjoying pip-tools is a Zsh user and interested in trying out my higher level functions to ease interactive use of pip-tools, venvs, and also isolated app installs (like pipx), I would love some feedback on zpy: https://github.com/AndydeCleyre/zpy

I'm very happy to answer any questions about it right here or as GitHub issues.

This is FAR from some big mainstream thing, but I use (and am happy to answer any questions about) my own Zsh frontend to venv+pip-tools+pip, zpy.

FWIW, with my project zpy, extras groups take the form of EXTRANAME-requirements.in, and the function pypc (or as a subcommand zpy pypc) will read those and inject them into your pyproject.toml in the generic, PEP-supported sections.