requests-html VS httpx

requests-html

Ya, thanks for advice. When using requests_html library, I am trying to lower down the speed using response.html.render(timeout=1000), but it raise Runtime error instead on Google Colab: https://github.com/psf/requests-html/issues/517.

For dynamic js, prefer requests-html with xpath selection.

Example: requests-html which has a rather exhaustive README.md, but their dedicated page is not that helpful, if I remember correctly, and currently the domain is suspended.

When it comes to web scraping, the usual people recommend is beautifulsoup, lxml, or selenium. But I highly recommend people check out requests-html also. Its a library that is a happy medium between ease of use as in beautifulsoup and also good enough to be used for dynamic, javascript data where it would be overkill to use a browser emulator like selenium.

Quite possible. If you look at requests-html source code, it is simply one single python file that acts as a wrapper around a bunch of other packages, like requests, chromium, parse, lxml, etc., plus a couple convenience functions. So it could easily be some sort of bad dependency resolution.

What I do is try to see if I can use requests_html first before trying selenium. requests_html is usually enough if I dont need to interact with browser widgets or if the authentication isnt too difficult to reverse engineer.

For reference, it's a butterfly, not a moth.

Source: https://github.com/encode/httpx/issues/834

Very cool, first I'm hearing of httpx https://www.python-httpx.org/

I think most people would start with trying out requests or something for this kind of work, I'm guessing that didn't work out? You've got a star from me.

To access 10 different commands at the same time, that is tricky but definitely doable.

First thing that comes to mind, you can use aliases.

To keep it simple, lets use 3 examples instead of 10: harlequin (this project), pgcli (https://www.pgcli.com/) and httpx (https://www.python-httpx.org/)

Setup a main home for all your venvs:

    cd ~

There are already some implementations for Python HTTP clients. One of them is aiometer. But it's not suitable for my use case. Since httpx already has the internal pool, it would be better to reuse the design.

Besides, flama also provides support for SQL databases via SQLAlchemy, an SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL. Finally, flama also provides support for HTTP clients to perform requests via httpx, a next generation HTTP client for Python.

We can use the async HTTP client provided by httpx, a fully featured HTTP client for Python with an API broadly compatible with requests, so it can be used in pretty much the same way in most cases.

httpx in place of requests library

The main value of Requests is that it provided an abstract interface on top of HTTP, which was designed well-enough to become a standard. But today it has fallen way behind in its field, and there are much better alternatives such as HTTPX [0].

[0] https://www.python-httpx.org/

HTTPX is a popular Python library that provides an asynchronous HTTP client, and it can be beneficial for enabling async support in Django. While Django itself does not require HTTPX for async support, using HTTPX in combination with Django's async views can bring several advantages:

The underlying library for both sync and async is httpx (https://www.python-httpx.org/) which may be limited from the HTTP Client perspective but it may be possible to add rate limiting at a Session level.