PySnooper VS requests-html

Definitely not for production, but for debugging (esp. in cases where interactive debugging doesn't work) I've found PySnooper very useful

Looks interesting I will definitely try this.

For those that find this interesting, you might also like pysnooper - I use it all the time.

https://github.com/cool-RR/PySnooper

https://python.plainenglish.io/pysnooper-stop-debugging-pyth...

Found PySnooper the other day.

pysnooper! https://github.com/cool-RR/PySnooper

Shameless plug: PySnooper is a debugging tool for Python that lets you debug in a way that's as easy as adding print statements, but gives you a lot more information automatically.

https://github.com/cool-RR/PySnooper/

HN thread: https://news.ycombinator.com/item?id=19717786

snoop or pysnooper

If you want to install a library, pysnooper is cool. It's like an automatic print on every line of your function, with values.

The Python package PySnooper is pretty good for "fancy" print debug statements: https://github.com/cool-RR/pysnooper

I've caught quite a few bugs using this show-me-all-locals() approach...

requests-html

Ya, thanks for advice. When using requests_html library, I am trying to lower down the speed using response.html.render(timeout=1000), but it raise Runtime error instead on Google Colab: https://github.com/psf/requests-html/issues/517.

For dynamic js, prefer requests-html with xpath selection.

Example: requests-html which has a rather exhaustive README.md, but their dedicated page is not that helpful, if I remember correctly, and currently the domain is suspended.

When it comes to web scraping, the usual people recommend is beautifulsoup, lxml, or selenium. But I highly recommend people check out requests-html also. Its a library that is a happy medium between ease of use as in beautifulsoup and also good enough to be used for dynamic, javascript data where it would be overkill to use a browser emulator like selenium.

Quite possible. If you look at requests-html source code, it is simply one single python file that acts as a wrapper around a bunch of other packages, like requests, chromium, parse, lxml, etc., plus a couple convenience functions. So it could easily be some sort of bad dependency resolution.

What I do is try to see if I can use requests_html first before trying selenium. requests_html is usually enough if I dont need to interact with browser widgets or if the authentication isnt too difficult to reverse engineer.