requests-html VS feedparser

requests-html

Ya, thanks for advice. When using requests_html library, I am trying to lower down the speed using response.html.render(timeout=1000), but it raise Runtime error instead on Google Colab: https://github.com/psf/requests-html/issues/517.

For dynamic js, prefer requests-html with xpath selection.

Example: requests-html which has a rather exhaustive README.md, but their dedicated page is not that helpful, if I remember correctly, and currently the domain is suspended.

When it comes to web scraping, the usual people recommend is beautifulsoup, lxml, or selenium. But I highly recommend people check out requests-html also. Its a library that is a happy medium between ease of use as in beautifulsoup and also good enough to be used for dynamic, javascript data where it would be overkill to use a browser emulator like selenium.

Quite possible. If you look at requests-html source code, it is simply one single python file that acts as a wrapper around a bunch of other packages, like requests, chromium, parse, lxml, etc., plus a couple convenience functions. So it could easily be some sort of bad dependency resolution.

What I do is try to see if I can use requests_html first before trying selenium. requests_html is usually enough if I dont need to interact with browser widgets or if the authentication isnt too difficult to reverse engineer.

i am using the feedparser library in python https://github.com/kurtmckee/feedparser/ which basically takes an RSS url and standardizes it to a reasonable extent. But I have noticed that different websites still get parsed slightly differently. For example look at how https://beincrypto.com/feed/ has a long description (containing actual HTML) inside but this website https://www.coindesk.com/arc/outboundfeeds/rss/ completely cuts the description out. I have about 50 such websites and they all have slight variations. So you are saying that in addition to storing parsed data (title, summary, content, author, pubdate, link, guid) that I currently store, I should also add an xml column and store the raw from each url till I get a good hang of how each site differs?

There is JSON Feed¹ already. One of the spec writers is behind micro.blog, which is the first place I saw it(and also one of the few places I've seen it). I don't think it is a bad idea, and it doesn't take all that long to implement it.

I have long hoped it would pick up with the JSON-ify everything crowd, just so I'd never see a non-Atom feed again. We perhaps wouldn't need sooo much of the magic that is wrapped up in packages like feedparser² to deal with all the brokeness of RSS in the wild then.

¹ https://www.jsonfeed.org/

² https://github.com/kurtmckee/feedparser

What sites did you try? Looked into RSS yet? Many sites have RSS feeds you can use with something like https://github.com/kurtmckee/feedparser nytimes.com feeds: https://www.nytimes.com/rss

feedparser

You can explore FeedParser too

feedparser is a python package receiving and parsing RSS/Atom newsfeeds. The maintainer is active but really need much more support.

Lo más sencillo que conozco para monitorizar canales de YouTube son los feeds RSS que tiene cada canal. El formato es https://www.youtube.com/feeds/videos.xml?channel_id=[CHANNEL_ID]. Si no conoces RSS, echa un vistazo en la wiki. Para leer RSSs en Python tienes feedparser (y seguramente muchas más).