Scraping info from multiple links (from inside each link) that are listed on a webpage?

This page summarizes the projects mentioned and recommended in the original post on reddit.com/r/webscraping

Our great sponsors
  • SonarLint - Deliver Cleaner and Safer Code - Right in Your IDE of Choice!
  • Scout APM - Less time debugging, more time building
  • OPS - Build and Run Open Source Unikernels
  • GitHub repo grequests

    Requests + Gevent = <3

    I'd use an asynchronous programming package like grequests or Scrapy. Basically, store all the links you want to visit in a list then prepare them, send the requests, and then iterate over the content of the response. If the website needs JavaScript to then you can use requests-html with AsyncHTMLSession to render the JavaScript. ``` import grequests

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts