Web scraping in 10 mins

This page summarizes the projects mentioned and recommended in the original post on dev.to

Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • LearnThisRepo.com - Learn 300+ open source libraries for free using AI.
  • WorkOS - The modern identity platform for B2B SaaS
  • Newman

    Newman is a command-line collection runner for Postman

    Well, there is this website that I have been trying to scrape for a few days now. I had tried everything from scrapy splash on docker to almost giving up because I read somewhere that it was JavaScript rendered. Since the source code from the inspect part of the developer tools was different from the source code from the view-source:https//... on the same developer tools.How could this be possible? Then I kept searching on internet and found this concept; where you can mimic web-browsers requests from a server using an API program,and it worked magically. Some of the API programs are postman and insomnia. I prefer using insomnia for this particular case , feel free to use any other API program of your choice.

  • insomnia

    The open-source, cross-platform API client for GraphQL, REST, WebSockets, SSE and gRPC. With Cloud, Local and Git storage.

    Well, there is this website that I have been trying to scrape for a few days now. I had tried everything from scrapy splash on docker to almost giving up because I read somewhere that it was JavaScript rendered. Since the source code from the inspect part of the developer tools was different from the source code from the view-source:https//... on the same developer tools.How could this be possible? Then I kept searching on internet and found this concept; where you can mimic web-browsers requests from a server using an API program,and it worked magically. Some of the API programs are postman and insomnia. I prefer using insomnia for this particular case , feel free to use any other API program of your choice.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts