Our great sponsors
-
-
insomnia
The open-source, cross-platform API client for GraphQL, REST, WebSockets, SSE and gRPC. With Cloud, Local and Git storage.
-
SurveyJS
Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.
Well, there is this website that I have been trying to scrape for a few days now. I had tried everything from scrapy splash on docker to almost giving up because I read somewhere that it was JavaScript rendered. Since the source code from the inspect part of the developer tools was different from the source code from the view-source:https//... on the same developer tools.How could this be possible? Then I kept searching on internet and found this concept; where you can mimic web-browsers requests from a server using an API program,and it worked magically. Some of the API programs are postman and insomnia. I prefer using insomnia for this particular case , feel free to use any other API program of your choice.
Well, there is this website that I have been trying to scrape for a few days now. I had tried everything from scrapy splash on docker to almost giving up because I read somewhere that it was JavaScript rendered. Since the source code from the inspect part of the developer tools was different from the source code from the view-source:https//... on the same developer tools.How could this be possible? Then I kept searching on internet and found this concept; where you can mimic web-browsers requests from a server using an API program,and it worked magically. Some of the API programs are postman and insomnia. I prefer using insomnia for this particular case , feel free to use any other API program of your choice.