Our great sponsors
-
-
-
SurveyJS
Open-Source JSON Form Builder to Create Dynamic Forms Right in Your App. With SurveyJS form UI libraries, you can build and style forms in a fully-integrated drag & drop form builder, render them in your JS app, and store form submission data in any backend, inc. PHP, ASP.NET Core, and Node.js.
Hey everyone,
we built a special-purpose web scraping client for Node.js. When scraping with pure HTTP clients, you want to blend in with the regular traffic as much as you can. This means your request signature needs to look like a browser's.
With got-scraping, we developed a special purpose header generator(https://github.com/apify/header-generator) that uses a bayesian network and real browser headers to make your headers undistinguishable.
We also override Node.js ciphers with the browser ones and simplify the use of proxies. HTTP protocol versions are auto-detected for both the target website and the proxy, so you can have a perfect HTTP2 connection even through a HTTP(S) proxy.
It's always a work in progress, so we would be grateful for any comments or tips how to make the requests even more stealthy!
Thanks!