How to use undocumented web APIs

• curl-impersonate

  31 3,291 7.1 Python

  curl-impersonate: A special build of curl that can impersonate Chrome & Firefox

The first problem can be solved with curl-impersonate: https://github.com/lwthiker/curl-impersonate

"A special compilation of curl that makes it impersonate Chrome & Firefox", and it now can also impersonate Edge and Safari.

Previously discussed: https://news.ycombinator.com/item?id=30378562 _Show HN: Curl modified to impersonate Firefox and mimic its TLS handshake_ (21 days ago, 58 comments)

• gobuster

  14 8,974 4.9 Go

  Directory/File, DNS and VHost busting tool written in Go

gobuster is an effective way to enumerate subdomains and their directories quickly.

https://github.com/OJ/gobuster

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• instagram-private-api

  5 5,692 4.3 TypeScript

  NodeJS Instagram private API SDK. Written in TypeScript.

The most trivial check a website owner can do is checking the user-agent, which Python requests automatically sets to show its name, unless you configure your own. Trivial way to work around is to set your own user-agent to one that looks like a browser.

Specifically regarding Instagram, you can take a look at the implementation of https://github.com/dilame/instagram-private-api to understand more workarounds, as Instagram is getting better and better at working against the workarounds.

• puppeteer

  356 86,704 9.9 TypeScript

  Node.js API for Chrome

  

I found puppeteer very nice to script against if you need a real headless browser:

https://github.com/puppeteer/puppeteer

• Capybara

  20 9,960 7.7 Ruby

  Acceptance test framework for web applications

  

• Playwright

  377 61,381 9.9 TypeScript

  Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.

  

[Playwright](https://playwright.dev/) (Node / Python) is my current preferred - mainly because I seem to have less reliability issues with the browser starting/stopping cleanly (although it's never perfect with any of the tools I've tried).

• clairvoyance

  2 901 4.3 Python

  Obtain GraphQL API schema even if the introspection is disabled

You may not even need introspection--

https://github.com/nikitastupin/clairvoyance

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• GraphQLmap

  1 1,280 0.0 Python

  GraphQLmap is a scripting engine to interact with a graphql endpoint for pentesting purposes. - Do not use for illegal testing ;)

• browsercookie

  2 100 6.2 Python

If you still use the website via browser, I find https://github.com/richardpenman/browsercookie/ is great for working around the expiring cookie problem

Suggest a related project