Best way to run parallel async http requests

This page summarizes the projects mentioned and recommended in the original post on reddit.com/r/learnpython

Our great sponsors
  • InfluxDB - Access the most powerful time series database as a service
  • Sonar - Write Clean Python Code. Always.
  • SaaSHub - Software Alternatives and Reviews
  • grequests

    Requests + Gevent = <3

    I found examples of running parallel async http requests using grequests, but in its GitHub page it recommends using requests-threads or requests-futures instead. Which of them would be the most straightforward tool for optimizing a sequence of GET requests against an API. Case scenario: API endpoint provides paginated responses. With first response, I get the total of itens, which allow me to prepare all the remaining urls. The API allows for 25 simultaneous requests from a single user session (JWT token).

  • requests-threads

    🎭 Twisted Deferred Thread backend for Requests.

    I found examples of running parallel async http requests using grequests, but in its GitHub page it recommends using requests-threads or requests-futures instead. Which of them would be the most straightforward tool for optimizing a sequence of GET requests against an API. Case scenario: API endpoint provides paginated responses. With first response, I get the total of itens, which allow me to prepare all the remaining urls. The API allows for 25 simultaneous requests from a single user session (JWT token).

  • InfluxDB

    Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

  • requests-futures

    Asynchronous Python HTTP Requests for Humans using Futures

    I found examples of running parallel async http requests using grequests, but in its GitHub page it recommends using requests-threads or requests-futures instead. Which of them would be the most straightforward tool for optimizing a sequence of GET requests against an API. Case scenario: API endpoint provides paginated responses. With first response, I get the total of itens, which allow me to prepare all the remaining urls. The API allows for 25 simultaneous requests from a single user session (JWT token).

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts