Best way to run parallel async http requests

This page summarizes the projects mentioned and recommended in the original post on reddit.com/r/learnpython

Our great sponsors
  • OPS - Build and Run Open Source Unikernels
  • SonarQube - Static code analysis for 29 languages.
  • Scout APM - Less time debugging, more time building
  • grequests

    Requests + Gevent = <3

    I found examples of running parallel async http requests using grequests, but in its GitHub page it recommends using requests-threads or requests-futures instead. Which of them would be the most straightforward tool for optimizing a sequence of GET requests against an API. Case scenario: API endpoint provides paginated responses. With first response, I get the total of itens, which allow me to prepare all the remaining urls. The API allows for 25 simultaneous requests from a single user session (JWT token).

  • requests-threads

    🎭 Twisted Deferred Thread backend for Requests.

    I found examples of running parallel async http requests using grequests, but in its GitHub page it recommends using requests-threads or requests-futures instead. Which of them would be the most straightforward tool for optimizing a sequence of GET requests against an API. Case scenario: API endpoint provides paginated responses. With first response, I get the total of itens, which allow me to prepare all the remaining urls. The API allows for 25 simultaneous requests from a single user session (JWT token).

  • OPS

    OPS - Build and Run Open Source Unikernels. Quickly and easily build and deploy open source unikernels in tens of seconds. Deploy in any language to any cloud.

  • requests-futures

    Asynchronous Python HTTP Requests for Humans using Futures

    I found examples of running parallel async http requests using grequests, but in its GitHub page it recommends using requests-threads or requests-futures instead. Which of them would be the most straightforward tool for optimizing a sequence of GET requests against an API. Case scenario: API endpoint provides paginated responses. With first response, I get the total of itens, which allow me to prepare all the remaining urls. The API allows for 25 simultaneous requests from a single user session (JWT token).

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts