requests-futures VS urllib3

- https://github.com/ross/requests-futures

For the simple one-off case, check out requests-futures or requests-threads

Lately I tried https://github.com/ross/requests-futures and it is really nice little wrapper for requests library that makes it run asynchronous via thread pool. Worth checking out if you are already using requests lib.

requests-futures can handle that

Thank you, I didn’t know about this before. Looks like either requests-threads or requests-futures can do the job. Not sure how I can add async and await to the script above, I’ll tinker around with it later!

I found examples of running parallel async http requests using grequests, but in its GitHub page it recommends using requests-threads or requests-futures instead. Which of them would be the most straightforward tool for optimizing a sequence of GET requests against an API. Case scenario: API endpoint provides paginated responses. With first response, I get the total of itens, which allow me to prepare all the remaining urls. The API allows for 25 simultaneous requests from a single user session (JWT token).

I would suggest using a session in requests if you're not already but also to look at asyncio/aiohttp , requests-futures or requests-html which can be used asynchronously and is written by the author of the requests library.

As opposed to what the article says, urllib3 now has experimental support for browser as of Jan 30th.

Source: https://github.com/urllib3/urllib3/releases/tag/2.2.0

Then, I tried to get a firm grip on urllib3 base code, contributing this and there until I was ready to kick things up with a proof of concept that would have put urllib3 far ahead. Without any breaking changes. I was delusional. This was a bit of a shock, but six months passed between my initial kick off and my formal give up, and here's why in a nutshell:

Oh wow, thanks for this story! Would love to hear more if you have time :) Good luck with testing it out.

Note that we found an issue w/ emitting an InsecureRequestWarning by default. The request is perfectly secure, it's just we aren't telling the ConnectionPool that information (see: https://github.com/urllib3/urllib3/issues/3331)

I've had a good experience doing a couple of bug fix bounties for urllib3 https://github.com/urllib3/urllib3/issues . I'd be interested in how the maintainers how found running the bug bounty and if it's given them more useful fixes or if it just adds more noise to deal with

ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with LibreSSL 2.8.3. See: https://github.com/urllib3/urllib3/issues/2168

> Could not import extension sphinx.builders.linkcheck (exception: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with OpenSSL 1.0.2n 7 Dec 2017. See: https://github.com/urllib3/urllib3/issues/2168)

This error is coming from Python, it's telling us Python is failing to import the urllib3 library, these lines here are important:

Requests allows you to send HTTP/1.1 requests extremely easily. There’s no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic, thanks to urllib3.

Judging projects based on stars is really immature. for example everyone knows requests https://github.com/psf/requests the python package that is used in every python project out there. it has 47k star too WOW. but the thing that less people know is urllib3. https://github.com/urllib3/urllib3. it has only 3k stars. It basically does the heavy lifting for requests!!

urllib3 – Python HTTP library with thread-safe connection pooling, file post support, user friendly, and more