requests VS Airflow

all languages also seem to have packages around providing cert bundles which get used directly (e.g., certifi [3]), which does cause some pain

[1] https://github.com/rustls/rustls-native-certs/issues/16#issu...

[2] https://github.com/psf/requests/issues/2899

[3] https://pypi.org/project/certifi/

Python’s urllib2 (now urllib.request) started out in the year 2000 [0].

.NET’s WebRequest was available in .NET Framework 1.1 in 2003 [1].

But since then, Microsoft noticed the issues with WebRequest and came up with HttpClient in 2012. It has some issues and footguns, like those related to HttpClient lifetime, but it’s a solid library. On the other hand, the requests library for Python started in 2011 [2], but the stdlib library hasn’t seen many improvements.

[0] https://github.com/python/cpython/blob/6d7e47b8ea1b8cf82927d...

[1] https://learn.microsoft.com/en-us/dotnet/api/system.net.webr...

[2] https://github.com/psf/requests/blob/main/HISTORY.md#001-201...

Python’s requests package, which uses urllib from the standard library, has a very distinctive TLS fingerprint, containing ciphers (amongst other things) that aren’t seen in a browser. This makes it very easy to spot. Both rnet, and other options such as curl-cffi, are able to send a TLS fingerprint similar to that of a browser. This reduces the chances of our request being blocked.

There's been a branch for it (https://github.com/psf/requests/tree/hatchling) for a little while apparently; I guess they won't merge it until absolutely necessary for the 2.33 release. But that is still just over a year after I offered (https://github.com/psf/requests/issues/6775).

... Ah, I got confused for a bit. When I first noticed the `pyproject.toml` deficiency, it was because Requests was affected by the major Setuptools 72 backwards incompatibility. Then this year they were hit again by the major Setuptools 78 backwards incompatibility (which the Setuptools team consciously ignored in testing because Requests already publishes their own wheel, so this only affected the build-from-source purists like distro maintainers). See also my writeup https://lwn.net/Articles/1020576/ .

This, I think, is the crux of the matter.

As an example, I always knew urllib3 as one of the foundational packages that Requests uses. And I was curious, what versions of urllib3 does Requests pull in?

Well, according to https://github.com/psf/requests/blob/main/setup.cfg, it's this:

    urllib3>=1.21.1,<3

I suggested exactly that (for the closely related Deprecation header) to Requests a few years ago, they feel it's the application's responsibility, discussion here: https://github.com/psf/requests/issues/5724

Code Reading: Study repositories like Flask, Requests, or Rich on GitHub. Notice how they handle imports, error cases, and API design. Reading production code reveals patterns you won't find in tutorials.

# Add latest version of requests package uv add requests # Specify version uv add 'requests==2.31.0' # Add git dependency uv add git+https://github.com/psf/requests # Migrate from requirements.txt uv add -r requirements.txt -c constraints.txt # If not using constraints.txt, use this uv add -r requirements.txt # Adding --dev makes it a development dependency # (added to dev in [dependency-groups] of pyproject.toml). # Example of adding moto needed only for testing as development dependency uv add moto --dev # Updates are also possible with uv add --upgrade uv add requests --upgrade

https://github.com/psf/requests/commit/79bb9ee1417afe2231972...

For senior engineers building custom job data visualization pipelines, the single biggest latency gain comes from pre-aggregating frequently accessed metrics instead of running joins at query time. In our benchmarks, querying raw job_postings tables with 1M rows took 210ms average, while pre-aggregated tables (updated hourly via PostgreSQL materialized views) reduced query time to 12ms. Use tools like Apache Airflow 2.7.3 to schedule materialized view refreshes during off-peak hours. For example, a materialized view for average salary by company can be defined as:

As of now, there is no new auth manager that supports LDAP natively with the new Airflow 3 UI. There is an open discussion in the Airflow repo about building a lightweight LDAP auth manager, but nothing shipped yet. If you are using LDAP, this is something to be aware of. It is cosmetic, not functional.

Airflow

Apache Airflow - Apache's Airflow project is a popular workflow system that supports DAG-based tasks and precise scheduling. It's an extensible Python project that supports several different providers and job executors, including Kubernetes.

Automation Tools: MLflow, Airflow, Kubernetes

Airflow GitHub Repository: Explore source code, example DAGs, or contribute to the project.

There is a lot of stuff for Python which follows the "express computation as a dag" approach, especially Apache Airflow

https://airflow.apache.org/