Our great sponsors
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
luigi
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
-
> And of course, Apple is not investing in these ports
What count as "investing in these ports"? Does submitting patches to CPython (and many other open source projects, including NumPy) for macOS 11 and Apple Silicon count? Here's a list of Apple-submitted PRs on python/cpython: https://github.com/python/cpython/pulls?q=is%3Apr+author%3Al... Also some co-authored patches excluded by the search. See https://bugs.python.org/issue41100 for more related PRs.
I'd add GRPC to the wishlist too: https://github.com/grpc/grpc/issues/4221
I used Luigi [1] to automate data processing at a previous job. It's a simple job queue with a UI. You request jobs from it, and then run them for minutes or hours, so it shouldn't normally be a bottleneck and it makes sense to use a language that's quick and easy to write.
It's written in Python and works fine to process thousands of jobs per day. Once you start having tens of thousands of jobs in the queue, it gets slow enough that it can back things up. This compounds the problem, eventually resulting in the whole thing crashing.
By switching the interpreter to PyPy, I was able to keep the data pipeline running at that scale without having to rewrite anything.
[1] https://github.com/spotify/luigi
> Apple changed some things that impact PyPy, like the register uses and ffi calling conventions.
I thought everyone who used 64-bit ARM used ARM's AAPCS64 (https://github.com/ARM-software/abi-aa/blob/master/aapcs64/a...), so the register usage and FFI calling convention should be the same as on Linux and Windows. What did Apple do that would affect the PyPy JIT?