Our great sponsors
-
Nuitka
Nuitka is a Python compiler written in Python. It's fully compatible with Python 2.6, 2.7, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 3.10, and 3.11. You feed it your Python app, it does a lot of clever things, and spits out an executable or extension module.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
I used a modified version of my combine_folder_multiprocess script to count the total objects for each subreddit for each month. Then a separate script to sum them all together, sort it and write out the result.
I also compiled it with Nuitka and was able to process a ~35 million line file in 1 hour and 22 minutes, averaging about 6,800 lines per second.