Data Ingestion - Build Your Own "Map Reduce"?

• mrjob

  1 2,609 0.0 Python

  Run MapReduce jobs on Hadoop or Amazon Web Services

  

Mrjob

• mmh3

  2 303 7.5 C

  Python extension for MurmurHash (MurmurHash3), a set of fast and robust hash functions.

Some notes: We don't need Sha256 and not evey base64; nothing will happen if keys will not distribute very equally. we could take MMH3; googling "python murmurhash" gives 2 interesting results; and since both use the same cpp code, let's take the one with most stars Other options would be to simply do (% NUM_SHARDS) or even shift right (however must have shards count == power of 2).

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• murmurhash

  2 41 5.0 C++

  💥 Cython bindings for MurmurHash2 (by explosion)

  

Some notes: We don't need Sha256 and not evey base64; nothing will happen if keys will not distribute very equally. we could take MMH3; googling "python murmurhash" gives 2 interesting results; and since both use the same cpp code, let's take the one with most stars Other options would be to simply do (% NUM_SHARDS) or even shift right (however must have shards count == power of 2).

• py-spy

  25 11,850 6.4 Rust

  Sampling profiler for Python programs

Q: Are we sure about it? A: we could use py-spy and see.

Suggest a related project