Data Ingestion - Build Your Own "Map Reduce"?

1. mrjob

   1 1 2,617 0.0 Python

   Run MapReduce jobs on Hadoop or Amazon Web Services

   

   Mrjob

2. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

3. mmh3

   2 2 327 8.6 C

   Python extension for MurmurHash (MurmurHash3), a set of fast and robust hash functions.

   

   Some notes: We don't need Sha256 and not evey base64; nothing will happen if keys will not distribute very equally. we could take MMH3; googling "python murmurhash" gives 2 interesting results; and since both use the same cpp code, let's take the one with most stars Other options would be to simply do (% NUM_SHARDS) or even shift right (however must have shards count == power of 2).

4. murmurhash

   3 2 43 5.0 C++

   💥 Cython bindings for MurmurHash2 (by explosion)

   

   Some notes: We don't need Sha256 and not evey base64; nothing will happen if keys will not distribute very equally. we could take MMH3; googling "python murmurhash" gives 2 interesting results; and since both use the same cpp code, let's take the one with most stars Other options would be to simply do (% NUM_SHARDS) or even shift right (however must have shards count == power of 2).

5. py-spy

   4 26 13,161 7.8 Rust

   Sampling profiler for Python programs

   

   Q: Are we sure about it? A: we could use py-spy and see.

Suggest a related project