hazelcast-python-client VS dedupe

More code samples here: https://github.com/hazelcast/hazelcast-python-client/tree/master/examples

The Map interface is the same when you activate the near cache. You can just use the usual Map methods, like set and put. See tests for near cache at: https://github.com/hazelcast/hazelcast-python-client/blob/master/tests/integration/backward_compatible/proxy/map_nearcache_test.py

See a sample usage with Map below. For all code sample, click here :)

I think this problem is known as data deduplication, in particular, entity deduplication. I googled a bit and it seems approaches vary from manual deduplication to some sort of active learning (if I am not mistaken). I am also curios if pre-trained transformer-based cross encoders can provide any good results (they are trained on sentences I think, but may be worth a try). Another problem here is how to measure progress (compare different approaches)?

I've had a good experience in the past with the dedupe package for these type activities. Unsure if it works for out-of-core type situations though, as my data set fit easily into memory.

Data deduplication is a super common problem, so it's useful experience to work on it. It's generally useful for companies, but I don't think it could be sold as a product unless is solving a very complicated, domain-specific de-duping problem. Otherwise, there are generic, open source de-duping tools such as: dedupe. It sounds like your model is similar to that.

Tools used: Python to scrape guest lists, dedupeio to better identify guests, Google Sheets to store and analyze the data, and Datawrapper to make the charts.

Try https://github.com/dedupeio/dedupe