splink VS zingg

https://github.com/moj-analytical-services/splink

Record linkage has been a big part of a project I've been working on for 6 months now. I personally think a great and free solution be using the splink package in Python which can handle 10+m rows which implements the Fellegi-Sunter model (equivalent to a naive-Bayes model) is the classical model in record linkage. It can be trained in an unsupervised manner using some initial parameter estimation (these are quite intuitive) and then expectation maximisation. The features in the model will be different pairwise string comparisons on your field of interest. These can include exact equality; edit distance comparisons like Levensthein distance and Jaro-Winkler; and phonetic comparisons like soundex and double metaphone. The splink pacakge will handle training the model and then all the graph theory at the end to connect all your links into clusters. All the details you'll need are in the links. https://www.robinlinacre.com/probabilistic\_linkage/ https://moj-analytical-services.github.io/splink/

https://moj-analytical-services.github.io/splink/ is a FOSS python package (but it runs against your db using SQL).

If you're curious, I've written a FOSS record linkage library that executes everything as SQL. It supports multiple SQL backends including DuckDB and Spark for scale, and runs faster than most competitors because it's able to leverage the speed of these backends: https://github.com/moj-analytical-services/splink

Splink - a python library for probabilistic record linkage (fuzzy matching/entity resolution).

Splink is dramatically faster and works on much larger datasets than other open source libraries. I'm particularly proud of the fact we support multiple execution backends (at the moment, DuckDb Spark Athena and Sqlite, but additional adaptors are relatively straightforward to write).

We've had >4 million pypi downloads and it's used in government, academia and the private sector, often replacing extremely expensive proprietary solutions.

https://github.com/moj-analytical-services/splink

More info in blog posts here:

Splink is a SQL tool that can do this https://github.com/moj-analytical-services/splink

Probabilistic record matching model such as a Fellegi-Sunter. Check out the splink package in Python.

Main docs here: https://moj-analytical-services.github.io/splink

It'd be great to see Splink add value in this area! Do give us a shout if you have any questions. The best place to post is on the Github discussions: https://github.com/moj-analytical-services/splink/discussions

As part of my data consulting, I struggled with identity resolution and started working on scalable no code identity resolution - https://github.com/zinggAI/zingg/ . It has pushed my limits as a software engineer and product builder, and I had to do a lot of learning to build it. Its cool to see people use Zingg in their workflows and save months of working on custom solutions. Big highlight has been North Carolina Open Campaign Data https://crossroads-cx.medium.com/building-open-access-to-nc-...

Check https://github.com/zinggAI/zingg/. We recently added Python to our stack and are looking for help with building dbt-zingg python models, databricks-zingg python notebooks, python api, building a python based front end etc.

If you are open to using a framework for this, check Zingg at https://github.com/zinggAI/zingg. It connects to Redshift, snowflake and other warehouses and can handle multiple columns

Thanks for your support. Yes we do ship with some examples and their models which can be run out of the box. We have 3 customer demographic datasets and an ecommerce items matching across Google and Amazon. You can check them here https://github.com/zinggAI/zingg/tree/main/examples

I have an open source project hosted at https://github.com/zinggAI/zingg/.

Check Zingg out at https://github.com/zinggAI/zingg and let me know if you would like to contribute

This is interesting, would love to get your feedback on Zingg(https://github.com/zinggAI/zingg) if you are upto it. Thanks!

I have released an open source entity resolution tool Zingg(https://github.com/zinggAI/zingg). Zingg uses Spark and ML to build single source of truth directly in the warehouse or the datalake. Would love to hear from the Reddit folks here what they think about it - do you find it useful? what can I do to make it better? any advice on the problem or the solution?