Splink Alternatives

splink reviews and mentions

• Finding near-duplicates with Jaccard similarity and MinHash
  4 projects | news.ycombinator.com | 4 Jul 2024

  For deduplicating people, the Fellegi Sunter model is also a powerful approach. Splink[0] is a Python library that implements this for big datasets. Probably you could combine parts of both approaches as well. Full disclosure: I am the lead author

  [0]https://github.com/moj-analytical-services/splink

• Splink: Fast, accurate, scalable probabilistic data linkage
  1 project | news.ycombinator.com | 13 Mar 2024
• Ask HN: What projects are you working on?
  1 project | news.ycombinator.com | 1 Jun 2023

  https://github.com/moj-analytical-services/splink

• Record linkage/Entity linkage
  2 projects | /r/datascience | 5 May 2023

  Record linkage has been a big part of a project I've been working on for 6 months now. I personally think a great and free solution be using the splink package in Python which can handle 10+m rows which implements the Fellegi-Sunter model (equivalent to a naive-Bayes model) is the classical model in record linkage. It can be trained in an unsupervised manner using some initial parameter estimation (these are quite intuitive) and then expectation maximisation. The features in the model will be different pairwise string comparisons on your field of interest. These can include exact equality; edit distance comparisons like Levensthein distance and Jaro-Winkler; and phonetic comparisons like soundex and double metaphone. The splink pacakge will handle training the model and then all the graph theory at the end to connect all your links into clusters. All the details you'll need are in the links. https://www.robinlinacre.com/probabilistic\_linkage/ https://moj-analytical-services.github.io/splink/

• What is the best approach to removing duplicate person records if the only identifier is person firstname middle name and last name? These names are entered in varying ways to the DB, thus they are free-fromatted.
  2 projects | /r/SQL | 25 Mar 2023

  https://moj-analytical-services.github.io/splink/ is a FOSS python package (but it runs against your db using SQL).

• DuckDB – in-process SQL OLAP database management system
  4 projects | news.ycombinator.com | 10 Feb 2023

  If you're curious, I've written a FOSS record linkage library that executes everything as SQL. It supports multiple SQL backends including DuckDB and Spark for scale, and runs faster than most competitors because it's able to leverage the speed of these backends: https://github.com/moj-analytical-services/splink

• Ask HN: What have you created that deserves a second chance on HN?
  44 projects | news.ycombinator.com | 26 Jan 2023

  Splink - a python library for probabilistic record linkage (fuzzy matching/entity resolution).

  Splink is dramatically faster and works on much larger datasets than other open source libraries. I'm particularly proud of the fact we support multiple execution backends (at the moment, DuckDb Spark Athena and Sqlite, but additional adaptors are relatively straightforward to write).

  We've had >4 million pypi downloads and it's used in government, academia and the private sector, often replacing extremely expensive proprietary solutions.

  https://github.com/moj-analytical-services/splink

  More info in blog posts here:

• Conformed Dimensions problem that keeps recurring on every project
  1 project | /r/dataengineering | 17 Jan 2023

  Splink is a SQL tool that can do this https://github.com/moj-analytical-services/splink

• How do you join two sources with attributes that aren't identical?
  1 project | /r/dataengineering | 12 Nov 2022

  Probabilistic record matching model such as a Fellegi-Sunter. Check out the splink package in Python.

• Splink 3: Fast, accurate and scalable record linkage (entity resolution) in Python
  1 project | /r/dataengineering | 29 Sep 2022

  Main docs here: https://moj-analytical-services.github.io/splink

• A note from our sponsor - SaaSHub
  www.saashub.com | 9 Dec 2024

  SaaSHub helps you find the best software and product alternatives Learn more →