The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →
Dedupe Alternatives
Similar projects and alternatives to dedupe
-
-
splink
Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
-
record-linkage-resources
Resources for tackling record linkage / deduplication / data matching problems
NOTE:
The number of mentions on this list indicates mentions on common posts plus user suggested alternatives.
Hence, a higher number means a better dedupe alternative or higher similarity.
dedupe reviews and mentions
Posts with mentions or reviews of dedupe.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2022-10-15.
- Using deep learning for Fuzzy Matching
-
String distance based network for fuzzy matching?
I think this problem is known as data deduplication, in particular, entity deduplication. I googled a bit and it seems approaches vary from manual deduplication to some sort of active learning (if I am not mistaken). I am also curios if pre-trained transformer-based cross encoders can provide any good results (they are trained on sentences I think, but may be worth a try). Another problem here is how to measure progress (compare different approaches)?
-
What's the toughest DE problem you faced in your work career?
I've had a good experience in the past with the dedupe package for these type activities. Unsure if it works for out-of-core type situations though, as my data set fit easily into memory.
-
Model detects duplicate records
Data deduplication is a super common problem, so it's useful experience to work on it. It's generally useful for companies, but I don't think it could be sold as a product unless is solving a very complicated, domain-specific de-duping problem. Otherwise, there are generic, open source de-duping tools such as: dedupe. It sounds like your model is similar to that.
- [D] Suggestions for large-scale company name standardization?
- Entity Resolution with Magniv
- How to do fuzzy matching in Redshift? A Python UDF, for example?
-
[OC] Media bias? US Sunday news shows book Republicans more than Democrats: Three of the five top Sunday news shows, altogether watched by almost 8 million people weekly, featured Republican partisans more often than Democrats in episodes aired this year through Oct. 31.
Tools used: Python to scrape guest lists, dedupeio to better identify guests, Google Sheets to store and analyze the data, and Datawrapper to make the charts.
-
Does there exist a python package that clears the dataset/columns in terms of exact and similar duplicates?
Try https://github.com/dedupeio/dedupe
-
A note from our sponsor - WorkOS
workos.com | 25 Apr 2024
Stats
Basic dedupe repo stats
9
3,973
7.1
about 1 month ago
dedupeio/dedupe is an open source project licensed under MIT License which is an OSI approved license.
The primary programming language of dedupe is Python.
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com