dedupe VS notes

I think this problem is known as data deduplication, in particular, entity deduplication. I googled a bit and it seems approaches vary from manual deduplication to some sort of active learning (if I am not mistaken). I am also curios if pre-trained transformer-based cross encoders can provide any good results (they are trained on sentences I think, but may be worth a try). Another problem here is how to measure progress (compare different approaches)?

I've had a good experience in the past with the dedupe package for these type activities. Unsure if it works for out-of-core type situations though, as my data set fit easily into memory.

Data deduplication is a super common problem, so it's useful experience to work on it. It's generally useful for companies, but I don't think it could be sold as a product unless is solving a very complicated, domain-specific de-duping problem. Otherwise, there are generic, open source de-duping tools such as: dedupe. It sounds like your model is similar to that.

Tools used: Python to scrape guest lists, dedupeio to better identify guests, Google Sheets to store and analyze the data, and Datawrapper to make the charts.

Try https://github.com/dedupeio/dedupe

I've documented some really cool things that vifm can do: https://github.com/sitaramc/notes/blob/master/vifm.mkd

So I wrote https://github.com/sitaramc/notes/blob/master/dac -- 30 lines of shell plus maybe another 30 or 40 of comments, and it does everything I want in a dotfile manager.

All this is painlessly taken care of by https://github.com/sitaramc/notes/blob/master/dac (documentation: https://github.com/sitaramc/notes/blob/master/dac.mkd)

If you have fzf installed, grab https://github.com/sitaramc/notes/blob/master/try, and run try dust. Then start typing -t jpg (for example). Then backspace over the jpg and change it to png. Or use some other options.

I hesitate to nominate any of them because (a) there are dozens or even hundreds of them and (b) I don't like any of them; I wrote my own because I needed a "hold" feature that no one had (i.e., when propagating changes to the repo, I want to hold back some parts of the change; https://github.com/sitaramc/notes/blob/master/dac.mkd explains better if you're interested

For those of you who are curious, https://github.com/sitaramc/notes/blob/master/vifm.mkd is my part "review" and part "tips and tricks" on vifm.

Most of my tools are in bash or perl, most of them less than 100 lines of code, (most of them are less than 200 even with comments). https://github.com/sitaramc/notes has all of them (terrible name for a repo full of tools I know; sorry!)

Absolutely vifm. My notes+tips/tricks on this at https://github.com/sitaramc/notes/blob/master/vifm.mkd

https://github.com/sitaramc/notes/blob/master/dac.mkd for documentation, https://github.com/sitaramc/notes/blob/master/dac.mkd for code, if you're interested.

if you're a vim user, you can't go wrong with vifm. My take on vifm is here: https://github.com/sitaramc/notes/blob/master/vifm.mkd -- can't hurt to give it a read (it's a wee bit outdated but not much; probably only the last section needs to be updated)