notes
dedupe
Our great sponsors
notes | dedupe | |
---|---|---|
17 | 9 | |
53 | 3,973 | |
- | 1.1% | |
1.5 | 7.1 | |
about 1 year ago | about 1 month ago | |
Vim Script | Python | |
GNU General Public License v3.0 or later | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
notes
-
Blog: Terminal file managers and my Vifm setup
I've documented some really cool things that vifm can do: https://github.com/sitaramc/notes/blob/master/vifm.mkd
-
Prevent Thunar from writing lines to config file, or alternatively make git ignore certain lines
So I wrote https://github.com/sitaramc/notes/blob/master/dac -- 30 lines of shell plus maybe another 30 or 40 of comments, and it does everything I want in a dotfile manager.
-
What tools/methods do you use to track/journal all changes to your (desktop) system?
All this is painlessly taken care of by https://github.com/sitaramc/notes/blob/master/dac (documentation: https://github.com/sitaramc/notes/blob/master/dac.mkd)
-
ncdu - ncurses disk usage - see which directories and files are hogging the most space
If you have fzf installed, grab https://github.com/sitaramc/notes/blob/master/try, and run try dust. Then start typing -t jpg (for example). Then backspace over the jpg and change it to png. Or use some other options.
-
what is the easiest way to backup your config files?
I hesitate to nominate any of them because (a) there are dozens or even hundreds of them and (b) I don't like any of them; I wrote my own because I needed a "hold" feature that no one had (i.e., when propagating changes to the repo, I want to hold back some parts of the change; https://github.com/sitaramc/notes/blob/master/dac.mkd explains better if you're interested
-
Do you use VIFM?
For those of you who are curious, https://github.com/sitaramc/notes/blob/master/vifm.mkd is my part "review" and part "tips and tricks" on vifm.
-
What tools / utilities have you written that you use regularly?
Most of my tools are in bash or perl, most of them less than 100 lines of code, (most of them are less than 200 even with comments). https://github.com/sitaramc/notes has all of them (terrible name for a repo full of tools I know; sorry!)
-
Seeking a terminal file manager
Absolutely vifm. My notes+tips/tricks on this at https://github.com/sitaramc/notes/blob/master/vifm.mkd
-
Should I use vim or neovim?
https://github.com/sitaramc/notes/blob/master/dac.mkd for documentation, https://github.com/sitaramc/notes/blob/master/dac.mkd for code, if you're interested.
-
difference between terminal file managers?
if you're a vim user, you can't go wrong with vifm. My take on vifm is here: https://github.com/sitaramc/notes/blob/master/vifm.mkd -- can't hurt to give it a read (it's a wee bit outdated but not much; probably only the last section needs to be updated)
dedupe
- Using deep learning for Fuzzy Matching
-
String distance based network for fuzzy matching?
I think this problem is known as data deduplication, in particular, entity deduplication. I googled a bit and it seems approaches vary from manual deduplication to some sort of active learning (if I am not mistaken). I am also curios if pre-trained transformer-based cross encoders can provide any good results (they are trained on sentences I think, but may be worth a try). Another problem here is how to measure progress (compare different approaches)?
-
What's the toughest DE problem you faced in your work career?
I've had a good experience in the past with the dedupe package for these type activities. Unsure if it works for out-of-core type situations though, as my data set fit easily into memory.
-
Model detects duplicate records
Data deduplication is a super common problem, so it's useful experience to work on it. It's generally useful for companies, but I don't think it could be sold as a product unless is solving a very complicated, domain-specific de-duping problem. Otherwise, there are generic, open source de-duping tools such as: dedupe. It sounds like your model is similar to that.
- [D] Suggestions for large-scale company name standardization?
- Entity Resolution with Magniv
- How to do fuzzy matching in Redshift? A Python UDF, for example?
-
[OC] Media bias? US Sunday news shows book Republicans more than Democrats: Three of the five top Sunday news shows, altogether watched by almost 8 million people weekly, featured Republican partisans more often than Democrats in episodes aired this year through Oct. 31.
Tools used: Python to scrape guest lists, dedupeio to better identify guests, Google Sheets to store and analyze the data, and Datawrapper to make the charts.
-
Does there exist a python package that clears the dataset/columns in terms of exact and similar duplicates?
Try https://github.com/dedupeio/dedupe
What are some alternatives?
clifm - The shell-like, command line terminal file manager: simple, fast, extensible, and lightweight as hell.
splink - Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
smenu - smenu started as a lightweight and flexible terminal menu generator, but quickly evolved into a powerful and versatile CLI selection tool for interactive or scripting use.
imgdupes - Identifying and removing near-duplicate images using perceptual hashing.
vifm.vim - Vim plugin that allows use of vifm as a file picker
orange - 🍊 :bar_chart: :bulb: Orange: Interactive data analysis
suda.vim - 🥪 An alternative sudo.vim for Vim and Neovim, limited support sudo in Windows
bees - Best-Effort Extent-Same, a btrfs dedupe agent
nbrowser - 🔗 🌐 : an easy way to open links in browsers, mimic the "Open URL with..." dialog on Android, `nbrowser` help you open links in a browser
pyDenStream - Implementation of the DenStream algorithm in Python.
Watson - :watch: A wonderful CLI to track your time!
hazelcast-python-client - Hazelcast Python Client