OpenRefine VS visidata

In April 2024, I was hired to polish the front-end of the Open Data Editor, an open source tool for non-technical data practitioners to explore and detect errors in tables. This app serves mostly as an easier-to-use alternative to Open Refine (with a lot less features, the tradeoff being a much flatter learning curve and more pleasing-to-the-eye interface).

Are you aware of this tool? https://openrefine.org

"OpenRefine is a powerful free, open source tool for working with messy data: cleaning it; transforming it from one format into another; and extending it with web services and external data." https://openrefine.org/

Yes, let's hope! The strategy has worked out sometimes - Google shut down 'Google Refine' 10 years ago, it got turned into 'Open Refine', last update 2 months ago. https://github.com/OpenRefine/OpenRefine

It's a hugely useful tool if you're working with messy Excel-scale data, i.e., most biologists or social scientists.

It seems to be pure JS with jQuery: https://github.com/OpenRefine/OpenRefine/blob/master/main/we...

EDIT: trim() does not remove unicode 0x200b (unicode character for zero width space). https://github.com/OpenRefine/OpenRefine/issues/5105 is worth a read.

There were six homework assignments. In order they were Regular Expressions, OpenRefine, Datalog, SQL, Provenance, and Python. None of these assignments took more than two to three hours to complete. They all were basic implementation and programming assignments with autograders.

I'd like to call out one of my favorite pieces of software from the past 10 years: VisiData [1] has completely changed the way I do ad-hoc data processing, and is now my go-to for pretty much all use cases that I previously used spreadsheets for, and about half of those I previously used databases for.

It's a TUI application, not strictly CLI, but scriptable, and I figure anyone building pipelines using tools like jq, q, awk, grep, etc. to process tabular data will find it extremely useful.

----

[1]: https://visidata.org

[4] "Is it possible to "flatten" structured data (like JSON?)": https://github.com/saulpw/visidata/discussions/1605

Visidata - https://visidata.org

You can try Visidata

VisiData is an awesome terminal spreadsheet tool. edbrowse for internet browsing.

Have you ever tried out visidata? It's not vim, but it's a terminal app with vim-like keybindings for visualizing tabular data (and it can convert from other types like json). Not quite a neovim buffer, but you could always open visidata in a new terminal buffer.

If you are a command-line user, try visidata[0]

[0] https://github.com/saulpw/visidata

The datasets will be available as SQLite, JSONL, and CSV. This will be great for sharpening your SQL/Python/VisiData skills.

Help Sarah find the family holiday tapestry before her father notices it's missing! Sharpen your SQL/Python/VisiData skills with Hanukkah of Data.