OpenRefine VS CSVLint

"OpenRefine is a powerful free, open source tool for working with messy data: cleaning it; transforming it from one format into another; and extending it with web services and external data." https://openrefine.org/

Yes, let's hope! The strategy has worked out sometimes - Google shut down 'Google Refine' 10 years ago, it got turned into 'Open Refine', last update 2 months ago. https://github.com/OpenRefine/OpenRefine

It's a hugely useful tool if you're working with messy Excel-scale data, i.e., most biologists or social scientists.

It seems to be pure JS with jQuery: https://github.com/OpenRefine/OpenRefine/blob/master/main/we...

EDIT: trim() does not remove unicode 0x200b (unicode character for zero width space). https://github.com/OpenRefine/OpenRefine/issues/5105 is worth a read.

There were six homework assignments. In order they were Regular Expressions, OpenRefine, Datalog, SQL, Provenance, and Python. None of these assignments took more than two to three hours to complete. They all were basic implementation and programming assignments with autograders.

Open Refine will get you about 70% there. It's FOSS

I’ve heard good things about https://openrefine.org/

It's not suited to SQL, use Open Refine or python fuzzywuzzy.

Also, a little self-promotion here, I've created the CSV Lint plug-in for Notepad++ to work with CSV text data files. It can reformat, validate and sort csv files, as well as convert csv to different formats including SQL. Meaning it can take a csv data file and generate INSERT INTO statements, including CREATE TABLE with the corresponding column datatypes and everything.

I've created the CSV Lint plug-in for Notepad++ which can do all kinds of validation and transformations on a CSV file, processing it just as a text-file. Although it's only on Windows.

fyi Notepad++ has a CSV Lint plug-in which can convert a csv file into an SQL INSERT VALUES script, including a CREATE TABLE statement with the appropriate column datatypes (based on the content of the csv data)

Maybe you could use Notepad++ with the CSV Lint plug-in to convert a csv file to an SQL INSERT VALUES script, including a CREATE TABLE statement.

You could try opening the file as a plain text file in notepad, or maybe using Notepad++ and the CSV Lint plug-in

I just want to mention I've created a CSV Lint plug-in for Notepad++, maybe not exactly what you're looking for but it can generate initial Python scripts based on csv files, so might be useful.

The CSV Lint plug-in for Notepad++ was updated recently, it's available in the latest release of Notepad++ (v8.5.3). It is a useful plug-in for anyone working with csv datasets. I have created the plugin and had posted about it before, and this latest update has some more improvements and bugfixes.

If you just want to create tables you could experiment a bit with Notepad++ and the CSV Lint plug-in (Disclaimer: I'm the author of this plugin).

There's also the CSV Lint plug-in for Notepad++ which can detect datatypes, and then you can do CSV Lint > Generate metadata > Python script. Although idk it might not work correctly for all datetime datatypes.

idk if it counts as an ETL tool, but with the CSV Lint plug-in for Notepad++ you can quickly check a csv file for errors, validate a dataset or get a column summary report.