DataProfiler VS jq

Great call out -- would you be willing to write up an issue for that on the repo? Thank you! https://github.com/capitalone/DataProfiler/issues/new/choose

My team built a similar tool in Python to load any delimited file, json, parquet and Avro with one command:

https://github.com/capitalone/DataProfiler

Effectively loads anything into a dataframe

The thing is, tensorflow has more ability to run cross platform.

I help maintain https://github.com/capitalone/DataProfiler

Our sensitive data detection library is exported to iOS, android, and Java; in addition to Python. We also run distributed and federated use cases with custom layers. All of which are improved in tensorflow.

That said, I’d use pytorch if I could. Simply put, it has a better user experience.

I really should write up how we did delimiter and quote detection in this library:

https://github.com/capitalone/DataProfiler

It turns out delimited files IMO are much harder to parse than say, JSON. Largely because they have so many different permutations. The article covers CSVs, but many files are tab or null separated. We’ve even seen @ separated with ‘ for quotes.

Given the above, it should still be possible to use the method described. I’m guessing you’d have to detect the separators and quote chars first, however. You’d have to also handle empty rows and corrupted rows (which happen often enough).

Having used both ray, dask, and writing custom threads, my personal view is that while there are advantages I wouldn’t want to use any of these unless absolutely necessary.

My personal approach for most of these tasks are to try to break down the problem to be as asynchronous as possible. Then you can create threads.

The nice thing about dask is really the way you can effectively use it as a pandas dataframe.

Having said that, we opted to write our own parallelization for this library:

https://github.com/capitalone/DataProfiler

As opposed to using the dask frame. Effectively, it’s a high overhead and easier to maintain the threading ourselves given the particular approaches taken.

That said, if I was working with large pandas dataframes, id likely use dask. For large datasets which couldn’t be stored in memory of use ray.io

My team has worked on a library for a similar purpose:

https://github.com/capitalone/DataProfiler

Load any document, profile and monitor the profiles for changes that would impact downstream applications.

Very common problem, you all are in a great space! Very interested and will check out!

We built a very similar application internally with our open source library: https://github.com/capitalone/dataprofiler

Effectively, you can monitor changes between profiles:

# Load a CSV file

Not exactly the same, but we wrote a library to easily load any delimited type of file and finds header (even if not first row). It also works to load JSON, Parquet, AVRO and loads it into a dataframe. Not CLI exactly, but pretty easy:

https://github.com/capitalone/dataprofiler

Anyway, pretty interesting Miller CLI

I think like you. But also, one does not necessarily know beforehand that they will want to make money.

Like a project could be born out of pure generosity, but after the happy initial phase the project might get too heavy on the maintenance requirements, causing the author to approach burnout, and possibly deciding that they want to make money to continue pulling the cart forward.

However, here's something I do think: if you create something as Open Source, it should be out of a mentality of goodwill and for the greater good, regardless of how it ends up being used. OSS licenses do mean this with their terms. If you later get tired or burned out, you should just retire and allow the community to keep taking care of it. Just like it happened with the Jq tool [1].

[1]: https://github.com/jqlang/jq/releases/tag/jq-1.7

Official Documentation: jqlang.github.io/jq

View on GitHub

To handle JSON files and JSON outputs in a script or format and highlight it, jq can be very handy. Many command line tools provide a json output, so you don't have to write a custom parser for a table a list in a terminal. Instead of that, you can use jq to get a specific value from the output or even modify the output. For more information, you can visit https://jqlang.github.io/jq/

In some projects I've wanted to use HTTPie to test APIs and jq to work with some JSON data. Nix has been really helpful in managing those dependencies that I can't easily get from npm.

> I'd love to see programs communicate through a typed JSON/proto format that shed enough details to make this more independent, and get useful shell command structuring/completion or full blown GUIs from simply introspecting the expected input and output types.

You should try PowerShell. It's basically Microsoft's .NET ecosystem molded into an interactive command line. I'm not entirely sure if PoweShell can make full use of the static types that build up its core, but its ability to exchange objects in the command line is almost unmatched.

On Linux you can use `jc` (https://github.com/kellyjonbrazil/jc) combined with `jq` (https://jqlang.github.io/jq/) to glue together command lines.

Yeah, but muscle memory bites me all the time and I put the backslash on the closing paren, too, because I'm so used to the regex usage of that syntax which needs them to match

I also want to draw the reader's attention to the magic of |@uri <https://github.com/jqlang/jq/blob/jq-1.7/docs/content/manual...> for a bunch of cases, but doubly so in TFA's case where they're plugging strings into a URI context. Simple string concat often works great for "hello world", but the world is not always just hello, so one quick use of the filter and jq's got your back

  echo "the world's scary" | jq -Rr '"\(.)"'

I think the original devs just got burnt out for a while https://github.com/jqlang/jq/issues/2305#issuecomment-157263...

jq 1.7 do preserve large integers but will truncate if any operation is done on them. Unfortunetly it currently truncates to a decimal64 which is a bit confusing, this will be fixed in next release where it follow the suggestion from the JSON spec and truncates to binary64 (double) https://github.com/jqlang/jq/pull/2949

The fact that jq takes almost a second to run on a Pi is crazy[0]. And the tool is written in C.

[0] https://github.com/jqlang/jq/issues/1411