DataProfiler
RecordStream
| DataProfiler | RecordStream | |
|---|---|---|
| 61 | 3 | |
| 1,363 | 298 | |
| 1.0% | - | |
| 6.3 | 2.6 | |
| 4 days ago | almost 4 years ago | |
| Python | Perl | |
| Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
DataProfiler
-
LongRoPE: Extending LLM Context Window Beyond 2M Tokens
It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler
For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.
- Data Profiler – What's in your data?
-
Data Profiler 0.9.0 -- offering a massive improvement to memory usage during profiling of large datasets
Great call out -- would you be willing to write up an issue for that on the repo? Thank you! https://github.com/capitalone/DataProfiler/issues/new/choose
- FLiPN-FLaNK Stack Weekly for 20 March 2023
- Release 0.8.3 · capitalone/DataProfiler
RecordStream
-
Miller – tool for querying, shaping, reformatting data in CSV, TSV, and JSON
It's interesting watching these types of tools get re-invented periodically:
https://github.com/benbernard/RecordStream
It shows the unix model of many small, composable tools is very powerful, but also shows that POSIX is missing some essential pieces that everyone keeps trying to add/reinvent.
-
Miller CLI – Like Awk, sed, cut, join, and sort for CSV, TSV and JSON
I don't know about MillerCLI's portability, but RecordStream (https://github.com/benbernard/RecordStream) is my go to swiss army knife.
-
A Lisp REPL as my main shell (article)
That record/field parsing library would be a tool to handle a broad category of command-line programs. Once the library has broken the input stream into a collection of records and fields, another layer would then turn them into internal representations. The JSON-based RecordStream tools are illustrative here: there are some tools that parse based on a delimiter or a regular expression, some that parse documented generic non-JSON formats like XML, and some that parse application-specific files like tcpdump outputs. In a Lisp world, all of the dedicated stream manipulation tools are redundant, and you avoid parsing and printing at every step in the chain.
What are some alternatives?
ydata-profiling - 1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
ocaml-containers - A lightweight, modular standard library extension, string library, and interfaces to various libraries (unix, threads, etc.) BSD license.
pyWhat - 🐸 Identify anything. pyWhat easily lets you identify emails, IP addresses, and more. Feed it a .pcap file or some text and it'll tell you what it is! 🧙♀️
vnlog - Process labelled tabular ASCII data using normal UNIX tools
usaddress - :us: a python library for parsing unstructured United States address strings into address components
jq - Command-line JSON processor [Moved to: https://github.com/jqlang/jq]
XlsxWriter - A Python module for creating Excel XLSX files.
rq - Record Query - A tool for doing record analysis and transformation
superset - Apache Superset is a Data Visualization and Data Exploration Platform
miller - Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
vtuber-livechat-dataset - 📊 VTuber 1B: Billion-scale Live Chat and Moderation Event Dataset
jq - Command-line JSON processor