vnlog
DataProfiler
Our great sponsors
vnlog | DataProfiler | |
---|---|---|
24 | 61 | |
158 | 1,357 | |
- | 2.1% | |
6.7 | 6.3 | |
4 months ago | 3 days ago | |
Perl | Python | |
- | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
vnlog
- Vnlog: Process labelled tabular ASCII data using normal Unix tools
- Process tabular data with Unix tools
-
Qsv: Efficient CSV CLI Toolkit
For simple analyses (i.e. what most people do most of the time) doing this on the commandline gets you there faster. I use vnlog (https://github.com/dkogan/vnlog/). By the time you fired up your editor to write your Python code, I already have analyses and plots ready.
-
Joining CSV Data Without SQL: An IP Geolocation Use Case
Alternative very appropriate for some uses cases: `vnl-join` from the vnlog toolkit (https://github.com/dkogan/vnlog). Uses the `join` tool from coreutils (works well, has been around forever), and `vnlog` for nice column labelling
-
Miller: Like Awk, sed, cut, join, and sort for CSV, TSV, and tabular JSON
There's also https://github.com/dkogan/vnlog/ which is a wrapper around the existing coreutils, so all the options work, and there's nothing to learn
- vnlog: making awk and sort and join (and friends) smarter
-
Awk equivalents to SQL query data manipulation
And to improve the ergonomics, the vnlog wrappers are available to operate on field names, while retaining the internals of the core tools:
https://github.com/dkogan/vnlog/
- Vnlog: Making Awk, grep, sort and join smarter
-
Learn to Process Text in Linux Using Grep, Sed, and Awk
I sorta, kinda agree. Tools written in AWK (and friends) are indeed somewhat unmaintainable, but they're really close to being just right for a LOT of applications. The vnlog toolkit (https://github.com/dkogan/vnlog) adds just a little bit of syntactic sugar to the usual commandline tools to make processing scripts robust and easy to read and write. This was not my intent initially, but I now do most of my data processing with the shell and vnl-wrapped awk (and sort and join, ...) It's really nice. If you write stuff in awk, you should check it out. (Disclaimer: I'm the author)
- Extending Awk with Field Labels
DataProfiler
-
LongRoPE: Extending LLM Context Window Beyond 2M Tokens
It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler
For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.
- Data Profiler โ What's in your data?
-
Data Profiler 0.9.0 -- offering a massive improvement to memory usage during profiling of large datasets
Great call out -- would you be willing to write up an issue for that on the repo? Thank you! https://github.com/capitalone/DataProfiler/issues/new/choose
- FLiPN-FLaNK Stack Weekly for 20 March 2023
- Release 0.8.3 ยท capitalone/DataProfiler
What are some alternatives?
ttyplot - a realtime plotting utility for terminal/console with data input from stdin
ydata-profiling - 1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
matplotplusplus - Matplot++: A C++ Graphics Library for Data Visualization ๐๐พ
pyWhat - ๐ธ Identify anything. pyWhat easily lets you identify emails, IP addresses, and more. Feed it a .pcap file or some text and it'll tell you what it is! ๐งโโ๏ธ
RecordStream - commandline tools for slicing and dicing JSON records.
usaddress - :us: a python library for parsing unstructured United States address strings into address components
nvim-ipy - IPython/Jupyter plugin for Neovim
superset - Apache Superset is a Data Visualization and Data Exploration Platform
jupytext.vim - Vim plugin for editing Jupyter ipynb files via jupytext
XlsxWriter - A Python module for creating Excel XLSX files.
matplotlib - C++ wrappers around python's matplotlib
vtuber-livechat-dataset - ๐ VTuber 1B: Billion-scale Live Chat and Moderation Event Dataset