DataProfiler
datatable
| DataProfiler | datatable | |
|---|---|---|
| 61 | 9 | |
| 1,363 | 1,790 | |
| 1.0% | 0.5% | |
| 6.3 | 6.1 | |
| 3 days ago | 5 months ago | |
| Python | C++ | |
| Apache License 2.0 | Mozilla Public License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
DataProfiler
-
LongRoPE: Extending LLM Context Window Beyond 2M Tokens
It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler
For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.
- Data Profiler – What's in your data?
-
Data Profiler 0.9.0 -- offering a massive improvement to memory usage during profiling of large datasets
Great call out -- would you be willing to write up an issue for that on the repo? Thank you! https://github.com/capitalone/DataProfiler/issues/new/choose
- FLiPN-FLaNK Stack Weekly for 20 March 2023
- Release 0.8.3 · capitalone/DataProfiler
datatable
-
Cheat Sheets for data.table to Python's pandas syntax?
Aside from that, there is a Python translation of data.table (see documentation here), which might be worth looking into. However, it hasn't had any major updates in a while: the last release 2 years ago ...
- Any advice on using Pandas as a data analyst?
-
Alternative to Pandas
There's datatable. I haven't used it much, but the R version (data.table) is phenomenal.
-
Need advice on whether to store data set for regression model in SQL database or by using Python modules like Pickle or Parquet
just use HDF5 or Parquet, or CSV + https://github.com/h2oai/datatable to speed up the file reading.
- Massive R analysis of Data Science Language and Job Trends 2022
-
Scikit-Learn Version 1.0
> For me I had with pandas the most issues using it's multiindex.
Yessss. I loathe indices, and have never been in a situation where I was better off with them than without them.
> Regarding fast you have something like Vaex on python sid
I've never used Vaex, but I've used datatable (https://github.com/h2oai/datatable) and polars (https://github.com/pola-rs/polars). Polars is my favorite API, but datatable was faster at reading data (Polars was faster in execution). I'll have to give Vaex a try at some point.
- Show HN: Sheet2dict – simple Python XLSX/CSV reader/to dictionary converter
-
Hey Reddit, here's my comprehensive course on Python Pandas, for free.
Yep. I think this is the downside to a package being entirely maintained by volunteers. In any case, Pandas is still the leading data wrangling package for Python. (I'm excited to see how datatable evolves.)
-
Ditching Excel for Python in a Legacy Industry (Reinsurance)
h2o's data.table clone is fine
https://github.com/h2oai/datatable
What are some alternatives?
ydata-profiling - 1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
polars - Dataframes powered by a multithreaded, vectorized query engine, written in Rust
pyWhat - 🐸 Identify anything. pyWhat easily lets you identify emails, IP addresses, and more. Feed it a .pcap file or some text and it'll tell you what it is! 🧙♀️
DataFrame - C++ DataFrame for statistical, Financial, and ML analysis -- in modern C++ using native types and contiguous memory storage
usaddress - :us: a python library for parsing unstructured United States address strings into address components
db-benchmark - reproducible benchmark of database-like ops
XlsxWriter - A Python module for creating Excel XLSX files.
scientific-visualization-book - An open access book on scientific visualization using python and matplotlib
superset - Apache Superset is a Data Visualization and Data Exploration Platform
sktime - A unified framework for machine learning with time series
vtuber-livechat-dataset - 📊 VTuber 1B: Billion-scale Live Chat and Moderation Event Dataset
vinum - Vinum is a SQL processor for Python, designed for data analysis workflows and in-memory analytics.