datagen
DataProfiler
| datagen | DataProfiler | |
|---|---|---|
| 7 | 61 | |
| 135 | 1,363 | |
| 3.0% | 1.0% | |
| 6.1 | 6.3 | |
| about 2 months ago | 7 days ago | |
| TypeScript | Python | |
| Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
datagen
-
What are your favorite tools or components in the Kafka ecosystem?
For fake data, shameless plug for https://github.com/MaterializeInc/datagen/tree/main
- What are some good publicly available real-time data sources?
-
Simulating Streaming Data for Fraud Detection with Datagen CLI
Building and testing a real-time fraud detection application requires a continuous stream of realistic data. But generating that data can be a challenge. That's why we recently created the Datagen CLI, a simple tool that helps you create believable fake data using the FakerJS API.
-
How train my SQL skills with real world data engineering problems ?
Generate fake data with a normalized schema of your choosing with this tool from Materialize, then denormalize it and build a warehouse model.
- FLiPN-FLaNK Stack Weekly for 20 March 2023
- Datagen CLI: Stream Fake Relational Data
DataProfiler
-
LongRoPE: Extending LLM Context Window Beyond 2M Tokens
It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler
For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.
- Data Profiler – What's in your data?
-
Data Profiler 0.9.0 -- offering a massive improvement to memory usage during profiling of large datasets
Great call out -- would you be willing to write up an issue for that on the repo? Thank you! https://github.com/capitalone/DataProfiler/issues/new/choose
- FLiPN-FLaNK Stack Weekly for 20 March 2023
- Release 0.8.3 · capitalone/DataProfiler
What are some alternatives?
ChatGLM-6B - ChatGLM-6B: An Open Bilingual Dialogue Language Model | 开源双语对话语言模型
ydata-profiling - 1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
CloudDemo2023 - 2023 Demos
pyWhat - 🐸 Identify anything. pyWhat easily lets you identify emails, IP addresses, and more. Feed it a .pcap file or some text and it'll tell you what it is! 🧙♀️
halp - A CLI tool to get help with CLI tools 🐙
usaddress - :us: a python library for parsing unstructured United States address strings into address components
awesome-public-real-time-datasets - A list of publicly available datasets with real-time data maintained by the team at bytewax.io
XlsxWriter - A Python module for creating Excel XLSX files.
RedfinScraper - Scrapes Redfin data.
superset - Apache Superset is a Data Visualization and Data Exploration Platform
cf-url-shortener - URL Shortener Cloudflare function that uses Upstash Redis and Kafka along with https://materialize.com
vtuber-livechat-dataset - 📊 VTuber 1B: Billion-scale Live Chat and Moderation Event Dataset