DataProfiler
presidio
Our great sponsors
| DataProfiler | presidio | |
|---|---|---|
| 61 | 5 | |
| 1,362 | 3,102 | |
| 2.5% | 9.8% | |
| 6.3 | 8.9 | |
| 6 days ago | 1 day ago | |
| Python | Python | |
| Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
DataProfiler
-
LongRoPE: Extending LLM Context Window Beyond 2M Tokens
It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler
For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.
- Data Profiler – What's in your data?
-
Data Profiler 0.9.0 -- offering a massive improvement to memory usage during profiling of large datasets
Great call out -- would you be willing to write up an issue for that on the repo? Thank you! https://github.com/capitalone/DataProfiler/issues/new/choose
- FLiPN-FLaNK Stack Weekly for 20 March 2023
- Release 0.8.3 · capitalone/DataProfiler
presidio
-
You can't build a moat with AI
Perhaps de-identification before training could be helpful here.
Microsoft does seem active in this, e.g. https://microsoft.github.io/presidio/
- Presidio – Data Protection and De-Identification SDK
-
Show HN: Cape API – Keep your sensitive data private while using GPT-4
Something like https://github.com/microsoft/presidio for stripping out PII might fill the role I expected https://github.com/capeprivacy/private-ai to do.
-
Handling PII data in Azure
Depending on your use case, you may want to check out Presidio as well. It’s a Microsoft product for PII scrubbing. Perfect for ADF and Synapse pipelines.
-
Data Anonymization
There's an API from Microsoft, named Presidio which is used for Anonymization. This is the Github link.
What are some alternatives?
ydata-profiling - 1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
exodus - Platform to audit trackers used by Android application
pyWhat - 🐸 Identify anything. pyWhat easily lets you identify emails, IP addresses, and more. Feed it a .pcap file or some text and it'll tell you what it is! 🧙♀️
databunker - A secure user directory built for developers to comply with the GDPR [Moved to: https://github.com/securitybunker/databunker]
usaddress - :us: a python library for parsing unstructured United States address strings into address components
Databunker - Secure SDK/vault for personal records/PII built to comply with GDPR
XlsxWriter - A Python module for creating Excel XLSX files.
techbench-json-dump - Dump Tech Bench metadata to a JSON file.
superset - Apache Superset is a Data Visualization and Data Exploration Platform
PrivacyEngCollabSpace - Privacy Engineering Collaboration Space
vtuber-livechat-dataset - 📊 VTuber 1B: Billion-scale Live Chat and Moderation Event Dataset
private-ai - Repo for Udacity's Secure & Private AI course