Top 3 Python Pii Projects

presidio

5 3,077 8.9 Python

Context aware, pluggable and customizable data protection and de-identification SDK for text and images

Project mention: You can't build a moat with AI | news.ycombinator.com | 2024-04-11

Perhaps de-identification before training could be helpful here.
Microsoft does seem active in this, e.g. https://microsoft.github.io/presidio/

DataProfiler

61 1,362 6.3 Python

What's in your data? Extract schema, statistics and entities from datasets

Project mention: LongRoPE: Extending LLM Context Window Beyond 2M Tokens | news.ycombinator.com | 2024-02-22

It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler
For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.

WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
metacrafter

1 38 3.7 Python

Metadata and data identification tool and Python library. Identifies PII, common identifiers, language specific identifiers. Fully customizable and flexible rules

Project mention: Metacrafter – semantic data types detection Python lib | news.ycombinator.com | 2024-03-13

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).