Top 8 Python Tokenizer Projects

bitextor

2 278 5.9 Python

Bitextor generates translation memories from multilingual websites
sentence-splitter

1 203 0.0 Python

Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.
InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
simplemma

0 125 5.2 Python

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency
spacy-experimental

5 93 4.2 Python

🧪 Cutting-edge experimental spaCy components and features
xontrib-output-search

1 37 6.2 Python

Get identifiers, paths, URLs and words from the previous command output and use them for the next command in xonsh shell.
Hebrew-Tokenizer

1 25 0.0 Python

A very simple python tokenizer for Hebrew text.
tokenstream

2 12 4.8 Python

A versatile token stream for handwritten parsers.
WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
word-piece-tokenizer

1 5 3.8 Python

A Lightweight Word Piece Tokenizer

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).