Top 8 Go Tokenizer Projects

kagome

1 789 6.4 Go

Self-contained Japanese Morphological Analyzer written in pure Go
sentences

0 419 4.5 Go

A multilingual command line sentence tokenizer in Golang
InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
tokenizer

1 136 6.1 Go

NLP tokenizers written in Go language (by sugarme)
tokenizer

0 83 4.0 Go

Tokenizer (lexer) for golang (by bzick)
goselect

1 29 10.0 Go

SQL like 'select' interface for files
assocentity

1 14 4.1 Go

Package assocentity returns the mean distance from tokens to an entity and its synonyms
shamoji

0 13 0.0 Go

The shamoji (杓文字) is a word filtering package
WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
maleeni

1 5 0.0 Go

A lexer generator for golang

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).