Tokenizer Alternatives

Similar projects and alternatives to tokenizer based on common topics and language

• maleeni

  1 5 0.0 Go tokenizer VS maleeni

  A lexer generator for golang

• spaGO

  11 1,693 0.0 Go tokenizer VS spaGO

  Discontinued Self-contained Machine Learning and Natural Language Processing library in Go

  

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• sentences

  0 419 4.5 Go tokenizer VS sentences

  A multilingual command line sentence tokenizer in Golang

• kagome

  1 789 6.4 Go tokenizer VS kagome

  Self-contained Japanese Morphological Analyzer written in pure Go

• assocentity

  1 14 4.1 Go tokenizer VS assocentity

  Package assocentity returns the mean distance from tokens to an entity and its synonyms

• cybertron

  1 258 7.2 Go tokenizer VS cybertron

  Cybertron: the home planet of the Transformers in Go (by nlpodyssey)