Top 20 Tokenization Open-Source Projects

• spaCy

  106 28,704 9.2 Python

  💫 Industrial-strength Natural Language Processing (NLP) in Python

  

Project mention: Step by step guide to create customized chatbot by using spaCy (Python NLP library) | dev.to | 2024-03-23

Hi Community, In this article, I will demonstrate below steps to create your own chatbot by using spaCy (spaCy is an open-source software library for advanced natural language processing, written in the programming languages Python and Cython):

• lunasec

  36 1,406 5.5 TypeScript

  LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/

  

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Databunker

  36 1,208 5.7 Go

  Secure SDK/vault for personal records/PII built to comply with GDPR

  

• Ravencoin

  76 1,070 0.0 C

  Ravencoin Core integration/staging tree

  

• trankit

  1 705 6.5 Python

  Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

• razdel

  1 243 2.1 Python

  Rule-based token, sentence segmentation for Russian language

  

• TokenScript

  1 238 0.0 JavaScript

  TokenScript schema, specs and paper

  

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• simplemma

  0 125 5.2 Python

  Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

• l8w8jwt

  1 123 7.8 C

  Minimal, OpenSSL-less and super lightweight JWT library written in C.

Project mention: L8w8jwt – a minimal, OpenSSL-less and lightweight JWT library written in C | news.ycombinator.com | 2023-07-23

• python-fpe

  1 79 5.7 Python

  FPE - Format Preserving Encryption with FF3 in Python

  

• cashtokens

  38 44 3.6

  A proposal to enable two new primitives on Bitcoin Cash: fungible tokens and non-fungible tokens.

  

Project mention: Cashtokens? | /r/btc | 2023-07-02

Useful References: https://cashtokens.org/ https://helpme.cash/CashTokens/

• xontrib-output-search

  1 37 6.2 Python

  Get identifiers, paths, URLs and words from the previous command output and use them for the next command in xonsh shell.

• bioseq

  1 19 2.6 C++

  Tokenizers and Machine Learning Models for biological sequence data

Project mention: ResearchAgent: Iterative Research Idea Generation Using LLMs | news.ycombinator.com | 2024-04-20

• mongo-search

  1 14 10.0 TypeScript

  Fuzzy Text Search And Autocompletion With MongoDB And Node.js

• wink-eng-lite-model

  5 10 0.0

  English lite language model for wink-nlp.

  

• boolean-expression-parser

  1 10 4.5 C#

  Boolean expression parser

• cashtokens.org

  2 9 5.6 JavaScript

  A community-maintained website about the CashTokens technology, including technical specifications, documentation, guides, and other resources.

  

• maleeni

  1 5 0.0 Go

  A lexer generator for golang

• pass3d

  1 4 3.5 Rust

  3D object recognition CLI tool for Linux

  

• dave.liquid.wine

  1 0 2.6 Mustache

  Blockstream AMP tokenization example

• SaaSHub

  www.saashub.com sponsored

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Tokenization related posts

• Guys, I taught ChatGPT to browse the internet and it is bloody amazing.
  1 project | /r/geek | 13 Mar 2023
• HIVEOS Raven Coin Mining Pool
  1 project | /r/Ravencoin | 25 Dec 2022
• Malicious Python Packages Replace Crypto Addresses in Developer Clipboards
  1 project | /r/netsec | 7 Nov 2022
• Evrmore Airdrop- the safe way without exposing your seed words.
  1 project | /r/Ravencoin | 11 Oct 2022
• Ignore 98% of dependency alerts: introducing Semgrep Supply Chain
  1 project | news.ycombinator.com | 4 Oct 2022
• Ask HN: How do you deploy your weekend project in 2022?
  3 projects | news.ycombinator.com | 9 Sep 2022
• Cdk8s: CNCF-Backed Infrastructure-as-Code (IaC) for Kubernetes
  2 projects | news.ycombinator.com | 6 Sep 2022
• A note from our sponsor - SaaSHub
  www.saashub.com | 25 Apr 2024

  SaaSHub helps you find the best software and product alternatives Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com