tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models. (by openai)

Tiktoken Alternatives

Similar projects and alternatives to tiktoken

  1. node

    1,445 tiktoken VS node

    Node.js JavaScript runtime ✨🐢🚀✨

  2. SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
  3. ollama

    Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

  4. Hugo

    The world’s fastest framework for building websites.

  5. whisper

    423 tiktoken VS whisper

    Robust Speech Recognition via Large-Scale Weak Supervision

  6. pgvector

    Open-source vector similarity search for Postgres

  7. CLIP

    113 tiktoken VS CLIP

    CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image

  8. FLiPStackWeekly

    FLaNK AI Weekly covering Apache NiFi, Apache Flink, Apache Kafka, Apache Spark, Apache Iceberg, Apache Ozone, Apache Pulsar, and more...

  9. glow

    85 tiktoken VS glow

    Render markdown on the CLI, with pizzazz! 💅🏻

  10. XcodeBenchmark

    XcodeBenchmark measures the compilation time of a large codebase on iMac, MacBook, and Mac Pro

  11. devspace

    DevSpace - The Fastest Developer Tool for Kubernetes ⚡ Automate your deployment workflow with DevSpace and develop software directly inside Kubernetes.

  12. chat-with-gpt

    39 tiktoken VS chat-with-gpt

    An open-source ChatGPT app with a voice

  13. zenml

    36 tiktoken VS zenml

    ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.

  14. sentencepiece

    Unsupervised text tokenizer for Neural Network-based text generation.

  15. graphql-ws

    21 tiktoken VS graphql-ws

    Coherent, zero-dependency, lazy, simple, GraphQL over WebSocket Protocol compliant server and client.

  16. tokenizers

    💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

  17. tensor2tensor

    9 tiktoken VS tensor2tensor

    Discontinued Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.

  18. openai-cookbook

    249 tiktoken VS openai-cookbook

    Examples and guides for using the OpenAI API

  19. mcptoon

    8 tiktoken VS mcptoon

    One zero-dependency MCP gateway CLI for every server and agent skill. Token optimization, tool discovery and context compression: 71,929 -> 581 tokens (-99.2%, measured), schemas stay out of context. One config for Claude Code, Codex, Cursor, every agent. 561KB, pure Python. | 零依赖 CLI:管所有 MCP 工具与技能,工具发现省 99.2% token。

  20. meetups

    Meetup Materials (by tspannhw)

  21. tokenizer-go

    A Go wrapper for GPT-3 token encode/decode. https://platform.openai.com/tokenizer

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better tiktoken alternative or higher similarity.

tiktoken discussion

Log in or Post with

Stats

Basic tiktoken repo stats
96
19,669
5.8
8 days ago

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com