Top 5 tiktoken Open-Source Projects
-
Scout Monitoring
Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.
-
1filellm
Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion
-
kgpt
A custom GPT based on [Zero To Hero](https://karpathy.ai/zero-to-hero.html) utilizing tiktoken with the intent to augment AI Transformer-model education and reverse engineer GPT models from scratch.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
Project mention: Show HN: FileKitty – Combine and label text files for LLM prompt contexts | news.ycombinator.com | 2024-05-01I created something similar, https://github.com/jimmc414/1filellm
It converts papers, repositories, PRs and web docs into one text file for llm ingestion
Project mention: A custom GPT based on Andrej Karpathy's Zero To Hero utilizing tiktoken with the intent to augment AI Transformer-model education and reverse engineer GPT models from scratch. | /r/aipromptprogramming | 2023-06-21
Tested on Apple Silicon M1 Pro, with Python 3.11.6 and Node.js 21.2.0. Here's the repo.
tiktoken discussion
tiktoken related posts
Index
What are some of the best open-source tiktoken projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | tiktoken-go | 585 |
| 2 | 1filellm | 253 |
| 3 | gpt4-tokenizer-visualizer | 20 |
| 4 | kgpt | 13 |
| 5 | tiktoken-bench | 2 |
Sponsored