tiktoken
glow
tiktoken | glow | |
---|---|---|
32 | 60 | |
9,980 | 14,860 | |
6.4% | 1.9% | |
6.7 | 6.7 | |
about 1 month ago | 7 days ago | |
Python | Go | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
tiktoken
- FLaNK AI - 01 April 2024
-
GPT-3.5 crashes when it thinks about useRalativeImagePath too much
Their tokenizer is open source: https://github.com/openai/tiktoken
Data files that contain vocabulary are listed here: https://github.com/openai/tiktoken/blob/9e79899bc248d5313c7d...
-
How fast is JS tiktoken?
OpenAI's refference tokeniser - https://github.com/openai/tiktoken
-
Anthropic announces Claude 2.1 – 200k context, less refusals
ChatGPT presumably adds them as special tokens to the cl100k_base tokenizer, as they demo in the tiktoken documentation: https://github.com/openai/tiktoken#extending-tiktoken
-
What is the best way to get an approximate number of tokens for a piece of text?
I want to measure the approximate number of tokens in a piece of text to understand if I will need to modify it before passing it into the context of an OpenAI API call. Tiktoken can do this, but I'm not sure if it's overkill to use that library just for this simple task. I don't need to actually tokenize the text, I just need an approximate count (e.g. within like 1% of the text's actual token length for text that represents the visible text on a webpage).
-
Show HN: LLaMA tokenizer that runs in browser
https://platform.openai.com/tokenizer or the official python library tiktoken https://github.com/openai/tiktoken or this JS port of tiktoken https://github.com/dqbd/tiktoken
-
Made a GPT-3.5-Turbo and GPT-4 Tokenizer
It's built on top of the tiktoken library and is basically just a lambda function in the backend.
- AiPrice - an API for calculating OpenAI tokens and pricing
-
Anyone able to explain what happened here?
"All" is a single token in OpenAI's tiktoken Tokenizer, unrelated to the token for capital "A". Even lowercase "all" is a distinct token from "All" or "ALL."
-
Which lib is the tokenizer page using to calculate the tokens?
check tiktoken
glow
-
Essential Command Line Tools for Developers
To get started, install Mods and check out some of the examples below. Since Mods has built-in Markdown formatting, you may also want to grab Glow to give the output some pizzazz.
- Ask HN: How do you synchronise your notes?
-
Not trying to start a rumble, but why neovim
I recently started using markdown in neovim (with an LSP) along with https://github.com/charmbracelet/glow to view markdown / navigate. Does everything I used to use Obsidian for minus the links / graph functionality which I don't really need and it's pretty snappy on an old Lenovo. Very customizable as well.
- How would you read your files if Obsidian disappeared?
-
Show HN: GPT-engineer – platform for devs to tinker with AI programming tools
Yup, those seem to be the key challenges. I've been making good progress on them, but there's plenty more work to do!
On the topic of "AI-generated PRs", I used my tool to file a PR to the `glow` CLI tool. I don't know the go language, so I had aider make the changes to glow.
https://github.com/charmbracelet/glow/pull/502
I've also been able solve a couple of github issues that were file by users by just pasting the issue into my tool... it fixed itself. Links below:
https://github.com/paul-gauthier/aider/issues/13#issuecommen...
https://github.com/paul-gauthier/aider/issues/5#issuecomment...
- FLiPN-FLaNK Stack Weekly May 8 2023
-
How to host your own Golang based Git server for the command line.
I'm personally also quite fond of Glow. I use it pretty much every time I touch a markdown file.
-
Show HN: Frogmouth – A Markdown browser for your terminal
Nice idea! I’m excited to check it out. I write a lot of docs in Markdown and this could be a great way to browse them.
Out of curiosity, have you seen glow[0]?
[0] https://github.com/charmbracelet/glow
-
Recommendations on file/dir/module structure, common dependencies, and/or anti-patterns for writing CLI tool in Rust
Charm's Glow is a joy to use, a good example of having the Charm's Bubbletea usage - but from the code perspective, it's a bit difficult to navigate as many code paths are put in the same package
-
AI - a commandline ChatGPT client in with conversation/completion support
thanks! Yeah all the markdown is handled through glow, which is one fairly awesome tool.
What are some alternatives?
tokenizer - Pure Go implementation of OpenAI's tiktoken tokenizer
markdown-preview.nvim - markdown preview plugin for (neo)vim
daath-ai-parser - Daath AI Parser is an open-source application that uses OpenAI to parse visible text of HTML elements.
pcstat - Page Cache stat: get page cache stats for files on Linux
CLIP - CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image
mdless
skypilot - SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution—all with a simple interface.
mdcat - cat for markdown
bricks - Open-source natural language enrichments at your fingertips.
glow.nvim - A markdown preview directly in your neovim.
terminal-copilot - A smart terminal assistant that helps you find the right command.
bat - A cat(1) clone with wings.