awesome-generative-ai VS stable-diffusion-webui-tokenizer

If you want links you can check out this list I made: https://github.com/steven2358/awesome-generative-ai

Great list! I've added it to my list on Generative AI under the section "Lists". https://github.com/steven2358/awesome-generative-ai

This: https://github.com/AUTOMATIC1111/stable-diffusion-webui-tokenizer seems to realize that, but I don't know how trustworthy this actually is. I wish to know to be able to have more fine control on what words I may train a model without having unforseen consequences on other parts of the same model.

Tokens are how the AI "reads" text, so that it can convert it to a numerical representation. You can think of them as individual words, but the relation isn't one-to-one. There's actually an extension for AUTOMATIC1111's WebUI for token viewing, and if you look at the example image, you can see that most words are represented by one token, but the word "nimbus" is represented by two. So a "short term" memory of 2000 tokens in KoboldAI means that the AI will base its generations on the last 2000 tokens and "forget" whatever is written before that. You can also define a "long term" memory via the Author's Notes and World Information, and the AI will prefix these before generating when appropriate, so that it "remembers" them better.

Your prompts are converted into numbered tokens. https://github.com/AUTOMATIC1111/stable-diffusion-webui-tokenizer I also saw an extension that let you change the weights of the numbered tokens in a textual inversion embedding. And textual inversion lets you specify how many tokens to use up.

By using the awesome Tokenizer extension for Automatic1111, I've been able to hunt around for uni-tokens in SD- words which are represented by just one token instead of being broken down into multiple ones. I put in a number, hit the Tokenize button, and it tells me what word or set of characters (or emji, etc) that represents. Like this.