awesome-generative-ai
stable-diffusion-webui-tokenizer
awesome-generative-ai | stable-diffusion-webui-tokenizer | |
---|---|---|
6 | 5 | |
4,863 | 128 | |
- | - | |
8.7 | 10.0 | |
8 days ago | over 1 year ago | |
Python | ||
Creative Commons Zero v1.0 Universal | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
awesome-generative-ai
- Generative AI – A curated list of Generative AI projects and services
- Using AI to solve individual problems and societal challenges in Pakistan
-
It would be even more interesting if we had numbers of users for each
If you want links you can check out this list I made: https://github.com/steven2358/awesome-generative-ai
- Which image would serve best as a cover to represent all of LLM-based generative AI?Here are some I generated -- other suggestions are welcome. (Must be panoramic though.)
-
Over the past several months I've put together a spreadsheet of 470 categorized SD resources and apps. Put it up online in case it helps someone (should be the biggest public list so far)
Great list! I've added it to my list on Generative AI under the section "Lists". https://github.com/steven2358/awesome-generative-ai
stable-diffusion-webui-tokenizer
-
How are prompt words tokenized in Stable Diffusion?
This: https://github.com/AUTOMATIC1111/stable-diffusion-webui-tokenizer seems to realize that, but I don't know how trustworthy this actually is. I wish to know to be able to have more fine control on what words I may train a model without having unforseen consequences on other parts of the same model.
-
It would be even more interesting if we had numbers of users for each
Tokens are how the AI "reads" text, so that it can convert it to a numerical representation. You can think of them as individual words, but the relation isn't one-to-one. There's actually an extension for AUTOMATIC1111's WebUI for token viewing, and if you look at the example image, you can see that most words are represented by one token, but the word "nimbus" is represented by two. So a "short term" memory of 2000 tokens in KoboldAI means that the AI will base its generations on the last 2000 tokens and "forget" whatever is written before that. You can also define a "long term" memory via the Author's Notes and World Information, and the AI will prefix these before generating when appropriate, so that it "remembers" them better.
- Automatic1111 prompt simplifier plugin
-
The difference between DreamBooth models, and Textual inversion embeddings, and why we should start pushing toward training embeddings instead of models.
Your prompts are converted into numbered tokens. https://github.com/AUTOMATIC1111/stable-diffusion-webui-tokenizer I also saw an extension that let you change the weights of the numbered tokens in a textual inversion embedding. And textual inversion lets you specify how many tokens to use up.
-
Has somebody dumped the ~50k 'text' tokens in SD 1.5.ckpt yet? If so, where? I'll share some examples I've found to illustrate:
By using the awesome Tokenizer extension for Automatic1111, I've been able to hunt around for uni-tokens in SD- words which are represented by just one token instead of being broken down into multiple ones. I put in a number, hit the Tokenize button, and it tells me what word or set of characters (or emji, etc) that represents. Like this.
What are some alternatives?
KoboldAI-Client
shift-attention - In stable diffusion, generate a sequence of images shifting attention in the prompt.
awesome-ai-tools - A curated list of AI-powered tools
stable-diffusion-webui-prompt-utilities - A set of utilities for the stable-diffusion-webui
generative-ai-dashboard
awesome-ai - A curated list of artificial intelligence resources (Courses, Tools, App, Open Source Project)
stable-diffusion-webui - Stable Diffusion web UI
whisper - Robust Speech Recognition via Large-Scale Weak Supervision
test_my_prompt - This script is to test your prompts with the AUTOMATIC1111 webui
hallucination-leaderboard - Leaderboard Comparing LLM Performance at Producing Hallucinations when Summarizing Short Documents
list-ai-extensions-chrome-addons-firefox - Complete list of AI-powered Extensions for Google Chrome & Addons for Firefox