CogVLM VS ollama

CogVLM is very good in my (brief) testing: https://github.com/THUDM/CogVLM

The model weights seem to be under a non-commercial license, not true open source, but it is "open access" as you requested.

increasing growth most places in world

https://twitter.com/elonmusk/status/1743028102446408026

heres a total feature map of what was released in 2023:

https://twitter.com/enriquebrgn/status/1740950767325024387

I think thats definitely a signal that the B and C teams werent needed, considering they cut 90% of staff LOL.

As for the bots, AI is making it easier than ever to bypass those systems. CogVLM is just sitting there menacingly on github https://github.com/THUDM/CogVLM

Jump to heading for benchmarks and examples: https://github.com/THUDM/CogVLM/tree/main?tab=readme-ov-file...

Honestly at this point there is no need for human for captioning except maybe for NSFW content. Img2text is just good enough for nearly all images. GPTVision or open source equivalent (like CogVLM https://github.com/THUDM/CogVLM ) are just good enough.

A core Llama.cpp contributor, named cmp-nct, discovered stumbled upon what might be the next leap forward for vision/language models. CogVLM (which uses a Vicuna 7B language model combined with a 9B vision tower) excels particularly in OCR (Optical Character Recognition), detail detection, and minimal hallucinations. It effectively understands both handwritten and typed text, context, fine details, and background graphics. It even provides pixel coordinates for small visual targets. CovVLM surpasses other models like llava-1.5 and Qwen-VL in performance.

https://github.com/THUDM/CogVLM (really impressive)

I'm researching using LLMs for alt-text suggestion for forum users, can you share your finding so far?

Outside of GPT-4V I had good first results with https://github.com/THUDM/CogVLM

I've got some decent results with CogVLM. Resolution kinda sucks at 490x490, though.

- Basic internet search (I start ollama CLI faster than I can start a browser - https://ollama.com)

- Formatting/changing text

- Troubleshooting code, esp. new frameworks/libs

- Recipes

- Data entry

- Organizing thoughts: High-level lists, comparison, classification, synonyms, jargon & nomenclature

- Learning esp. by analogy and example

RAG for:

- Website assistants (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)

- Game NPCs (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)

- Discord/Slack/forum bots (https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...)

- Character-driven storytelling and creating art in a specific style for video game loading screens, background images, avatars, website art, etc. (https://github.com/bennyschmidt/ragdoll-studio/tree/master/r...)

Jan goes a step further by integrating with other local engines like LM Studio and ollama.

# install the Ollama curl -fsSL https://ollama.com/install.sh | sh # get the llama3 model ollama pull llama2 # install the MLFlow pip install mlflow

Ollama for running LLMs locally

curl -fsSL https://ollama.com/install.sh | sh

Streaming is not a problem (it's just a simple flag: https://github.com/wiktor-k/llama-chat/blob/main/index.ts#L2...) but I've never used voice input.

The examples show image input though: https://github.com/ollama/ollama/blob/main/docs/api.md#reque...

Maybe you can file an issue here: https://github.com/ollama/ollama/issues

I’m a huge fan of open source models, especially the newly release Llama 3. Because of the performance of both the large 70B Llama 3 model as well as the smaller and self-host-able 8B Llama 3, I’ve actually cancelled my ChatGPT subscription in favor of Open WebUI, a self-hostable ChatGPT-like UI that allows you to use Ollama and other AI providers while keeping your chat history, prompts, and other data locally on any computer you control.