LMFlow
CogVLM
LMFlow | CogVLM | |
---|---|---|
10 | 16 | |
8,042 | 5,193 | |
3.5% | 10.2% | |
9.6 | 9.0 | |
5 days ago | 27 days ago | |
Python | Python | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
LMFlow
- Your weekly machine learning digest
-
Any guide/intro to fine-tuning anywhere?
You might want to have a look at LMFlow.
- Robin V2 Launches: Achieves Unparalleled Performance on OpenLLM!
-
[D] Have you tried fine-tuning an open source LLM?
I'd like to recommend LMFlow (https://github.com/OptimalScale/LMFlow), a fast and extensible toolkit for finetuning and inference of large foundation models.
-
[R] DetGPT: Detect What You Need via Reasoning
The "reasoning-based object detection" is a challenging problem because the detector needs to understand and reason about the user's coarse-grained/abstract instructions and analyze the current visual information to locate the target object accurately. In this direction, researchers from the Hong Kong University of Science and Technology and the University of Hong Kong have conducted some preliminary explorations. Specifically, they use a pre-trained visual encoder (BLIP-2) to extract visual features from images and align the visual features to the text space using an alignment function. They use a large-scale language model (Robin/Vicuna) to understand the user's question, combined with the visual information they see, to reason about the objects that users are truly interested in. Then, they provide the object names to the pre-trained detector (Grounding-DINO) for specific location prediction. In this way, the model can analyze the image based on any user instructions and accurately predict the location of the object of interest to the user. It is worth noting that the difficulty here mainly lies in the fact that the model needs to achieve task-specific output formats for different specific tasks as much as possible without damaging the model's original abilities. To guide the language model to follow specific patterns and generate outputs that conform to the object detection format, the research team used ChatGPT to generate cross-modal instruction data to fine-tune the model. Specifically, based on 5000 coco images, they used ChatGPT to create a 30,000 cross-modal image-text fine-tuning dataset. To improve the efficiency of training, they fixed other model parameters and only learned cross-modal linear mapping. Experimental results show that even if only the linear layer is fine-tuned, the language model can understand fine-grained image features and follow specific patterns to perform inference-based image detection tasks, showing excellent performance. This research topic has great potential. Based on this technology, the field of home robots will further shine: people in homes can use abstract or coarse-grained voice instructions to make robots understand, recognize, and locate the objects they need, and provide relevant services. In the field of industrial robots, this technology will bring endless vitality: industrial robots can cooperate more naturally with human workers, accurately understand their instructions and needs, and achieve intelligent decision-making and operations. On the production line, human workers can use coarse-grained voice instructions or text input to allow robots to automatically understand, recognize, and locate the items that need to be processed, thereby improving production efficiency and quality. With object detection models that come with reasoning capabilities, we can develop more intelligent, natural, and efficient robots to provide more convenient, efficient, and humane services to humans. This is a field with broad prospects and deserves more attention and further exploration by more researchers. DetGPT supports multiple language models and has been validated based on two language models, Robin-13B and Vicuna-13B. The Robin series language model is a dialogue model trained by the LMFlow team ( https://github.com/OptimalScale/LMFlow) at the Hong Kong University of Science and Technology, achieving results competitive to Vicuna on multiple language ability evaluation benchmarks (model download: https://github.com/OptimalScale/LMFlow#model-zoo). Previously, the LMFlow team trained a vertical GPT model using a consumer-grade 3090 graphics card in just 5 hours. Today, this team, in collaboration with the NLP Group at the University of Hong Kong, has brought us a multimodal surprise. Welcome to try our demo and open-source code! Online demo: https://detgpt.github.io/ Open-source code: https://github.com/OptimalScale/DetGPT
-
Leaderboard for LLMs? [D]
Hi LMFlow Benchmark (https://github.com/OptimalScale/LMFlow) evaluates 31 open-source LLMs with an automatic metric: negative log likelihood.
-
[R] LMFlow Benchmark: An Automatic Evaluation Framework for Open-Source LLMs
LMFlow: https://github.com/OptimalScale/LMFlow
-
[R] Foundation Model Alignment with RAFT🛶 in LMFlow
Its implementation is available from https://github.com/OptimalScale/LMFlow.
- LMFlow – Toolkit for Finetuning and Inference of Large Foundation Models
CogVLM
-
Mixtral: Mixture of Experts
CogVLM is very good in my (brief) testing: https://github.com/THUDM/CogVLM
The model weights seem to be under a non-commercial license, not true open source, but it is "open access" as you requested.
-
IT Employment Grew by Just 700 Jobs in 2023, Down From 267,000 in 2022
increasing growth most places in world
https://twitter.com/elonmusk/status/1743028102446408026
heres a total feature map of what was released in 2023:
https://twitter.com/enriquebrgn/status/1740950767325024387
I think thats definitely a signal that the B and C teams werent needed, considering they cut 90% of staff LOL.
As for the bots, AI is making it easier than ever to bypass those systems. CogVLM is just sitting there menacingly on github https://github.com/THUDM/CogVLM
- Show HN: I built an open source AI video search engine to learn more about AI
-
CogAgent-18B – visual-based GUI Agent capabilities
Jump to heading for benchmarks and examples: https://github.com/THUDM/CogVLM/tree/main?tab=readme-ov-file...
-
What do you think. When should we expect the next SDXL version?
Honestly at this point there is no need for human for captioning except maybe for NSFW content. Img2text is just good enough for nearly all images. GPTVision or open source equivalent (like CogVLM https://github.com/THUDM/CogVLM ) are just good enough.
-
shinning the spotlight on CogVLM
A core Llama.cpp contributor, named cmp-nct, discovered stumbled upon what might be the next leap forward for vision/language models. CogVLM (which uses a Vicuna 7B language model combined with a 9B vision tower) excels particularly in OCR (Optical Character Recognition), detail detection, and minimal hallucinations. It effectively understands both handwritten and typed text, context, fine details, and background graphics. It even provides pixel coordinates for small visual targets. CovVLM surpasses other models like llava-1.5 and Qwen-VL in performance.
-
Image-to-Caption Generator
https://github.com/THUDM/CogVLM (really impressive)
-
Gemini: Google's most capable AI model yet
I'm researching using LLMs for alt-text suggestion for forum users, can you share your finding so far?
Outside of GPT-4V I had good first results with https://github.com/THUDM/CogVLM
-
Open-source LLMs with Image Interpretation
I've got some decent results with CogVLM. Resolution kinda sucks at 490x490, though.
- FLaNK Stack Weekly for 27 November 2023
What are some alternatives?
axolotl - Go ahead and axolotl questions
LLaVA - [NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.
chatgpt_macro_for_texstudio - The ChatGPT Macro for TeXstudio is a user-friendly integration that connects TeXstudio with OpenAI's API.
ollama - Get up and running with Llama 3, Mistral, Gemma, and other large language models.
llm-foundry - LLM training code for Databricks foundation models
ComfyUI - The most powerful and modular stable diffusion GUI, api and backend with a graph/nodes interface.
const_layout - Official implementation of the MM'21 paper "Constrained Graphic Layout Generation via Latent Optimization" (LayoutGAN++, CLG-LO, and Layout evaluation)
Qwen-VL - The official repo of Qwen-VL (通义千问-VL) chat & pretrained large vision language model proposed by Alibaba Cloud.
giskard - 🐢 Open-Source Evaluation & Testing framework for LLMs and ML models
vimGPT - Browse the web with GPT-4V and Vimium
qlora - QLoRA: Efficient Finetuning of Quantized LLMs
uform - Pocket-Sized Multimodal AI for content understanding and generation across multilingual texts, images, and 🔜 video, up to 5x faster than OpenAI CLIP and LLaVA 🖼️ & 🖋️