FastChat VS mlc-llm

FastChat

How does it compare with something like FastChat? https://github.com/lm-sys/FastChat

Feature set seems like a decent amount of overlap. One limitation of FastChat, as far as I can tell, is that one is limited to the models that FastChat supports (though I think it would be minor to modify it to support arbitrary models?)

Looks like the Vicuna repo is Apache 2.0 also[1].

What's the interpretation of copyright law that would prevent the code being Apache 2.0 based on the source of the fine-tuning dataset?

[1] https://github.com/lm-sys/FastChat

Check how to start with FastChat. Support FastChat on GitHub ⭐

For something a little heavier but much more robust in terms of features/functionality I've been enjoying FastChat: https://github.com/lm-sys/FastChat

It allows you to plug in different backends so that you can use OpenAI compatible clients with various LLM's, selfhosted or otherwise.

This one uses gpu, it doesn't support Mistral yet: https://github.com/mlc-ai/mlc-llm

I have tried running mistral 7B with MLC on my m1 metal. And it kept crushing (git issue with description). Memory inefficiency problems.

Project: https://github.com/mlc-ai/mlc-llm

For LLM inference, a shoutout to MLC LLM, which runs LLM models on basically any API that's widely available: https://github.com/mlc-ai/mlc-llm

One of your problems might be that gfx1032 is not supported by AMD's ROCm packages, which has a laughably short list of supported hardware: https://rocm.docs.amd.com/en/latest/release/gpu_os_support.h...

The normal workaround is to assign the closest architecture, eg gfx1030, so `HSA_OVERRIDE_GFX_VERSION=10.3.0` might help

Also, it looks like some of your tested projects are OpenCL? For me, I do something like: `yay -S rocm-hip-sdk rocm-ml-sdk rocm-opencl-sdk` to cover all the bases.

My recent interest has been LLMs and this is my general step by step for those (llama.cpp, exllama) for those interested: https://llm-tracker.info/books/howto-guides/page/amd-gpus

I didn't port the docs back in, but also here's a step-by-step w/ my adventures getting TVM/MLC working w/ an APU: https://github.com/mlc-ai/mlc-llm/issues/787

From my experience, ROCm is improving, but there's a good reason that Nvidia has 90% market share even at big price premiums.

Maybe they're talking about https://github.com/mlc-ai/mlc-llm which is used for web-llm (https://github.com/mlc-ai/web-llm)? Seems to be using TVM.

you already have TVM for the cross platform stuff

see https://tvm.apache.org/docs/how_to/deploy/android.html

or https://octoml.ai/blog/using-swift-and-apache-tvm-to-develop...

or https://github.com/mlc-ai/mlc-llm