llm-mlc VS llm-cluster

I'm still iterating on that. Plugins get complete control over the prompts, so they can handle the various weirdnesses of them. Here's some relevant code:

https://github.com/simonw/llm-gpt4all/blob/0046e2bf5d0a9c369...

https://github.com/simonw/llm-mlc/blob/b05eec9ba008e700ecc42...

https://github.com/simonw/llm-llama-cpp/blob/29ee8d239f5cfbf...

I'm not completely happy with this yet. Part of the problem is that different models on the same architecture may have completely different prompting styles.

I expect I'll eventually evolve the plugins to allow them to be configured in an easier and more flexible way. Ideally I'd like you to be able to run new models on existing architectures using an existing plugin.

What is the advantage of this versus running something like https://github.com/simonw/llm , which also gives you options to e.g. use https://github.com/simonw/llm-mlc for accelerated inference?

I'm quite impressed with Llama 2 13B - the more time I spend with it the more I think it might be genuinely useful for more than just playing around with local LLMs.

I'm using the MLC version (since that works with a GPU on my M2 Mac) via my https://github.com/simonw/llm-mlc plugin.

I'm trying to understand the clustering code but not doing too well.

https://github.com/simonw/llm-cluster/blob/main/llm_cluster....

So does this take each row from the DB, convert to a numpy array (?), then uses an existing model called MiniBatchKMeans (?) to go over that array and generate a bunch of labels. Then add it to a dictionary and print to console.

I imagine there are all kinds of improvements that could be made to this kind of thing.

I'd love to understand if there's a good way to automatically pick an interesting number of clusters, as opposed to picking a number at the start.

https://github.com/simonw/llm-cluster/blob/main/llm_cluster....