llamafile VS ok-robot

I think the llamafile[0] system works the best. Binary works on the command line or launches a mini webserver. Llamafile offers builds of Mixtral-8x7B-Instruct, so presumably they may package this one up as well (potentially a quantized format).

You would have to confirm with someone deeper in the ecosystem, but I think you should be able to run this new model as is against a llamafile?

[0] https://github.com/Mozilla-Ocho/llamafile

Thermostats: https://www.sinopetech.com/en/products/thermostat/

I haven't tried running a local text-to-speech engine backed by an LLM to control Home Assistant. Maybe someone is working on this already?

TTS: https://github.com/SYSTRAN/faster-whisper

LLM: https://github.com/Mozilla-Ocho/llamafile/releases

LLM: https://huggingface.co/TheBloke/Nous-Hermes-2-Mixtral-8x7B-D...

It would take some tweaking to get the voice commands working correctly.

While I did not succeed in making the matmul code from https://github.com/Mozilla-Ocho/llamafile/blob/main/llamafil... work in isolation, I compared eigen, openblas, and mkl: https://gist.github.com/Dobiasd/e664c681c4a7933ef5d2df7caa87...

In this (very primitive!) benchmark, MKL was a bit better than eigen (~10%) on my machine (i5-6600).

Since the article https://justine.lol/matmul/ compared the new kernels with MLK, we can (by transitivity) compare the new kernels with Eigen this way, at least very roughly for this one use-case.

Yes, they're just ZIP files that also happen to be actually portable executables.

https://github.com/Mozilla-Ocho/llamafile?tab=readme-ov-file...

have you seen llamafile[0]?

[0] https://github.com/Mozilla-Ocho/llamafile

llama.cpp has integrated gemma support. So you can use llamafile for this. It is a standalone executable that is portable across most popular OSes.

https://github.com/Mozilla-Ocho/llamafile/releases

So, download the executable from the releases page under assets. You want either just main or just server. Don't get the huge ones with the model inlined in the file. The executable is about 30MB in size,

https://github.com/Mozilla-Ocho/llamafile/releases/download/...

The improvements in ease of use for locally hosting LLMs over the last few months have been amazing. I was ranting about how easy https://github.com/Mozilla-Ocho/llamafile is just a few hours ago [1]. Now I'm torn as to which one to use :)

1: Quite literally hours ago: https://euri.ca/blog/2024-llm-self-hosting-is-easy-now/

That's it, isn't it. The question is not, how far away from that are we, but when can you and I actually afford it? Because, as the other commenter snarkily replies, human maid's already exist. The lifestyle of the singularity is already here for the rich. It's trickling down that kind of lifestyle to the rest of us that AI robots will enable. (with some amount of social upheaval.)

Lets say the robot that can do that comes out next year for $15 million. Could you afford one? I certainly can't. So pretend that it does, what changes for you and I? Nothing. So the robots that can do that won't be used as robot maids until the price comes down. Which; it will. Open source robotics and model-available AI will force things to be affordable sooner, rather than later, because we'd all like a robot to do that for us.

The industrial versions will be used to do hideously dangerous things. underwater welding, chainsaw helicoptering, manual nuclear reactor rod removal. We already use machines for a lot of those difficult/impossible tasks, it's just a matter of programming the robots.

Which takes us back to today. How far away from that are we? The pieces are already here. Between https://ok-robot.github.io/ and https://mobile-aloha.github.io/ the building blocks are here. It's just a matter of time before someone puts the existing pieces together to make said robot, the only question is who will be first to make it, who will be first to open source it. Who will make it not just possible, but affordable?

Thanks! Appreciate the kind words. I should have in the next month or so (interviewing and finishing my Master's, so there's been delays) a follow up that follows more advancements in the router style VLA, sensoiromotor VLM, and advances in embedding enriched vision models in general.

If you want a great overview of what a modern robotics stack would look like with all this, https://ok-robot.github.io/ was really good and will likely make it into the article. It's a VLA combined with existing RL methods to demonstrate multi-tasking robots, and serves as a great glimpes into what a lot of researchers are working on. You won't see these techniques in robots in industrial or commercial settings - we're still too new at this to be reliable or capable enough to deploy these on real tasks.

The Ok-robot demo shows that the technology for it to be fairly general is there, though no idea if figure one is using their technology or not. Simply being able to command a robot instead of moving a turtle with gcode is nothing short of astounding to those who aren’t deeply involved and tracking the sota progress in this area.

https://ok-robot.github.io/

Disclaimer: I'm not one of the authors, but I work in this area.

You basically hit the nail on the head with these questions. This work is super cool, but you named a lot of the limitations with contemporary robot learning systems.

1. It's using an object classifier. It's described here (https://github.com/ok-robot/ok-robot/tree/main/ok-robot-navi...), but if I understanding it correctly basically they are using a ViT model (basically an image classification model) to do some labeling of images and projecting them onto a voxel grid. Then they are using language embeddings from CLIP to pair the language with the voxel grid. The limitations of this are that if they want this to run on the robot, they can't use the super huge versions of these models. While they could use a huge model on the cloud, that would introduce a lot of latency.