rllama VS ultraviolet

Location: San Francisco

Remote: No preference, as long as I don't have to move far from Bay Area

Willing to relocate: No

Technologies: C, Rust, Golang, Haskell, Lisp, Python, Lua, OpenGL, SQLite3, JavaScript, PostgreSQL, AWS EC2, S3, ECS, Batch.

Resume: https://www.linkedin.com/in/mikjuola

Email: [email protected]

---

I've been working at the Bay Area since 2015, most recently at Pinterest. At work, I've done big data pipelines, designed some batch job systems, computing metrics, handling billing APIs, lots of Python, Go and Java and working with AWS, i.e. backend and data engineer stuff.

But I'm trying to look for work that's more in line with what I do on my free time: Challenging low-level C or Rust programming, machine learning implementations (see e.g. this thing I made https://github.com/Noeda/rllama/, graphics programming or research-type work, uncommon programming languages.

If you scroll through my random crap repositories you can see what kind of things I'm interested in: https://github.com/Noeda?tab=repositories

No, my project is called rllama. No relation to GGML. https://github.com/Noeda/rllama

I have seen some Rust attempts as well such as https://github.com/Noeda/rllama/ but they are still way behind the C++ ones. This seems like an interesting space to get into.

I ran it on a 128 RAM machine with a Ryzen 5950X. It's not fast, 4 seconds per token. But it's just about fits without swapping. https://github.com/Noeda/rllama/

I've counted three different Rust LLaMA implementations on r/rust subreddit this week:

https://github.com/Noeda/rllama/ (pure Rust+OpenCL)

https://github.com/setzer22/llama-rs/ (ggml based)

https://github.com/philpax/ggllama (also ggml based)

There's also a discussion on GitHub issue on setzer's repo to collaborate a bit on these separate efforts: https://github.com/setzer22/llama-rs/issues/4

This one has some more traction: https://github.com/fu5ha/ultraviolet

Some other good places to look would be the source code for the ultraviolet crate and for wide.

2. Using macros for codegen, as seen in almost all other linear algebra crates, e.g. ultraviolet (https://crates.io/crates/ultraviolet)

The drawback of (1) is that it's hard to understand or extend, and the type errors are insane, and put Boost to shame. The drawback of (2) is that's just a macro. Of course (1) and (2) are not interoperable.

C++ templates are really underappreciated, because they are a single mechanism that gives you both powerful generics and macro-like codegen, while being easy to adopt gradually. (E.g. you can add concepts when it gets unwieldy, but you don't have to rewrite the whole thing.)