llama2.f90 VS ferrite

I'm mostly a python programmer, but I find a lot of the ML frameworks are overkill for what they actually do, especially for inference. Fortran is pretty close to numpy - it handles arrays natively including slicing and matmul instinsics, you don't have to worry about memory etc. But it compiles into something fast and lightweight much more easily than python. It's nothing you couldn't do in C but I think Fortran is better suited for linear algebra.

See also https://github.com/rbitr/llama2.f90 which is basically the same thing but for running llama models and has 16-bit and 4-bit options and a lot more optimization.

I clicked on this because of Fortran and it had nothing to do with it. It was more an advertisement for Mojo.

If anyone is interested in Fortran for AI, I am working on a Fortran LLM project: https://github.com/rbitr/llama2.f90

This is interesting, I'd like to see how fast it runs. A challenge even with fast implementations of models is there is always some overhead starting up.

I also want to mention this is exactly the kind of use case I had in mind for Ferrite, a dependency free Fortran implementation of sentence transformers: https://github.com/rbitr/ferrite The idea is a very simple script to run transformer inference on a CPU that can be incorporated into a project without pulling in a million dependencies.

I am happy to see though that this is a C++ project and appears not to require pytorch and HF transformers to use!

I'll plug the project I've started work on along the same lines - a simplified CPU focused embedding model (right now with distillbert) that's coded as a single file with no dependencies and no abstraction. https://github.com/rbitr/ferrite