Show HN: Speeding up LLM inference 2x times (possibly)

• effort

  2 181 9.7 Swift

  An implementation of bucketMul LLM inference

I think it was somewhere around that tag:

https://github.com/kolinko/effort/releases/tag/5.0-last-mixt...

Cannot rerun easily any more, because the underlying model/weight names changed in the meantime. It doesn't help that Mixtral's published .safetensor files seem messed up, and I needed to hack a conversion from pytorch - it added an extra layer of confusion into the project.

• cria

  3 77 2.5 Python

  Tiny inference-only implementation of LLaMA (by recmo)

It originally started as a fork to Recmo’s cria pure numpy llama impl :)

https://github.com/recmo/cria

Took a whole night to compute a few

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project