Applying All Recent Innovations To Train a Code Model

• lion-pytorch

  3 1,918 3.8 Python

  🦁 Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam(w), in Pytorch

Various people are trying LiON on their projects, with varying degrees of success. A good starting point to look around is the lion-pytorch on github from Phil Wang aka lucidrains (thank you man!).

• llm-foundry

  37 3,710 9.7 Python

  LLM training code for Databricks foundation models

  

MosaicML released the MPT-7B model, which has a context of 60k tokens, thanks to the ALiBi position encoding.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project