CodeGen VS FasterTransformer

CodeGen (I included this model as there's some evidence in recent months that LLMs trained on code-related tasks better capture long-range dependencies compared to LLMs trained on 'just words')

https://github.com/salesforce/CodeGen these are used in fauxpilot https://github.com/fauxpilot/fauxpilot

You can always use https://github.com/salesforce/CodeGen . But it does require managing the model hosting. You can use fauxpilot to mimic copilot functionality https://github.com/fauxpilot/fauxpilot

freediver 2 days ago | prev | next [–] Here is an example of one general purpose open source LLM, probably the best you can get: https://github.com/EleutherAI/gpt-neox To manage your expectations it is nowhere as good as ChatGPT. If you are interested in programming only: https://github.com/salesforce/CodeGen

Replit built Ghostwriter on the open source scene based on Salesforce’s Codegen, using Nvidia’s FasterTransformer and Triton server for highly optimized decoders, and the knowledge distillation process of the CodeGen model from two billion parameters to a faster model of one billion parameters.

https://github.com/moyix/fauxpilot is working on an open source variant. Based on https://github.com/salesforce/CodeGen

https://docs.nvidia.com/ai-enterprise/overview/0.1.0/platfor...

RIVA: NVIDIA® Riva, a premium edition of NVIDIA AI Enterprise software, is a GPU-accelerated speech and translation AI SDK

FasterTransformer: https://github.com/NVIDIA/FasterTransformer an

There could be some spectrum of this expertise. For instance, https://github.com/NVIDIA/FasterTransformer, https://github.com/microsoft/DeepSpeed

Have anyone tried to optimize the forward and backward using custom Cuda code or fused kernel to speed up the training time of current LLMs? I only have seen FasterTransformer ( NVIDIA/FasterTransformer) and other similar tools but they're only focusing on inference.

Replit built Ghostwriter on the open source scene based on Salesforce’s Codegen, using Nvidia’s FasterTransformer and Triton server for highly optimized decoders, and the knowledge distillation process of the CodeGen model from two billion parameters to a faster model of one billion parameters.

Nvidia FasterTransformer is a mix of Pytorch and CUDA/C++ dedicated code. The performance boost is huge on T5, they report a 10X speedup like TensorRT. However, the speedup is computed on a translation task where sequences are 25 tokens long on average. In our experience, improvement on very short sequences tend to decrease by large margins on longer ones. Still we plan to dig deeper into this project as it implements very interesting ideas.

On the other side of the spectrum, there is Nvidia demos (here or there) showing us how to build manually a full Transformer graph (operator by operator) in TensorRT to get best performance from their hardware. It’s out of reach for many NLP practitioners and it’s time consuming to debug/maintain/adapt to a slightly different architecture (I tried). Plus, there is a secret: the very optimized model only works for specific sequence lengths and batch sizes. Truth is that, so far (and it will improve soon), it’s mainly for MLPerf benchmark (the one used to compare DL hardware), marketing content, and very specialized engineers.