[P] 4.5 times faster Hugging Face transformer inference by modifying some Python AST

• transformer-deploy

  8 1,614 6.8 Python

  Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀

  

Regarding CPU inference, quantization is very easy, and supported by Transformer-deploy , however performance on transformer are very low outside corner cases (like no batch, very short sequence and distilled model), and last Intel generation CPU based instance like C6 or M6 on AWS are quite expensive compared to a cheap GPU like Nvidia T4, to say it otherwise, on transformer, until you are ok with slow inference and takes a small instance (for a PoC for instance), CPU inference is probably not a good idea.

• sparsednn

  1 51 0.0 Python

  Fast sparse deep learning on CPUs

If you are ever interested in looking at pruning, happy to integrate my open source library https://github.com/marsupialtail/sparsednn. Latest update has unstructured and structured sparse int8 kernels. 3x speedup over dense int8 at 90 percent sparsity with 1x4 blocks.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• TensorRT

  5 2,319 9.6 Python

  PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT (by pytorch)

  

Have you tried the new Torch-TensorRT compiler from NVIDIA?

• mmrazor

  4 1,352 2.8 Python

  OpenMMLab Model Compression Toolbox and Benchmark.

  

https://github.com/open-mmlab/mmrazor ,it may work for you~

Suggest a related project