nnl

a low-latency and high-performance inference engine for large models on low-memory GPU platform. (by fengwang)
Add to my DEV experience Cuda Deep Learning inference-engine language-model
Source Code
Suggest alternative
Edit details
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured

Nnl Alternatives

Similar projects and alternatives to nnl based on common topics and language

  • petals

    98 nnl VS petals

    🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

  • FlexGen

    39 nnl VS FlexGen

    Running large language models on a single GPU for throughput-oriented scenarios.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • cortex

    8 nnl VS cortex

    Drop-in, local AI alternative to the OpenAI stack. Multi-engine (llama.cpp, TensorRT-LLM). Powers 👋 Jan (by janhq)

  • Daisykit

    1 nnl VS Daisykit

    Daisykit is an easy AI toolkit with face mask detection, pose detection, background matting, barcode detection, and more. With Daisykit, you don't need AI knowledge to build AI software.

  • rwkv.cpp

    12 nnl VS rwkv.cpp

    INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model

  • wyGPT

    5 nnl VS wyGPT

    Discontinued Wang Yi's C++ GPT Character Language Model

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better nnl alternative or higher similarity.
Suggest an alternative to nnl

nnl reviews and mentions

Posts with mentions or reviews of nnl. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2023-12-03.
  • Run 70B LLM Inference on a Single 4GB GPU with This New Technique
    3 projects | news.ycombinator.com | 3 Dec 2023
    I did roughly the same thing in one of my hobby project https://github.com/fengwang/nnl. But in stead of using SSD, I load all the weights to the host memory, and while inferencing the model layer by layer, I asynchronously copy memory from global to shared memory in the hope of better performance. However, my approach is bounded by the PCI-E bandwidth.

Stats

Basic nnl repo stats
1
4
5.4
5 months ago

The primary programming language of nnl is C++.

Popular Comparisons

  1. nnl VS cortex

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com