StreamingLLM: Efficient streaming technique enable infinite sequence lengths

• streaming-llm

  11 6,206 7.2 Python

  [ICLR 2024] Efficient Streaming Language Models with Attention Sinks

  

• CTranslate2

  14 2,825 8.9 C++

  Fast inference engine for Transformer models

  

Etc.

Now, what this allows you to do is reuse the attention computed from the previous turns (since the prefix is the same).

In practice, people often have a system prompt before the conversation history, which (as far a I can tell) makes this technique not applicable (the input prefix will change as soon as the conversation history is long enough that we need to start dropping the oldest turns).

In such case, what you could do is to cache at least the system prompt. This is also possible with https://github.com/OpenNMT/CTranslate2/blob/2203ad5c8baf878a...

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project