[D] Why are decoder only models used for autoregressive generation instead of encoder-only models? What value is the causal mask if the new token doesn't exist yet?

• streaming-llm

  11 6,285 7.2 Python

  [ICLR 2024] Efficient Streaming Language Models with Attention Sinks

  

That is a great question. I wish I had a mathematical explanation for it, but I can only provide some intuitive "yeah but then again"... Fwiw, there was a paper recently that indeed showed that the first few tokens of any sequence, starting with the special '[START]' token does hold special information (they call it the Attention Sink) compared to all other tokens. Here is a link to that paper:https://arxiv.org/abs/2309.17453

• Scout Monitoring

  www.scoutapm.com featured

  Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

  

Suggest a related project