[D] In decoder models, if later tokens attend to early tokens but early tokens don't attend to later tokens, what stops the influence of the early tokens from growing with each layer?

• streaming-llm

  11 6,285 7.2 Python

  [ICLR 2024] Efficient Streaming Language Models with Attention Sinks

  

Just quickly glanced through the question but you might be interested in the attention sink for example where they use the fact that earlier tokens are overly attended to in general --> paper: https://arxiv.org/abs/2309.17453

• Scout Monitoring

  www.scoutapm.com featured

  Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

  

Suggest a related project