NoLiMa VS Elemental

Thanks for the answer, sir. OK, yes. That makes a lot more sense. I am context greedy ever since I read that Adobe research paper that I shared with you months ago. [0]

The whole "context engineering" seems like a thing, though I dislike throwing around the word "engineer" all willy-nilly like that. :)

In any case, thanks for the response. I just wanted to make sure I was not missing something.

[0] https://github.com/adobe-research/NoLiMa

Updated results from the authors: https://github.com/adobe-research/NoLiMa

It's the best known performer on this benchmark, but still falls off even relatively modest context lengths. (Cutting edge reasoning models like Gemini 2.5 Pro haven't been evaluated due to their cost and might be better.)

I only ran the benchmark on Quasar Alpha*; the rest of the scores come from the original paper [0] which was published before 3.7 was available. This is a pretty expensive benchmark to run if you're paying for API usage - I'd actually originally set out to run it on Llama 4 but abandoned that after estimating the cost.

* - I also reproduced the Llama 3.1 8B result to check my setup.

[0] - https://arxiv.org/abs/2502.05167 / https://github.com/adobe-research/NoLiMa*

They are testing for a very straightforward needle retrieval, as LLMs traditionally were terrible for this in longer contexts.

There are some more advanced tests where it's far less impressive. Just a couple of days ago Adobe released one such test- https://github.com/adobe-research/NoLiMa

I know many people who can and will one-shot a rewrite of 500 LOC. In my world, 500 LOC is about the length of a single function. I don't understand why we should be talking about generating a high level plan with multiple tests etc. for a single function.

And I don't think this is uncommon. Just a random example from Github, this file is 1800 LOC and 4 functions. It implements one very specific thing that's part of a broader library. (I have no affiliation with this code.)

https://github.com/elemental/Elemental/blob/master/src/optim...