kcores-llm-arena VS chat.md

Gemini 2.5 pro is far ahead of even Claude

Chart:

https://raw.githubusercontent.com/KCORES/kcores-llm-arena/re...

Description of the challenges:

https://github.com/KCORES/kcores-llm-arena

FWIW, in limited trials so far, agentic capabilities feel worse than gpt-4.1 with various tool calling error modes not seen in gpt-4.1 [4].

Attempt 1: couldn't understand correct syntax, aborted. [1]

Attempt 2: problems in understanding tool result and passing correct inputs, aborted [2]

Attempt 3: waits multiple times for user confirmation, shows lack of native agentic looping unlike sonnet. Finally has same failure as 2 [3]

Sonnet 3.7 obviously has least number of such errors, followed by gemini 2.5 pro [5]

[1] https://github.com/rusiaaman/chat.md/blob/main/samples/o4-mi...

[2] https://github.com/rusiaaman/chat.md/blob/main/samples/o4-mi...

[3] https://github.com/rusiaaman/chat.md/blob/main/samples/o4-mi...

[4] https://github.com/rusiaaman/chat.md/blob/main/samples/4.1/t...

[5] https://github.com/rusiaaman/chat.md/blob/main/samples/gemin...

Did some quick tests. I believe its the same model as Quasar. It struggles with agentic loop [1]. You'd have to force it to do tool calls.

Tool use ability feels ability better than gemini-2.5-pro-exp [2] which struggles with JSON schema understanding sometimes.

Llama 4 has suprising agentic capabilities, better than both of them [3] but isn't as intelligent as the others.

[1] https://github.com/rusiaaman/chat.md/blob/main/samples/4.1/t...