I decided to actually test the claim I made about local models

A claim I hadn’t actually proven

When I introduced Polyglot, the whole premise rested on one claim: open-weight models are unreliable at emitting tool calls through the “native” function-calling channel, so an agent that instead parses tool calls out of the model’s text - and repairs the broken ones - will get more real work done on the model you’re actually running at home.

I believed that claim. I’d watched it happen on my own machine enough times to build a whole tool around it. But “I’ve seen this happen” and “this is actually, measurably true, more than it is for other agents” are different things, and I hadn’t done the second one yet.

So I built a small benchmark, pointed three agents - Polyglot, pi, and Nous Research’s Hermes Agent - at the same local models through the same Ollama endpoint, and ran the same six coding tasks through each of them, three times over, scored automatically rather than eyeballed.

What came back

The short version: on qwen2.5-coder, a genuinely capable and popular local coding model, both pi and Hermes Agent made zero tool calls, on every single run, on both the 7B and the 14B. Not because the tasks were too hard - because the model kept putting the tool call in the wrong channel, and neither agent was looking there. Polyglot recognized the shape and ran it anyway: 39% of tasks completed on the 7B, climbing to around 80% on the 14B.

That gap was bigger and more consistent than I expected going in. I’d guessed Polyglot would have an edge on the smallest, shakiest models - not that two otherwise well-built agents would come back with a flat zero on a model as capable as qwen2.5-coder:14b.

The Hermes result also taught me something I got wrong. Their docs mention “11 tool-call parsers,” which made me expect a strong competitor here. Reading their source, those parsers live at the model-training layer, not in the live agent loop - and the loop actually strips text-formatted tool calls out as noise before they’d ever be acted on. It’s a deliberate, defensible choice for a team building around their own model family. It’s just the opposite of the bet Polyglot makes.

Why I’m writing this up separately

I’m keeping the harness and the raw transcripts private for now - it’s a rougher tool than Polyglot itself, and not something I want to maintain as a public artifact yet. But I didn’t want the numbers to just live in a private repo either, so I wrote up the full methodology, the tables, and the caveats I’d want called out if someone else ran this - on the Polyglot blog.

If you run a coding agent against a local model and it seems to work fine, it might be. But if it’s ever quietly “finished” a task without touching a single file, this is probably why.

npm install -g @usepolyglot/cli