I decided to actually test the claim I made about local models

A claim I hadn’t actually proven

When I introduced Polyglot, the whole premise rested on one claim: open-weight models are unreliable at emitting tool calls through the “native” function-calling channel, so an agent that instead parses tool calls out of the model’s text - and repairs the broken ones - will get more real work done on the model you’re actually running at home.

I believed that claim. I’d watched it happen on my own machine enough times to build a whole tool around it. But “I’ve seen this happen” and “this is actually, measurably true, more than it is for other agents” are different things, and I hadn’t done the second one yet.

So I built a small benchmark, pointed four agents - Polyglot, pi, Nous Research’s Hermes Agent, and Goose (the Linux Foundation’s Agentic AI Foundation project, by far the biggest name of the bunch) - at the same local models through the same Ollama endpoint, and ran the same six coding tasks through each of them, three times over, scored automatically rather than eyeballed.

What came back

The short version: on qwen2.5-coder, a genuinely capable and popular local coding model, pi, Hermes Agent, and Goose all made zero tool calls, on every single run, on both the 7B and the 14B. Not because the tasks were too hard - because the model kept putting the tool call in the wrong channel, and none of them were looking there. Polyglot recognized the shape and ran it anyway: 39% of tasks completed on the 7B, climbing to around 80% on the 14B.

That gap was bigger and more consistent than I expected going in. I’d guessed Polyglot would have an edge on the smallest, shakiest models - not that three otherwise well-built agents would come back with a flat zero on a model as capable as qwen2.5-coder:14b.

The Hermes result also taught me something I got wrong. Their docs mention “11 tool-call parsers,” which made me expect a strong competitor here. Reading their source, those parsers live at the model-training layer, not in the live agent loop - and the loop actually strips text-formatted tool calls out as noise before they’d ever be acted on. It’s a deliberate, defensible choice for a team building around their own model family. It’s just the opposite of the bet Polyglot makes.

Goose is the one I was most nervous about going in, and I’m still a little surprised by the result. It’s backed by the Linux Foundation’s Agentic AI Foundation - AWS, Anthropic, Block, Bloomberg, Cloudflare, Google, and Microsoft are all platinum members - and unlike pi or Hermes, it actually ships a documented mechanism aimed at this exact problem: a “toolshim” that routes a non-tool-calling model’s output through a second model to reinterpret it as a proper call. I turned it on and reran everything. It clearly did something - some runs made 20 tool-call attempts instead of one - but task completion barely moved: 0/18 on the 7B, 1/18 on the 14B. A foundation-scale project with a purpose-built fix for this exact failure mode still didn’t close the gap, at least not in my testing.

Why I’m writing this up separately

I’m keeping the harness and the raw transcripts private for now - it’s a rougher tool than Polyglot itself, and not something I want to maintain as a public artifact yet. But I didn’t want the numbers to just live in a private repo either, so I wrote up the full methodology, the tables, and the caveats I’d want called out if someone else ran this - on the Polyglot blog.

If you run a coding agent against a local model and it seems to work fine, it might be. But if it’s ever quietly “finished” a task without touching a single file, this is probably why.

npm install -g @usepolyglot/cli