Marginal tool utility in agentic debugging

TL;DR

Giving an AI agent more tools does not always help. In our test, removing distracting tools kept its answers just as accurate and made its tool use much more efficient.

What we measured

An agent can call many tools and still fail to make progress. Looking only at the final answer hides which calls helped and which calls added noise, so we use two measurements:

Marginal tool utility asks whether one tool call made the agent more likely to solve the task. A call has positive utility when it helps, zero utility when it changes nothing, and negative utility when it moves the agent away from the solution.
Tool efficiency is the share of all tool calls in a trajectory that had positive utility. A higher score means the agent spent more of its calls making useful progress.

The experiment

We evaluated these measurements on APEX-SWE Observability, where agents debug software problems with read-only MCP tools. We compared the default harness, which included Grafana/Loki, Mattermost, and Plane, with a smaller harness that kept only Grafana/Loki.

This comparison tests whether every available tool improves the debugging process. Mattermost and Plane can contain useful context, but they can also give an agent more places to search without bringing it closer to the root cause.

Results

Removing the noisy tools preserved task accuracy while increasing tool efficiency.

For GPT-5.3-Codex, mean tool efficiency moved from 0.359 with the default suite to 0.720 with Grafana/Loki only. For Gemini 3.1 Pro, it moved from 0.367 to 0.593.

Definitions

Marginal Tool Utility

Does a specific tool call increase the agent’s probability of solving the task? Positive, negative, or zero per call.

Tool Efficiency

The share of tool calls in a trajectory that carried positive marginal utility. A number worth optimizing.

Tool efficiency before and after removing noisy tools

GPT-5.3-CODEX

Default suite (Grafana + Mattermost + Plane)0.359

Grafana / Loki only0.720

+100% efficiencyaccuracy preserved

GEMINI 3.1 PRO

Default suite (Grafana + Mattermost + Plane)0.367

Grafana / Loki only0.593

+62% efficiencyaccuracy preserved

MTU signs by tool — GPT-5.3-Codex · default APEX-SWE trajectories

PositiveNegativeZero

Grafana / Loki

+62% / −18% / 0:20%

Mattermost

+22% / −48% / 0:30%

Plane

+18% / −55% / 0:27%

Mattermost and Plane produce predominantly negative MTU — they make the agent less likely to solve the task.

The result is not that agents should always receive fewer tools. It is that tool suites deserve measurement. Some tools add signal. Some add noise. Marginal tool utility gives us a way to separate the two.

For GPT-5.3-Codex, this means the useful share of tool calls roughly doubled. Gemini 3.1 Pro also made substantially more useful calls with the smaller tool suite.

What this means

The lesson is not that agents should always receive fewer tools. The lesson is that each tool should earn its place. Some tools add signal, while others create extra paths that the agent must explore.

Marginal tool utility shows which calls help, hurt, or do nothing. Tool efficiency summarizes that behavior across a full debugging trajectory, giving teams a concrete number they can use when designing and comparing agent tool suites.