3code vs pi vs opencode vs hermes vs zcode: 10-task SWE-bench Verified on GLM-5.3

Detailed data for the 10-task subset, second round. All five agents routed through the same LiteLLM proxy to Z.ai glm-5.3, so every difference reflects the agent harness, not the model. Per-task timeout 600s. All agents ran with vanilla config.

Scorecard

Zero eval errors. “empty (timeout)” means the 600s cap was hit without a diff; “unresolved” means a patch was produced but the task tests failed.

Noise floor notes: psf__requests-1766 was resolved by all agents in the GLM-5.2 round and went unresolved for four of five here, so that reads as a model-level regression, not a harness difference. Z.ai’s rate limiter (429s) interrupted several tasks mid-run; affected tasks were retried after a cooldown and the table shows the final state.

Batch totals (all 10 tasks)

Same table as a percentage of the median agent (zcode):

3code used 38% less input than opencode and 9% less than pi, while resolving one more task. Output tokens are lowest for 3code as well (terse system prompt and reply style are deliberate design).

Caveat on input tokens: due to a LiteLLM proxy configuration error, cache hit rates were not recorded this round (Z.ai’s implicit caching was almost certainly active, but the proxy dropped the cached_tokens field from streamed responses). Input totals here include tokens that were actually served from cache and billed at a discount; reported input is therefore an upper bound on billed cost. Cache reporting is fixed for future tests. Token windows are reconstructed from per-task timing; pi’s django-15368 and scikit-learn numbers include overlapping retry attempts, so pi’s total is an upper bound.

Per-task detail

sympy__sympy-23534 (easy)

pydata__xarray-6461 (easy)

django__django-15368 (easy)

psf__requests-1766 (easy)

matplotlib__matplotlib-26342 (medium)

django__django-15037 (medium)

scikit-learn__scikit-learn-14087 (medium)

pytest-dev__pytest-7236 (medium)

sphinx-doc__sphinx-8056 (medium)

astropy__astropy-14369 (hard)

Observations

• 3code resolves 7 vs 6 for the others, the difference being matplotlib-26342, which pi, opencode, and hermes all lost to the 600s timeout while running local builds. Same pattern as the GLM-5.2 round (opencode timed out there too). zcode also resolved matplotlib-26342, but gave the win back with timeouts on pytest-7236 and an unresolved patch on scikit-learn-14087.
• hermes is the only agent that resolved psf__requests-1766 this round (unresolved by the other four), but it paid with timeouts on pytest-7236, which three of the other four resolved.
• The three tasks nobody solved (scikit-learn, pytest got solved this time, astropy) still involve slow local build/test cycles that eat the whole budget.
• pi sits between 3code and opencode on token use, closer to 3code on input but with the chattiest output of the five (118K output tokens, 1.6x 3code’s).
• zcode lands in the same band as pi (4.77M total, 103% of pi). Its signature is bimodal: lean runs on the tasks it solves, then 1.2-1.3M input tokens on the two it timed out on (10x its own easy-task usage), where it keeps retrying until the cap. The exception is astropy-14369: every other agent burned 0.9-1.6M tokens before timing out there, zcode stalled at 123K input tokens, it never got going at all.
• hermes lands between pi and opencode on input (5.97M) with mid-range output. Its runs were the most disrupted by Z.ai’s rate limiting: its tool-call pattern (small frequent turns, plus a title-generation call every session start) interacts badly with the requests-per-minute cap, and three tasks needed overnight retries.
• opencode’s input profile is again the outlier: 1.6x 3code’s input, driven by higher tool-call volume and larger context re-sent per turn. sphinx-8056 is the extreme case: 1.64M input tokens for a patch 3code wrote with 423K and zcode with 406K.
• GLM-5.3 vs 5.2 on the same harness: psf__requests-1766 flipped from resolved-by-all to unresolved-by-all, and 3code went 6/10 to 7/10 (matplotlib, which 5.2-3code also solved; pytest flipped too). Small sample, treat as directional only.

The agents that didn’t run

• Claude Code speaks only the Anthropic Messages protocol. Z.ai’s Anthropic endpoint (/api/anthropic) is gated on the GLM Coding Plan, which is expired on this account; the pay-as-you-go paas endpoint (used by everything else here) works fine. LiteLLM can translate Anthropic Messages to the OpenAI paas endpoint in-process, but its anthropic adapter rewrites usage into Anthropic’s shape, breaking the raw pass-through counting this benchmark relies on. Re-enabling that path needs a usage-forwarding fix in the proxy logger first.
• ZCode’s desktop app could not execute in the benchmark shell (its Electron binary requires a root SUID helper and offers no headless mode reachable from the harness), but its standalone npm CLI (zcode-app-cli) is a full headless agent and ran in its place. Results above.

Methodology

Same as the GLM-5.2 round: SWE-bench Verified harness for evaluation, patches extracted as git diff HEAD from a fresh clone at the task’s base commit, per-task 600s timeout, agents run once per task with retries only for rate-limit (429) failures. Token counts come from the LiteLLM proxy’s usage log, which captures Z.ai’s usage object verbatim per request (pass-through, no recomputation) - except for the cached-token field this round, see the caveat above. GLM-5.3 requires thinking enabled; the proxy normalizes this for vanilla clients by injecting thinking: {type: enabled} (and translating reasoning_effort for OpenAI-protocol clients) before relaying, at Z.ai’s default effort.

Changes since the GLM-5.2 round: 3code grew a --no-sandbox flag (used here; the harness already isolates in a throwaway clone) and its system prompt now points multi-step work at the cybernetic-plan skill. The harness grew a --subset10 flag and pi, hermes, and zcode agent entries. zcode runs through the proxy as an openai-compatible provider against an isolated HOME config, keyed with ZCODE_API_KEY.