GLM-5.3 agent rerun: 6 agents incl Claude Code
GLM-5.3 agent rerun — full report (2026-09-05/06)
10-task SWE-bench Verified subset, per-task 600s cap, all agents vanilla config, same box, health-gated (quiet host verified before each agent). One model for everyone: glm-5.3. Claude Code connects direct to the Z.ai coding plan; the other five route through the LiteLLM counting proxy (now also on the coding-plan endpoint).
Final scorecard
| rank | agent | resolved | notes |
|---|---|---|---|
| 1 | 3code | 9/10 | best ever; astropy solved; sklearn timeout (2026-09-05 run, pre-contamination window, kept) |
| 1 | opencode | 9/10 | clean rerun 2026-09-06 11:57 |
| 3 | claude code | 8/10 | direct coding-plan; astropy failed |
| 4 | hermes | 7/10 | rerun with HERMES_HOME fix; astropy+matplotlib timeouts |
| 5 | zcode | 6/10 | sympy eval-infra error regraded RESOLVED (eval-only rerun) |
| 5 | pi | 6/10 | sympy eval-infra error regraded RESOLVED (eval-only rerun) |
Per-task grid
R = resolved, F = failed patch, T = timeout (no diff), E = eval-infra error (since regraded)
| task | 3code | opencode | claude | hermes | zcode | pi |
|---|---|---|---|---|---|---|
| sympy__sympy-23534 | R | R | R | R | E→R | E→R |
| pydata__xarray-6461 | R | R | R | R | R | R |
| django__django-15368 | R | R | R | R | F | R |
| psf__requests-1766 | R | R | R | R | R | R |
| matplotlib__matplotlib-26342 | R | R | R | T | F | R |
| django__django-15037 | R | R | R | R | R | R |
| scikit-learn__scikit-learn-14087 | T | F | F | T | F | F |
| pytest-dev__pytest-7236 | R | R | R | R | R | T |
| sphinx-doc__sphinx-8056 | R | R | R | R | R | T |
| astropy__astropy-14369 | R | R | F | T | T | T |
Only 3code and opencode solved astropy. sklearn-14087 fell to everyone except 3code — the sole 3code miss.
Token accounting (corrected)
All numbers are true prompt traffic (cache-inclusive) +
output, corrected for the litellm 1.97.0 stream bug (see
“Incidents” below). Corrected source:
tokens-proxy-glm-5.3-CORRECTED.tsv.
| agent | prompt total | cached (hit) | output | resolved |
|---|---|---|---|---|
| 3code | 4.98M | 4.77M (95.7%) | 80k | 9/10 |
| pi | 6.91M | 6.40M (92.7%) | 320k | 6/10 |
| opencode | ≥9.85M | 9.78M (≥99.2%) | 132k | 9/10 |
| zcode | ≥13.84M | 13.53M (≥97.8%) | 339k | 6/10 |
| hermes | ≥16.57M | 16.01M (≥96.6%) | 216k | 7/10 |
| claude code | 22.88M | 22.25M (97.2%) | 236k | 8/10 |
- zcode/hermes/opencode rows are conservative lower bounds: their per-call tool-block size could not be recovered exactly from the polluted logs (pre-fix). 3code and pi are exact.
- Claude Code talks the Anthropic protocol: it sends only new tokens and the server holds the prefix (97% of its prompt traffic is cache reads). The OpenAI-protocol agents resend the full prefix each call and rely on Z.ai implicit caching (93–99% hit — verified, see below).
- Cost reading: with every agent ≥93% cached, spend tracks prompt volume: claude processes 4.6x 3code’s context to score one task less.
Pi: did it really do that poorly?
Yes — with one asterisk. Pi’s sympy patch (13 lines) was graded RESOLVED by the eval-only rerun, so the original 5/10 report was really 6/10; it’s now folded in. The remaining misses are real: sklearn patch failed its tests (0/3 FAIL_TO_PASS, 172/172 PASS_TO_PASS — a genuinely wrong patch), and astropy/pytest/sphinx all hit the 600s timeout with no diff. Pi is also the second-noisiest model caller (320k output tokens, 4x 3code).
Incidents found & fixed during this rerun
- litellm 1.97.0 stream usage bug (root cause of the
bogus “51% cache hit”): the provider’s final stream chunk arrives as the
OpenAI SDK type
CompletionUsage; litellm’s usage assembly probes it with dict-style"prompt_tokens" in chunk, which is False for pydantic objects, so it re-counted prompts with its own tokenizer over messages only — excluding the tools block (~0.6–1.1k tokens/call). Cached tokens were unaffected. Fixed inusage_logger.py(attr-style restore + provider-usage stash preferred in the log writer); verified wire-identical (direct vs proxy: 1459/1408). - Docker Hub TLS flakes produced eval-infra errors on sympy-23534 (zcode, pi, claude-flash). All regraded via eval-only reruns. All 10 eval images are now pre-pulled and cached, so evals no longer hit the registry.
- run-lineup.sh resume-guard hole: zcode reports
don’t encode the model (lineage
default), so the guard could rerun valid cells. Fixed via predictions-lineage check + explicit-mfor zcode. - Invalid artifacts archived (never deleted) in
archive/: the dead-PAAS-window hermes/opencode glm-5.3 runs and all five junk glm-5.3-flash proxy-agent reports.
Provenance
- 3code glm-5.3: 2026-09-05 01:56–02:44 (kept, pre-contamination).
- pi 01:23–02:16, zcode 02:16–03:29, claude 04:07–05:00 (2026-09-06).
- hermes 10:16–11:12, opencode 11:13–11:57 (2026-09-06, coding-plan endpoint, post health-gate).
- Eval reports in repo root; eval-only regrades archived in
archive/. - Token TSVs:
tokens-proxy-glm-5.3-CORRECTED.tsv(proxy agents),tokens-claude-glm-5.3-20260906.tsv(claude, raw transcripts).
Still open
- glm-5.3-flash: only claude 6/10 valid; the other five cells await a rerun (now with correct usage logging from the start).
- k3: no valid cells yet.
- Both parked pending explicit go.
The poster
Methodology
SWE-bench Verified, 10-task subset (3 easy / 6 medium / 1 hard),
patches extracted as git diff HEAD from a fresh clone at
the task’s base commit, per-task 600s cap, one run per task per agent,
vanilla configs. Claude Code 2.1.239 connects direct to the Z.ai coding
plan (Anthropic protocol); the other five route through the local
LiteLLM counting proxy to the same endpoint. Evals run in the official
swebench docker harness; three eval-infra failures (Docker Hub TLS
timeouts while pulling the sympy image) were regraded with eval-only
reruns. Token numbers are corrected for a litellm 1.97.0 stream-usage
bug that undercounted prompt tokens by the tools block (~0.6-1.1k/call)
on streamed calls; cached-token fields were unaffected. zcode / hermes /
opencode prompt totals are conservative lower bounds (their pre-fix tool
blocks were not exactly recoverable); 3code, pi and claude are
exact.
Data
- Per-task corrected token rows: tokens-proxy-glm-5.3-CORRECTED.tsv
- Claude Code raw token rows (transcript-derived):
tokens-claude-glm-5.3-20260906.tsvin the harness repo - Eval reports + archived eval-only regrades:
~/p/3code-swe(harness repo, commita4453b1)