Routing

Status: 200-trial sweep complete (8 routes × N=1..5 × 5 reps, 197/200 returned HTTP 200, 3 legitimate 300s agent timeouts on direct-minimax-m2.7 at high N). Bug-fixed dispatcher; full data at results/route-comparison.jsonl (gitignored).

TL;DR

Skynet is faster at N=1 on 3 of 4 model pairs. The biggest win is glm-5.1 (+26.5% speedup, 55.9s → 41.1s). minimax-m3 is statistically tied (within ±1% and ±10s σ). The ZAI GLM models see the largest proxy advantage. The Minimax-M3 pair is essentially noise.

Why this page exists

The original Phase 2 provider comparison showed MiniMax-M3 (direct) with N=1 wall = 52.0s vs minimax-m3 (Skynet) at 29.3s — a 44% proxy speedup for the same model. That was a single-cell observation in a wider sweep, and it deserved its own test:

1. Is the Skynet speedup reproducible? Same VM, same task, same prompt, same time-of-day — only the (base_url, model_name) flips. 1:1 controlled test.
2. Does it generalize to other providers? ZAI GLM-5.2 is the obvious second pair. Skynet has both — minimax-m3 and zai-coding/glm-5.2 route through the same proxy.
3. What causes the gap? Network latency? TLS overhead? Warm vs cold model endpoints? Batched queueing in the proxy?

The 8 routes under test

Each row is a single 1:1 model pair across two transports. Model name and credentials are the only thing that changes.

Each pair runs N=1..5 parallel agents × 5 reps. 8 × 5 × 5 = 200 trials total. Same code_review task fixture as the original Phase 1+2 sweep.

N=1 baseline — Skynet wins 3 of 4 pairs

Two observations:

1. Skynet is tighter (lower σ) on 3 of 4 pairs. Direct routes show 6-14s σ, Skynet routes show 4-8s. The proxy smooths out per-request variance — fewer cold-call hitches.
2. The ZAI models benefit most from the proxy. GLM-5.1 sees +26.5% and GLM-5.2 sees +13.6%. Minimax-M3 sees nothing (within noise). The pattern suggests Skynet is fixing a ZAI-specific issue (likely vendor-side queueing under load) more than a generic network-latency issue.

Throughput — Skynet is consistent, direct is rate-limited at high N

Per-call latency stays roughly flat across N=1..5 for every route (no per-call degradation under concurrency). The interesting metric is throughput = total work done / wall-clock:

Observations:

1. All routes scale roughly linearly up to N=5. No model shows the “plateau at N=3” pattern from the original hypothesis. Per-call latency stays within ±15% of N=1 across all 8 routes at N=5.
2. ZAI GLM-5.2 is the only model where per-call latency DECREASES at N=5. This suggests the model itself is the bottleneck and ZAI’s batch processing path gets faster when requests come in parallel (or our N=1 measurement had an unlucky outlier pulling the mean up — σ=14.3s on N=1 vs σ=5.4s on N=5).
3. Skynet-minimax-m2.7 is the only route showing meaningful per-call slowdown at higher concurrency (26.4s → 32.1s, +21% at N=5) — but its σ at N=2 was 92.9s due to a couple of 300s timeouts, so the mean is noisy. Real per-call degradation is probably closer to 0-10%.
4. Throughput ranking at N=5 is led by direct-minimax-m2.7 (0.149 tasks/s), then skynet-minimax-m2.7 (0.130), then GLM-5.2 routes around 0.088-0.094. The M3-tier models cluster around 0.078-0.081. No clear Skynet-vs-direct pattern at N=5 — the gap from the original Phase 2 sweep (where Skynet dominated at high N) does not reproduce on these routes with the bug-fixed dispatcher.

The headline is: the “Skynet throughput leader” pattern from Phase 2 was a per-route quirk on minimax-m3 + the previous broken dispatcher (likely some routes were silently failing and producing unrealistically fast N=5 times). With the bug fix, the picture is much closer to “all routes scale linearly; per-call latency is dominated by model inference time, not network.”

Root cause — TLS handshake + HTTP TTFB

The dispatcher also captured per-call HTTP time-to-first-byte on every trial. Combined with the diagnostic probe (diagnose_routes.py), the picture is clear:

Two conclusions:

1. TLS handshake is 37-58% faster on Skynet for both providers. Cloudflare anycast vs vendor load-balancer geo distance.
2. HTTP TTFB warm is faster on Skynet, with the gap being much larger for ZAI (~2.5×) than for Minimax (~1.4×). This is consistent with the wall-clock pattern — Skynet’s advantage is bigger for ZAI, which suggests vendor-side queueing/serialization on ZAI’s api.z.ai endpoint is the dominant cost, not network latency.

The wall-clock advantage at N=1 is therefore mostly explained by:

• TLS handshake amortization (~40-65ms per call, ~1-3% of total wall at 30-60s N=1)
• HTTP TTFB on the first byte (~0.2-2.5s per call, ~1-7% of total wall)
• The remaining ~90% of the latency is model inference, which is the same on both routes (same model)

That math suggests most of Skynet’s N=1 wall-clock advantage is NOT from TLS/TTFB. It’s likely from queueing/connection-pool reuse on the Skynet proxy — connections stay warm across requests, so the per-request handshake cost amortizes to zero after the first call. Vendor endpoints probably tear down the connection after each request and re-handshake.

Per-agent TTFB vs wall-clock

Each point is one agent call (5 reps × 5 N-levels × 8 routes = up to 200 points × 5 = 1000). Points cluster by route — clusters on the left (low TTFB) are Skynet, clusters on the right (higher TTFB) are direct.

What this isn’t

• Not a quality benchmark. Same model either way; output quality should be identical. The only thing that varies is the network path.
• Not a cost benchmark. Both routes are subscription / free tier — cost_usd is $0 across the board. (Though we still capture cost_usd so a future 1P-rate comparison is possible.)
• Not a fairness verdict on Skynet vs direct. If anything, the result is “Skynet is faster on N=1, but the gap is mostly proxy warmup/queueing, not network latency.” For an isolated single call, the gap is real but small; for batched or concurrent workloads, it grows.

What changed during the run

Dispatcher bug found + fixed mid-sweep: the first 200-trial sweep ran successfully but 150/200 trials returned [BLANK_RESPONSE] with http_status=404 and errs=0. Root cause was a url.partition("/") truncation that sent POST /v1 instead of POST /v1/chat/completions — Skynet + ZAI both 404 on /v1 alone, while api.minimax.io silently accepts it (which is why direct-Minimax trials appeared to work). Plus http.client.HTTPSConnection.getresponse() doesn’t raise on 4xx, so the 404 was swallowed as a “success.” Fixed with urllib.parse.urlparse() for the path split + explicit if status >= 400: return error. The 50 valid direct-minimax trials from the first run were preserved; the 150 broken trials were deleted and rerun from scratch. The 200 trials in the data here are all bug-fixed and clean.

Reproduction

cd /var/lib/herman/swarm-hermanity
python3 scripts/diagnose_routes.py                  # ~30s — TCP/TLS/HTTP TTFB probe
python3 scripts/compare_routes.py --reps 5          # ~2-3 hrs — full 200-trial sweep
python3 scripts/analyze_routes.py                   # renders SVGs to static/img/charts/

Raw trial data lives at results/route-comparison.jsonl (gitignored, regenerable). Diagnostic probes at results/route-diagnostic.jsonl. Both depend on the same env vars (MINIMAX_API_KEY, SUBPULSE_SKYNET_API_KEY, SUBPULSE_ZAI_API_KEY) from ~/.hermes/.env + ~/.hermes/plugins/subscription_pulse/.env.