Choosing a side-kick for Claude

I'm a huge fan of the BMAD Method using Claude Code on a Claude Pro Max subscription, but one bit of advice I've been ignoring for a while now is that the Code Review pass should be performed with a different LLM. When you're paying $$$ for the Claude subscription anyway, it's tempting to use it for everything.

But now I'm curious about whether a different LLM for code review delivers better results. Obviously an alternate frontier model would be top choice, but there's not much money left in the kitty, so I'm casting about for a Chinese model to take on the task.

My top pick for CR alone would probably be DeepSeek V4 Pro in Thinking mode, at 0.435 / 0.87 ($/Mtok) and cache hits ~$0.0036/M. The CR workload is output-light and input-heavy, and DeepSeek's output cost undercuts most other models' input cost. Prefix caching means re-reviewing iterations of the same PR is nearly free.

Kimi K2.6, GLM-5.1 and Qwen3.5/Qwen-Max are not far behind, to be honest.

At the same time I fancy experimenting with OpenClaw a bit, and I'd rather use a single account for everything. OpenClaw requires strong tool-calling abilities over straight reasoning, so once you factor that in, I figure the best option for the combined task is GLM 5.1.

Logically, buying your credits direct from z.ai makes sense, and the blogs will tell you their Coding Plan Lite is excellent value, with prices quoted from $3 up to $10... but when I looked into it, it was $18/month, and that's the quarterly price.

Oh well, PAYG API use is probably a better fit anyway: code review is bursty, and OpenClaw is always-on and constantly replaying stuff.

Rather than go through z.ai direct (they are, by all accounts, the worst choice host for the AI which they themselves developed) I'll use a host like OpenRouter or DeepInfra.

OpenRouter gives you some fail-over resilience, but at a 5.5% premium: both my intended use-cases can tolerate a bit of down time, so I plumped for DeepInfra. DeepInfra also has explicit cache pricing ($0.205/Mtok), and Claude.ai reckons that "the realistic blended workload is 7:2:1 cache:input:output — cache behavior is the workload, not an edge case." Input costs $1.40/Mtok, output costs $4.40/Mtok.

A bit of .zshrc magic later:

glm() {
  ANTHROPIC_BASE_URL=https://api.deepinfra.com/anthropic \
  ANTHROPIC_AUTH_TOKEN=$DEEPINFRA_TOKEN \
  ANTHROPIC_MODEL=zai-org/GLM-5.1 \
  ANTHROPIC_DEFAULT_HAIKU_MODEL=zai-org/GLM-4.7 \
  CLAUDE_CODE_MAX_OUTPUT_TOKENS=32768 \
  claude "$@"
}

and we've got an alternative to claude I can use specifically for code review tasks.

I've doubled the CLAUDE_CODE_MAX_OUTPUT_TOKENS target to 32768, which is twice DeepInfra's recommended default, but half of z.ai's recommendation. GLM5.1 is a reasoning model, so a good 10k of that will go on "thinking". The Haiku model shouldn't ever get anywhere near that limit.

I'll use that lighter ("haiku") model for the OpenClaw stuff: it's GLM-4.7 (FP4): $0.40/Mtok in, $1.75/Mtok out. We could probably push routine OpenClaw turns cheaper still, GLM-4.7-Flash drops to $0.06/M input.

Both GLM models on DeepInfra support function calling + JSON mode, which OpenClaw's tool loop needs.

Using it

Now it's simply a matter of remembering to start another session and run glm in it for code review, instead of /clear and /code-review in the existing session, which will be a tough habit to break.

My initial impression is that it is....astonishingly slow. A first-pass CR using Claude alone takes maybe 10 minutes, the equivalent with GLM-5.1 via DeepInfra takes... 2 hours and 38 minutes. 😱

Benchmarks, then

OK that was unexpected! I think this means we need to compare all the things. For reference my testing is done at 6pm onwards UK time, so the US is awake. I'm doing a BMAD code-review pass on a moderately complex project, quite far into a complex plan (epic 22, story 3). I'll stash the results of each pass and reset the git repo in between each run so that it's a level playing field.

Ground truth

I've been saying Claude code takes 10 minutes to complete this task, but let's get a better idea, using my original BMAD code-review workflow (with a clear context).

Ok, Claude Opus 4.8 takes 13 minutes.

Contestant #2, please

Let's revisit the whole "z.ai is the slowest thing evah!" claim. I got Claude to stash those CR results, then reconfigured to use z.ai directly, with the same models:

zai() {
 ANTHROPIC_BASE_URL="https://api.z.ai/api/anthropic" \
 ANTHROPIC_AUTH_TOKEN="..." \
 ANTHROPIC_DEFAULT_HAIKU_MODEL="glm-4.7" \
 ANTHROPIC_DEFAULT_SONNET_MODEL="glm-5-turbo" \
 ANTHROPIC_DEFAULT_OPUS_MODEL="glm-5.1" \
 CLAUDE_CODE_MAX_OUTPUT_TOKENS=32768 \
 claude "$@"
}

For reference, z.ai's API for GLM-5.1 costs $1.4/Mtok in, $0.26/Mtok cached, $4.4/Mtok out (pretty much the same as the DeepInfra pricing).

z.ai's direct API took 36 minutes 😮. This cost about $10 to run (hard to be sure, their website is dreadful).

Contestant #3, please

Same models, but this time running from Fireworks.ai. My invocation script is:

fw() {
  ANTHROPIC_BASE_URL="https://api.fireworks.ai/inference" \
  ANTHROPIC_AUTH_TOKEN="..." \
  ANTHROPIC_DEFAULT_HAIKU_MODEL="accounts/fireworks/models/glm-4p7" \
  ANTHROPIC_DEFAULT_OPUS_MODEL="accounts/fireworks/models/glm-5p1" \
  CLAUDE_CODE_MAX_OUTPUT_TOKENS=32768 \
  claude "$@"
}

fireworks.ai charges $1.40/Mtok in, $0.26/Mtok cached and $4.40/Mtok out. So on a par with the rest: there aren't any hidden costs or quantisation penalties that I can see.

Fireworks.ai completed the task in 33 minutes. This cost me $8 to run.

Contestant #4, please

Same models, but this time running from BaseTen. My invocation script is:

bst() {
  ANTHROPIC_BASE_URL="https://inference.baseten.co" \
  ANTHROPIC_AUTH_TOKEN="..." \
  ANTHROPIC_DEFAULT_HAIKU_MODEL="zai-org/GLM-4.7" \
  ANTHROPIC_DEFAULT_OPUS_MODEL="zai-org/GLM-5.1" \
  CLAUDE_CODE_MAX_OUTPUT_TOKENS=32768 \
  claude "$@"
}

D'oh!

A hot issue that many are experiencing thanks to recent Claude Code updates. There's a fix on the issue trackers though, a small node proxy:

#!/usr/bin/env node
'use strict';

// bst-proxy: a single-purpose shim between Claude Code and Baseten.
//
// Problem it solves: Claude Code (Opus 4.8 shape) injects mid-conversation
// system instructions as { "role": "system" } entries INSIDE messages[].
// Anthropic's own API folds those into system handling; Baseten's Anthropic
// endpoint validates strictly and 400s with:
//   messages[N].role: unknown variant `system`, expected `user` or `assistant`
//
// What it does: for any request body containing messages[], it lifts every
// role:"system" entry out of messages[] and appends its text to the top-level
// `system` field, then forwards the request to Baseten UNCHANGED otherwise.
// Anthropic-in, Anthropic-out. No OpenAI translation, so cache_control and
// content blocks pass through intact.
//
// Run:  node bst-proxy.js        (listens on 127.0.0.1:8788)
// Point Claude Code at it with ANTHROPIC_BASE_URL=http://127.0.0.1:8788
// (Claude Code appends /v1/messages itself.)

const http = require('http');
const https = require('https');

const UPSTREAM = 'https://inference.baseten.co';
const PORT = process.env.BST_PROXY_PORT ? Number(process.env.BST_PROXY_PORT) : 8788;

// Lift role:system entries out of messages[] and fold them into top-level system.
function foldSystemMessages(body) {
  if (!body || !Array.isArray(body.messages)) return body;

  const folded = [];      // text blocks pulled from mid-conversation system msgs
  const remaining = [];   // messages[] with the system entries removed

  for (const msg of body.messages) {
    if (msg && msg.role === 'system') {
      if (typeof msg.content === 'string') {
        folded.push({ type: 'text', text: msg.content });
      } else if (Array.isArray(msg.content)) {
        for (const block of msg.content) {
          if (block && block.type === 'text') {
            folded.push({ type: 'text', text: block.text });
          } else if (block) {
            // non-text block in a system message is unusual; keep it defensively
            folded.push({ type: 'text', text: JSON.stringify(block) });
          }
        }
      }
      // (entry intentionally dropped from messages[])
    } else {
      remaining.push(msg);
    }
  }

  if (folded.length === 0) return body; // nothing to fold; leave untouched

  // Normalize existing top-level system into an array of blocks so we can append
  // without disturbing the original blocks (and their cache_control markers).
  let systemArr = [];
  if (typeof body.system === 'string') {
    if (body.system.length) systemArr.push({ type: 'text', text: body.system });
  } else if (Array.isArray(body.system)) {
    systemArr = body.system.slice();
  }

  body.system = systemArr.concat(folded); // folded content goes AFTER the prefix
  body.messages = remaining;
  return body;
}

const server = http.createServer((req, res) => {
  if (req.url === '/healthz') {
    req.resume();
    res.writeHead(200, { 'content-type': 'text/plain' });
    res.end('ok');
    return;
  }

  const chunks = [];
  req.on('data', (c) => chunks.push(c));
  req.on('end', () => {
    let outBuf = Buffer.concat(chunks);

    // Transform only JSON bodies that carry a messages[] array.
    if (outBuf.length && /json/i.test(req.headers['content-type'] || '')) {
      try {
        const json = JSON.parse(outBuf.toString('utf8'));
        if (json && Array.isArray(json.messages)) {
          foldSystemMessages(json);
          outBuf = Buffer.from(JSON.stringify(json), 'utf8');
        }
      } catch (_) {
        // not JSON / parse failed: forward bytes unchanged
      }
    }

    const headers = Object.assign({}, req.headers);
    delete headers['host'];
    delete headers['content-length'];
    delete headers['connection'];
    // Baseten reads Authorization; Claude Code already sends ANTHROPIC_AUTH_TOKEN
    // as a Bearer Authorization header. Fall back from x-api-key just in case.
    if (!headers['authorization'] && headers['x-api-key']) {
      headers['authorization'] = 'Bearer ' + headers['x-api-key'];
    }
    headers['content-length'] = Buffer.byteLength(outBuf);

    const u = new URL(UPSTREAM + req.url);
    const upstream = https.request(
      { method: req.method, hostname: u.hostname, path: u.pathname + u.search, headers },
      (up) => {
        res.writeHead(up.statusCode, up.headers);
        up.pipe(res); // streams SSE through untouched
      }
    );
    upstream.on('error', (err) => {
      res.writeHead(502, { 'content-type': 'application/json' });
      res.end(JSON.stringify({
        type: 'error',
        error: { type: 'api_error', message: 'bst-proxy upstream error: ' + err.message },
      }));
    });
    if (outBuf.length) upstream.write(outBuf);
    upstream.end();
  });
});

server.listen(PORT, '127.0.0.1', () => {
  process.stderr.write(`bst-proxy on http://127.0.0.1:${PORT} -> ${UPSTREAM}\n`);
});

and that means changing our zshrc function to:

bst() {
  # bring the proxy up if it isn't already
  curl -sf http://127.0.0.1:8788/healthz >/dev/null 2>&1 || { node ~/bin/bst-proxy.js & sleep 0.4; }
  ANTHROPIC_BASE_URL="http://127.0.0.1:8788" \
  ANTHROPIC_AUTH_TOKEN="$BASETEN_API_KEY" \
  ANTHROPIC_DEFAULT_HAIKU_MODEL="zai-org/GLM-4.7" \
  ANTHROPIC_DEFAULT_OPUS_MODEL="zai-org/GLM-5.1" \
  CLAUDE_CODE_DISABLE_1M_CONTEXT=1 \
  CLAUDE_CODE_MAX_OUTPUT_TOKENS=32768 \
  claude "$@"
}

baseten.co's GLM-5.1 costs $1.30/Mtok in, $0.26/Mtok cached, and $4.30/Mtok out, so it's ever so slightly cheaper than the rest.

baseten.co failed to complete the task. After 51m and 111.6k tokens it had spent $20 so I pulled the plug. Shame, because at the 32m mark it looked like it was going to turn in a similar result to fireworks.ai, and then it just went quiet and sat in a loop hoovering up tokens.

Conclusions

1. DeepInfra offers no cost advantages, and is glacially slow
2. z.ai direct is not as bad as everyone makes out (although I only ran one test, and people say it's highly variable)
3. fireworks.ai is a bit faster than z.ai
4. basten.co is super flakey
5. Using a different LLM for code reviews is going to cost me $10 a pop, I'm not convinced it's worth it