In part 1, we wired a fully local coding agent: Gemma 4 served by Docker Model Runner on the host, driving a cagent-based agent inside a Docker Sandbox microVM, with a default-deny network policy between them. It fixed a planted single-file bug, recovered from its own compile error, and cost exactly zero API credits.
But let's be honest, fixing one bug in one file is the "hello world" of agentic coding. This post is about finding the edges: a multi-file refactor, the same model under a different harness, and the inevitable question everyone asks: "OK, but how does it compare to Claude?" Spoiler: the most interesting answer came from the network policy log, not the models.
Raising the bar: a multi-file refactor
I upped the difficulty with a task that spans three files: I wrote failing tests for a Percentiles class that didn't exist (nearest-rank method, contract only visible by reading the test), and asked the agent to create it, extract the sort-a-copy logic out of Stats.median into a shared helper, use it from both classes, and keep the original suite green. Read one file to reverse-engineer a contract, create a second, refactor a third, don't break anything.
Gemma did all of it. It read the test to reverse-engineer the contract, added a getSortedCopy helper to Stats, rewired median to use it, created a correct Percentiles.p() (with bounds-checking on the percentile argument that I never asked for, but fair enough), and ran both suites. It stumbled once, briefly nuking the variance method while editing Stats.java, then noticed via the failing test and put it back. Failure sucks but instructs, even for models apparently.
The nits are real but cosmetic: it leaves its chain of thought behind as code comments, and it ate a trailing newline. I've reviewed worse from humans :)
Same model, different harness
Since the base_url trick from part 1 works with anything OpenAI-compatible, I reran the original bug with a completely different agent, opencode, driving the exact same local Gemma (Docker Sandboxes supports it natively: sbx run opencode). The provider config is a few lines of opencode.json:
{
"provider": {
"dmr": {
"npm": "@ai-sdk/openai-compatible",
"options": { "baseURL": "http://host.docker.internal:12434/engines/v1" },
"models": { "ai/gemma4": { "name": "Gemma 4 (local)" } }
}
},
"model": "dmr/ai/gemma4"
}
Same model, same bug, same fix, different journey. Under cagent, Gemma fixed the median with streams and Collectors, broke compilation on the missing import, and recovered. Under opencode, it sidestepped imports entirely by writing new java.util.ArrayList<>(values) fully qualified, and its one stumble was an edit that didn't match the file (recovered on the next attempt). Two runs are hardly a benchmark, but the pattern is worth chewing on: the model brings the coding ability, and the harness shapes which mistakes even get a chance to happen. I found that genuinely fascinating.
And against the big guns?
For calibration, I gave the exact same task to Claude Code in its own sandbox (sbx run claude, then a one-time /login inside). The result was humbling and expected at the same time: 24 seconds, one pass, zero stumbles. Imports added properly, ArrayList copy, Collections.sort, tests green. What took local Gemma a few minutes and one recovered compile error, the frontier model dispatched before my coffee got cold.
Two observations to keep it honest, though. First, Gemma got there too, on my hardware, for free, with inference never leaving my machine. For a scoped fix, "slower but local" is a perfectly rational trade. Second, the policy log had the best plot twist of the whole experiment: the claude sandbox template allowlists Anthropic's API endpoints, but the balanced policy quietly blocked Claude Code's telemetry:
Blocked requests:
claude-demo http-intake.logs.us5.datadoghq.com:443 No matching allow rule (default deny) 5
Allowed requests:
claude-demo api.anthropic.com:443 56
The sandbox let the agent do its job and ate its phone-home. I never wrote a single Claude-specific rule; the template's defaults plus default-deny produced that on their own. If you needed one image to explain why agents belong in sandboxes, this is it.
Takeaways
- Isolation plus local models is a real combo now. Agent in a microVM, model on the host, policy proxy in between. Inference stays on your machine, and every other outbound byte has to get past the receptionist.
- Default-deny is the right default. It blocked my own model endpoint (mildly annoying) and a frontier agent's telemetry (deeply satisfying) with the same rule: no.
- Small models earn their keep on scoped tasks. A failing test suite and one clear instruction, and Gemma delivered, including a small multi-file refactor. I went looking for the ceiling and didn't find it; it sits higher than I expected.
- The harness shapes the failure modes. Same model made different mistakes under different agents. Worth remembering next time you evaluate "the model".
Everything is on GitHub: the demo project with the agent-authored commits on main, and the opencode and Claude Code runs on the comparison/* branches. Next, I want to race Gemma against bigger open models on longer-horizon tasks (a dependency upgrade, a real feature) and see where the wheels come off. The version-skew bug from part 1 is filed upstream as docker-agent#3871 if you want to follow the fix. Stay tuned, and exciting times!