Hermetic Agent Evidence

Read this when:

• running AI-generated code and tests from the same specification;
• keeping the code writer, test writer, and QA reviewer in separate contexts;
• using Crabbox to require and download durable proof files from the remote run.

Crabbox is a useful execution layer for hermetic-agent workflows because the repository owns the agent protocol while Crabbox owns the remote run and evidence plumbing: sync the checkout, run the command on a clean runner or delegated sandbox, require proof artifacts, download selected evidence, and keep the exact command output reviewable.

The best Crabbox fit is a run-evidence pattern, not a new agent framework:

• crabbox run executes the repo-owned hermetic harness.
• --require-artifact turns the proof JSON into a post-run gate.
• --download pulls selected proof files back for review or agent handoff.
• Provider choice stays ordinary Crabbox policy: SSH-backed Linux providers when the operator wants Crabbox-managed SSH and rsync, or delegated providers when their adapter advertises bounded artifact/download support.

Crabbox should not judge model output, store reasoning traces, decide whether a test is correct, or deliver model credentials for this pattern. Those decisions belong to the repo-owned harness.

This page records the repo-local pattern tracked by openclaw/crabbox#1020.

#Pattern Shape

A hermetic-agent harness usually models three roles:

A useful validation fixture intentionally seeds a bad generated test expectation. For example, the implementation follows the spec, the test expects the wrong case-folding result, and QA assigns the disagreement to test_writer. That matters because hidden or generated tests are not automatically truth; the reviewer must still decide which side violated the spec.

#Local Proof

An application repo should keep a local proof command that works without Crabbox. For example:

./scripts/run_hermetic_agents_demo.sh
python3 scripts/hermetic_agents_demo.py --self-test
bazelisk test //:hermetic_agents_e2e_test

Those commands write:

docs/metrics/hermetic-agents-e2e.json
docs/metrics/hermetic-agents-e2e.md

The JSON includes the role manifests, leak-check results, artifact digests, QA verdict, and the exact disagreement assigned to the test writer.

#Crabbox Run

A repository can expose the same proof as a .crabbox.yaml job. With an SSH-backed Linux provider, the user-facing command stays ordinary Crabbox:

crabbox job run hermetic-agents

The job runs the local proof script, requires the JSON proof file after command success, and downloads JSON/Markdown evidence into .crabbox/runs/hermetic-agents/. Crabbox always excludes .crabbox/runs/ from later workspace syncs, so a prior proof is not uploaded as source on the next run.

The same flow without the job wrapper is:

crabbox run \
  --require-artifact docs/metrics/hermetic-agents-e2e.json \
  --download docs/metrics/hermetic-agents-e2e.json=.crabbox/runs/hermetic-agents/hermetic-agents-e2e.json \
  --download docs/metrics/hermetic-agents-e2e.md=.crabbox/runs/hermetic-agents/hermetic-agents-e2e.md \
  --shell './scripts/run_hermetic_agents_demo.sh'

Pick the provider the same way you would for any other Crabbox run:

• Use brokered or direct SSH-backed Linux providers (aws, azure, gcp, hetzner, ssh, and similar) when you want Crabbox-managed SSH, rsync, broker history, telemetry, or warm-box reuse.
• Use a delegated provider only when its adapter supports the required evidence features, such as bounded single-file --require-artifact and --download.

#Concept Map

#Trust Boundary

This pattern does not turn Crabbox into a hostile multi-tenant sandbox. Treat the repository config and harness as executable project automation, just like a Makefile or CI workflow.

Keep these boundaries explicit:

• The code/test writer separation is enforced by the harness manifests, not by Crabbox itself.
• Crabbox can require that the proof file exists; it does not validate the proof schema unless the repo command does so.
• Provider API keys and model/tool credentials must stay out of repo YAML and command arguments.
• Proof artifacts should be small, bounded, and redacted before sharing.
• If a real model-backed version needs credentials, do not forward ambient secrets through env.allow; Crabbox does not provide workload-specific credential delivery.

The path stays deliberately simple:

repo harness -> crabbox run -> required proof artifact -> downloaded evidence

#Why This Belongs In Repo Config

The agent protocol should stay in the application repo:

• the distilled spec and role guides are project-specific;
• the oracle and blame policy are part of the review contract;
• the proof schema can evolve with the demo or product under test.

Crabbox should stay generic:

• sync the exact checkout being reviewed;
• execute the declared command remotely;
• require bounded proof artifacts before reporting success;
• download evidence for reviewers, agents, or CI handoff.

This boundary keeps Crabbox from becoming an agent framework while still making agent output auditable.

#Good Repo Config

Keep the Crabbox YAML focused on execution policy:

jobs:
  hermetic-agents:
    # Omit provider to inherit user/team defaults, or set aws/hetzner/gcp/azure/ssh/etc.
    # provider: aws
    target: linux
    shell: true
    command: ./scripts/run_hermetic_agents_demo.sh
    requiredArtifacts:
      - docs/metrics/hermetic-agents-e2e.json
    downloads:
      - docs/metrics/hermetic-agents-e2e.json=.crabbox/runs/hermetic-agents/hermetic-agents-e2e.json
      - docs/metrics/hermetic-agents-e2e.md=.crabbox/runs/hermetic-agents/hermetic-agents-e2e.md
    stop: always

Keep the agent policy in repo-owned files such as:

spec/problem.md
guides/code-writer.md
guides/test-writer.md
guides/qa-arbiter.md
scripts/hermetic_agents_demo.py

Switch provider to islo or another delegated backend only after confirming that backend supports the required artifact/download capabilities used by the job.

#Review Checklist

Before sharing a hermetic-agent run, verify:

• The proof JSON contains separate code_writer and test_writer manifests.
• Leak checks passed for both writer roles.
• QA verdict and blame assignment are present.
• Required artifacts were enforced by Crabbox, not only generated locally.
• Downloaded proof files contain no secrets or raw customer data.

For artifact semantics and storage limits, see Artifacts. For provider capability details, see Provider Reference, Provider backends, and any selected per-provider page.