Capacity And Fallback
Read this when:
- adding or changing machine classes;
- debugging "why did Crabbox pick this instance type?";
- working on AWS or Azure Spot/on-demand failover;
- configuring multi-region or multi-AZ capacity for AWS or Azure.
Crabbox treats capacity in three layers:
- Class fallback — the ordered list of provider instance types that a class request expands into.
- Market fallback — AWS/Azure Spot-to-On-Demand failover within a class.
- Region/AZ routing — where the brokered AWS or Azure path may launch when one region or zone is configured against several.
Hetzner only exercises class fallback. AWS exercises all three; Azure exercises class, market, and region fallback for VM leases. Static SSH and the delegated runners (Blacksmith, Daytona, Islo, E2B, Modal, and the rest) have no capacity fallback because the operator or the external service owns the underlying resource.
#Classes
Class names are provider-neutral intent labels:
tiny smoke checks and minimal repositories
small small repositories and light CI lanes
standard typical CI lane
fast more cores for parallel-friendly suites
large memory-heavy or many-process workloads
beast maximum capacity within the provider's family
Each provider maps the six class names to an ordered list of concrete instance types. That list *is* the fallback chain: try the first; if it is rejected, try the next; and so on until one succeeds or the chain is exhausted.
Inspect the compiled per-provider class catalogs (including AWS Windows, Windows WSL2, and macOS) as shown in Providers.
#When Class Fallback Triggers
Hetzner falls back when:
- the requested server type is unavailable in the configured location;
- the project quota rejects the request;
- the API returns a transient capacity error.
AWS and Azure fall back when a candidate is rejected by:
- insufficient instance capacity (
InsufficientInstanceCapacity); - a vCPU or Spot quota limit (
VcpuLimitExceeded,MaxSpotInstanceCountExceeded); - an unsupported type or account policy (e.g. Free-Tier restrictions).
Crabbox classifies each provider rejection (capacity, quota, unsupported, policy, region) rather than guessing, then advances to the next candidate type. Azure VM leases also bound slow long-running create operations so requests can advance through the candidate chain instead of waiting on a stalled allocation. Non-retryable failures stop the chain immediately so a genuine misconfiguration fails fast instead of churning through every type.
When the chain is exhausted the command fails. On the brokered (Worker) path the failure carries provisioningAttempts — one record per type tried, with its region, market, failure category, and message — and the same metadata is attached to the failed lease record on the coordinator, so an operator can see what went wrong without rerunning the workflow:
crabbox history --lease cbx_...
#Explicit Type Override
--type c7a.16xlarge (and the matching type: config key) pins one exact instance type and skips the class fallback chain. The contract is "give me this type, not a substitute": if the provider rejects it, Crabbox fails loudly rather than silently choosing a different type.
Use --type when:
- you want deterministic capacity for a benchmark;
- you are pinning a specific generation to work around a known issue;
- you are debugging the capacity layer itself.
For everything else, prefer a class so the fallback chain absorbs transient rejections without operator intervention.
#AWS And Azure Market Fallback
AWS and Azure support two markets: spot and on-demand.
capacity:
market: spot
fallback: on-demand-after-120s
capacity.market: spot (the default) requests Spot capacity first. capacity.fallback controls whether a rejected Spot request retries the same class on On-Demand. Any value that starts with on-demand (the shipped default is on-demand-after-120s) enables the On-Demand retry pass; set it to none (or leave it empty) to never fall back.
The On-Demand pass runs after every Spot candidate in the class chain has been tried and rejected. AWS retries only the candidates whose individual failures are Spot-recoverable, including Spot capacity or quota errors and request errors that explicitly identify Spot as unsupported. Market-independent request or image errors, such as an invalid block-device mapping, are not retried on On-Demand. Azure also treats a slow Spot VM provisioning operation as a capacity miss after the configured on-demand-after-* duration, or after the default 120 seconds when on-demand fallback is disabled with spot-only or none. Slow Azure On-Demand creates are bounded too, so the coordinator can keep trying the class chain before the CLI lease wait expires.
Per-command overrides:
crabbox warmup --market spot
crabbox run --market on-demand -- pnpm test
--market overrides capacity.market for one lease without editing repo config. It must be spot or on-demand. Reach for it when an account is temporarily out of Spot quota or when Spot interruption rates spike. The same value is also available as the CRABBOX_CAPACITY_MARKET environment variable.
#Tencent Cloud Market Selection
Tencent Cloud also honors explicit spot and on-demand choices, mapping them to CVM SPOTPAID and POSTPAID_BY_HOUR. This is selection only: Tencent does not perform market, instance-type, or region fallback, and capacity.fallback does not change that behavior. Spot requests use Tencent's default discounted bid without a Crabbox maximum-price override.
An unconfigured Tencent lease keeps its historical hourly billing, independent of the generic spot default. After upgrading, an explicitly configured capacity.market: spot, environment override, CLI flag, or job market now requests interruptible Spot capacity; select on-demand to keep hourly billing. See the Tencent Cloud provider guide.
#AWS Capacity Hints
Successfully provisioned brokered AWS, Azure, and GCP leases retain ordered provisioningAttempts, including failed candidates from earlier markets and regions or zones. Each record describes the actual attempted type, market, location, and provider-classified failure; AWS quota preflight rejections are recorded even when no instance launch was needed. These diagnostics do not authorize retry or cleanup: each provider still owns those decisions and its durable allocation claims. Failed brokered leases still report their terminal failure separately; this history is not a durable journal of unresolved provisioning or cleanup work.
The brokered AWS path can preflight large requests against Service Quotas:
capacity:
hints: true
When hints is enabled (the shipped default), the broker checks the applied Spot or On-Demand vCPU quota for each candidate type before launching it. Candidates that exceed the quota are recorded as a quota attempt and skipped. Types that fit the quota ceiling still need enough unused regional quota; the preflight does not measure existing usage or Capacity Reservations. Brokered failures also emit advisory CapacityHint records (for example, when a large class is under capacity pressure) alongside provisioningAttempts.
The set of classes treated as high-pressure for hint purposes is controlled on the broker by the CRABBOX_CAPACITY_LARGE_CLASSES environment variable (comma-separated, default beast); it is not a repo config key.
Hints apply only on the brokered path. Direct AWS mode still walks the class chain but runs no quota preflight. To preflight quota before the first warmup in either mode, run:
crabbox doctor --provider aws
It reports the applied EC2 vCPU quota for the relevant market(s) and warns when the default class needs more vCPUs than the account is allowed. A sufficient ceiling reports capacity=unknown quota_limit=sufficient usage=unchecked. Removing an exact --type enables class fallback but does not free quota; a smaller type helps only when enough quota remains for it.
#Region And Availability Zone Routing
capacity:
regions:
- eu-west-1
- us-east-1
availabilityZones:
- eu-west-1a
- eu-west-1b
Ordinary brokered AWS leases try the requested aws.region first, followed by the coordinator's preferred region and the configured capacity regions, with duplicates removed. Private AWS workspaces and pinned Mac leases retain their single-region routing. Promotions from another region do not override the request.
regions is the ordered list of additional brokered AWS regions the coordinator considers. Single-region setups use aws.region and leave capacity.regions empty; multi-region setups list every region the broker may launch into. Regions are tried in order: when every candidate type in a region is rejected for a retryable reason, Crabbox advances to the next region.
Automatic brokered AWS images are selected for each candidate region. If that region has no matching promoted image, Crabbox resolves its normal stock image for the requested OS and architecture. If AWS definitively reports an automatic promotion missing (InvalidAMIID.NotFound), Crabbox retries once with stock and records the image failure in the provisioning attempts. Explicit AMIs, snapshots, and required image capabilities do not silently switch to stock. An empty stock-image lookup permits the next configured region; changing the capacity market cannot fix a missing image.
Brokered Azure uses CRABBOX_AZURE_REGIONS on the coordinator for its Azure-specific region list, so AWS and Azure do not accidentally share incompatible region names. The CLI-level capacity.regions list is still included in Azure lease requests for direct or explicitly shared setups.
availabilityZones narrows zone selection within a region (used today mainly for EC2 Mac Dedicated Host allocation). Zones that do not belong to the active region are ignored.
Both keys also accept comma-separated values from the CRABBOX_CAPACITY_REGIONS and CRABBOX_CAPACITY_AVAILABILITY_ZONES environment variables.
#Capacity Strategy
capacity:
strategy: most-available
capacity.strategy is accepted and persisted (it is one of most-available [default], price-capacity-optimized, capacity-optimized, or sequential), and it shows up in crabbox config show. It does not yet change provisioning behavior: both direct and brokered AWS provisioning currently follow the class chain in order. Treat the field as forward-looking configuration rather than a behavior switch, and prefer ordering the class chain itself when you need a specific preference today.
#Direct Mode Differences
Direct provider mode (no coordinator) supports class fallback and the spot-to-on-demand failover above, but has no Service Quotas preflight, no provisioningAttempts metadata, and no central run history. If a direct AWS run fails to provision, rerun it once through the broker to capture structured provisioningAttempts evidence, then return to direct mode for the rest of the iteration loop.
#Failure Surface
crabbox returns 0 on success. Crabbox-internal failures — including capacity exhaustion, provisioning that never reached SSH, and a lease that expired during a long warmup — are reported before the remote command runs and exit non-zero; when the remote command does run, its own exit code is passed through verbatim. There is no fixed numeric enum for the internal categories (see the CLI exit codes reference); branch on 0 versus non-zero and read stderr or --json for the reason.
The error message names the class chain, the markets that were tried, and — for brokered runs — the provisioningAttempts you can inspect with crabbox history --lease cbx_....
Related docs:
#Optional AWS resource minimums
For a new Linux AWS lease, add --min-vcpus and/or --min-memory-mib to warmup or a one-shot run. These filter the existing class candidates; they do not change their order or turn the preferred type into a minimum. For example, a workload needing four vCPUs and 15 GiB can request:
crabbox warmup --provider aws --class standard --min-vcpus 4 --min-memory-mib 15360
The matching configuration keys are capacity.minVCPUs and capacity.minMemoryMiB. Values are whole numbers from 0 to 2147483647. Zero or omission disables that dimension; command flags can override configured values. MiB means 1,048,576 bytes.
Crabbox checks the candidates against EC2 DescribeInstanceTypes metadata. Candidates below a requested minimum, or with unknown metadata for a requested dimension, cannot launch. Metadata lookup failure is reported as an inability to verify capacity. An exact --type must satisfy the minimum too and retains its exact-type behavior. Market and region fallback, quotas and lease lifetimes remain unchanged. Without minimums, existing requests retain their current candidate lists and advisory metadata behavior.
Both the direct AWS CLI and a supporting coordinator enforce these minimums. A constrained request uses a distinct coordinator creation route, so an older coordinator fails instead of silently ignoring the fields. Fixed --lease-id creation binds the minimums to its original request; changing them on replay conflicts with that request. Updating the client alone does not add coordinator support.
Minimums initially apply to AWS Linux creation on amd64 and arm64. Existing lease --id reuse, ready pools, checkpoint creation, jobs and prewarm do not accept nonzero minimums; unsupported combinations fail before allocation. Use creation-time flags when warming a lease for subsequent ordinary run --id commands. Those commands still require the caller's normal runtime checks.
EC2 metadata describes nominal hardware. Workloads must also check effective process CPU and memory limits before expensive preparation or execution, and select a concurrency mode that fits both. These options do not guarantee runtime headroom, execution duration or a particular worker count.