Benchmarks

This page describes a reproducible benchmark you can run on your own account to compare machine.dev runners against GitHub-hosted runners: a cold LLVM build. We have not published our own results, so no claim on this site depends on them.

Methodology

We’re publishing a single defensible benchmark that anyone can reproduce on their own account: a cold LLVM build, run across multiple machine.dev runner configurations and compared against GitHub-hosted baselines.

Workload

LLVM 18.1.x compilation, the same workload OpenBenchmarking uses as their reference compile-time benchmark. LLVM is:

• Industry standard for compile-time benchmarking
• Heavy C++ that exercises real compiler workloads
• Parallelizable to as many cores as the runner has
• Cross-architecture — same source builds on X64 and ARM64
• Long enough (~30 minutes baseline) that improvements are clearly measurable
• Reproducible — pinned source tarball, deterministic flags

Configuration

git clone --depth 1 --branch llvmorg-18.1.8 https://github.com/llvm/llvm-project
cd llvm-project
cmake -G Ninja -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DLLVM_ENABLE_PROJECTS="clang;lld" \
  -DLLVM_TARGETS_TO_BUILD=X86 \
  llvm
ninja -C build -j$(nproc)

• Cold cache (no ccache/sccache)
• Clean checkout each run
• 3 runs per configuration, median reported

Runners

Metrics

• Wall-clock build time (cold cache, median of 3 runs)
• Cost in dollars (build_time_min × spot_$/min)
• Builds per dollar for cost-efficiency comparison
• Memory peak via /usr/bin/time -v

Run it yourself

1. Add a workflow with one job per runner in the table above, each running the configuration script
2. Trigger it via workflow_dispatch
3. Compare wall-clock time and the per-job cost shown on the machine.dev dashboard

Related

• GPU Runners
• CPU Runners
• Pricing