vLLM GPU tests on GitHub Actions

vLLM tests can run in GitHub Actions on a Machine GPU runner. Start with a small model to check installation and inference before you spend time loading the one you use in production.

Set up the runner

Complete the quickstart and give the Machine GitHub App access to your repository. Start with $10 of free GPU time. No charge, no commitment. The free tier allows one job at a time, up to 30 minutes each.

Use an L4 for this example. Save the following workflow and start it from the Actions tab. It installs vLLM into an isolated environment and lets vLLM choose its compatible PyTorch dependency.

name: vLLM inference smoke test
on: workflow_dispatch
permissions:
  contents: read
jobs:
  inference:
    runs-on: machine/id=${{ github.run_id }}-${{ github.run_attempt }}/gpu=l4/tenancy=spot
    timeout-minutes: 25
    steps:
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install vLLM
        run: |
          python -m venv .venv
          .venv/bin/python -m pip install vllm==0.10.2 transformers==4.55.2
      - name: Generate tokens
        run: |
          nvidia-smi
          .venv/bin/python - <<'PY'
          from vllm import LLM, SamplingParams
          model = LLM(
              model='Qwen/Qwen2.5-0.5B-Instruct',
              dtype='half',
              max_model_len=512,
              gpu_memory_utilization=0.7,
              enforce_eager=True,
          )
          result = model.generate(
              ['Write one sentence about GPU tests.'],
              SamplingParams(temperature=0, max_tokens=32),
          )
          assert len(result) == 1
          assert result[0].outputs[0].token_ids, 'No tokens generated'
          print(result[0].outputs[0].text)
          PY

These versions keep vLLM and its tokenizer API compatible. Keep your application and CI on the same tested versions. For repeatable model tests, also set revision to the model commit you use.

Decide what passing means

Generating tokens proves the inference path ran. It doesn’t prove the answer is useful. Add assertions for your application’s contract: valid JSON, required fields, tool names or a score on a fixed evaluation set.

A zero temperature isn’t a promise of identical output across different hardware and library versions. Avoid an exact prose comparison unless that is the behaviour you’re deliberately testing.

The model download and package installation count towards job time. Keep the first test small; use purchased compute or a plan for runs over 30 minutes. Only enable PR runs for code you trust, and keep model-access tokens out of untrusted workflows.

See the vLLM offline inference API and the LLM evaluation guide for the next step beyond a smoke test.