CI with GitHub Actions or any CI

The GitHub App already runs suites as PR checks. Use this when you deploy yourself (a custom preview environment, a staging deploy step, another CI system) and want qaitai to test that deploy and gate the pipeline. Runs happen on your qaitai instance (qaitai Cloud or self-hosted) with the same domain verification, plan limits and model keys as runs started in the app.

1. Create an API key

In the app, Settings → API keys → Create key (workspace admins). For CI, the default scopes are enough: runs:read, runs:write, issues:read and projects:read. The key is shown once; store it as a CI secret, e.g. QAITAI_API_KEY.

On qaitai Cloud, verify the domain you test first: runs against unverified hosts are refused with 422 run_rejected (domain_unverified).

2. GitHub Actions

name: qaitai
on:
  deployment_status:
jobs:
  qa:
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4 # only needed for prompt-file
      - id: qaitai
        uses: qaitai/action@v1
        with:
          api-key: ${{ secrets.QAITAI_API_KEY }}
          # instance: https://qaitai.acme.internal   # self-hosted; default https://app.tryqaitai.com
          url: ${{ github.event.deployment_status.environment_url }}
          path: /checkout
          prompt-file: .github/qaitai/checkout.md
          workers: 2
          fail-on: high
      - if: always()
        run: echo "qaitai ${{ steps.qaitai.outputs.status }}: ${{ steps.qaitai.outputs.run-url }}"

Outputs: run-id, run-url, status (passed, failed, flaky, cancelled, errored), issues and blocking-issues. The step writes a job summary with every finding linked to its page, and fails when a finding is at or above fail-on, or when the run errors or is cancelled.

Retries are safe: the action sends an Idempotency-Key per workflow run attempt, so a retried request never starts (or bills) a second run, while Re-run jobs does start a new one.

Any CI

The CLI does the same from any shell. --instance (or QAITAI_INSTANCE) switches it from a local run to a run on your instance; the key comes only from QAITAI_API_KEY.

export QAITAI_API_KEY=qak_…   # from your CI's secret store
npx qaitai run "$DEPLOY_URL" --file qa/checkout.md --workers 2 \
  --instance https://app.tryqaitai.com --fail-on high --json > qaitai.json

npx qaitai suites run <suite-id> --url "$DEPLOY_URL" --instance https://app.tryqaitai.com
npx qaitai doctor --instance https://app.tryqaitai.com   # checks the instance and the key's scopes

It streams progress, prints the findings and the run URL, and exits like a local run: 0 ok, 1 findings at or above --fail-on, 2 error (including an errored run or a refused request), 130 cancelled. Ctrl-C cancels the run on the instance. Local-only flags (--headed, --browser, --secret, --secrets-file, --report, --max-steps, --max-minutes, --path) are rejected in this mode: secrets come from the workspace and limits from its runners.

Plain HTTP

Everything above is built on the REST API: POST /api/v1/runs, then follow GET /api/v1/runs/{id}/events (server-sent events) or poll GET /api/v1/runs/{id}.

curl -X POST "$QAITAI_INSTANCE/api/v1/runs" \
  -H "Authorization: Bearer $QAITAI_API_KEY" \
  -H "Idempotency-Key: $CI_PIPELINE_ID" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://staging.acme.dev","prompt":"Sign up, create a project and invite a teammate"}'