Troubleshooting
Find answers to common issues. These include problems encountered during setup, job execution, and CI/CD pipeline integration.
Common API Errors
The API gateway intercepts unauthorized or invalid requests before they execute.
Why this happens: The request lacked the required credentials. MCP clients authenticate using the standard Authorization: Bearer <token> header. Direct API and SDK callers use the x-api-key header or the REGRESSIONBOT_API_KEY environment variable.
Solution: Retrieve your API key from the Dashboard. We recommend configuring your client to use the standard Authorization: Bearer <token> header. For MCP integrations using environment variables (such as Codex's bearer_token_env_var), set the configuration parameter to the name of the environment variable (e.g. "REGRESSIONBOT_API_KEY") and export that variable in your shell (export REGRESSIONBOT_API_KEY="your_key"). Do not put the literal API key string directly into the config parameter. The legacy x-api-key header remains supported for backwards compatibility.
Why this happens: Your organization's free trial has expired. The free trial includes 500 screenshot comparisons or 7 days of access. Alternatively, your paid subscription is no longer active.
Solution: Log into the Dashboard. Navigate to the Billing tab. Subscribe to a paid plan (Hobby, Starter, or Pro) to resume triggering jobs.
Why this happens: You queried the status of a Job ID that does not exist or has expired. Visual regression jobs are archived after 30 days.
Solution: Double-check the Job ID UUID string returned from your start request. Verify you are using the correct target organization namespace.
Why this happens: You triggered a job for an existing project. The parameters in your request body did not match the configuration stored in the project's record. This includes parameters like testOrigin, devices, or paths.
Solution: Omit these parameters from your request to use the saved configuration automatically. Alternatively, trigger the job via POST /project/{name}/run. This endpoint always loads the saved configuration. To update the configuration, use the MCP update_project tool. Note that updating the configuration will invalidate existing baselines.
Why this happens: Your organization has run out of its monthly AI summaries allocation. To prevent blocking your CI/CD pipeline, the visual regression job completes successfully. Generating the AI summaries is bypassed.
Solution: Navigate to the Billing tab in your Dashboard. Upgrade to a plan with higher limits. Alternatively, wait for the monthly cycle to reset.
Why this happens: Another job has approved and updated the project baseline since this test run executed. Approving the outdated job would overwrite the newer baseline changes.
Solution: Re-run the visual check to compare the branch changes against the latest updated baseline.
Handling Failed Jobs
If a visual regression job enters the FAILED state, the worker could not capture one or more pages. Common reasons include:
- DNS Failures: The host could not be resolved from inside the cloud network. Examples include
localhostor local company development URLs. - SSL Errors: Self-signed certificates are used on staging servers. Use a publicly trusted certificate for environments you want the cloud workers to capture.
- Authentication Walls: Basic auth, OAuth redirect screens, or captcha verification pages prevent headless browser crawls.
Timeouts & Page Height Limits
Headless visual workers allocate a standard 30-second budget. This budget is used for screenshot capture and lazy-load scroll execution.
Tips to fix timeouts:
- Exclude large media or infinite-scrolling pages from your sitemaps. Examples include product lists or social feeds.
- Provide dynamic element masking using the
data-vr-maskattribute. This bypasses dynamic elements. - Limit comparison depths when invoking sitemap crawler endpoints.
Additionally, visual workers enforce a safe page height limit of 36,000 physical pixels (divided by the Device Pixel Ratio, or DPR). This prevents high-memory crashes on extremely tall pages. For example, at a 1x device scale, the limit is 36,000px; at 2x scale, it is 18,000px; and at 3x scale, it is 12,000px.
Capped Screenshot Fallback Behavior:
If a page exceeds this height limit or causes a browser capture timeout, the system does not fail the job. Instead, it automatically captures and compares a **capped full-page screenshot** up to the maximum safe height. This ensures that you still get coverage of the top sections of extremely tall pages without causing pipeline failures or defaulting to a single viewport.
If only the top portion of a long page is compared, verify that the page height fits within the safe limits, or use viewports or device profiles with lower scale factors.
SSIM & Visual Match Metrics
RegressionBot computes two separate visual comparison metrics. This eliminates false positives caused by subpixel antialiasing or rendering shifts.
Diff Percentage (% Offset)
This is the ratio of exact mismatched pixels divided by total viewport pixels. A minor font weight variation across a whole document can mark 15% of pixels as regression pixels. However, the layout remains structurally correct.
Visual Match Score (SSIM)
Structural Similarity Index (SSIM) measures human-perceived luminance, contrast, and structure. If SSIM stays above 98.5%, the visual variation may represent structural noise. This is not a meaningful product regression.
Pro Tip on Noise Filtering:
Fetch the job summary after completion. Inspect visualMatchScore alongside diffPercentage to identify likely rendering noise:
await job.waitForCompletion();
const summary = await job.getSummary();
const likelyNoise = summary.regressions.filter((regression) =>
(regression.visualMatchScore ?? 0) >= 99 &&
(regression.diffPercentage ?? 0) < 0.5
);Initial Job (Empty Baseline)
Initial jobs run without prior baselines. RegressionBot treats this as a successful execution. It marks each page result status with isNewBaseline: true. Subsequent jobs will automatically compare against this newly established baseline.
Short Screenshot URLs
Troubleshoot common issues when using shortened, clickable screenshot redirect URLs.
Why this happens: The short screenshot URL token has expired. Redirection tokens are only valid for one hour.
Solution: Request the job summary again using the API. This generates fresh short URLs and new tokens in the database.
Why this happens: You requested the job summary before the job finished processing. The status is still PROCESSING or SUMMARIZING.
Solution: Poll the job status endpoint (/job/{jobId}). Wait for the status to become COMPLETED, APPROVED, or FAILED before fetching the summary.
Why this happens: The backend fell back to standard pre-signed URLs. This is caused by a database write error when saving short-lived tokens.
Solution: Check the RegressionBot system status page. Alternatively, contact support. The client received the response successfully. Your network connection is functioning correctly.