QA Sphere CLI Tool: Playwright Integration

This guide shows you how to integrate Playwright test results with QA Sphere using the QAS CLI tool.

Installation

Prerequisites

• Node.js version 18.0.0 or higher
• Playwright test project

Installation Methods

1. Via NPX (Recommended) Simply run npx qas-cli. On first use, you'll need to agree to download the package. You can use npx qas-cli in all contexts instead of the qasphere command.

   Verify installation:

   npx qas-cli --version

2. Via NPM

   npm install -g qas-cli

   Verify installation:

   qasphere --version

Configuration

The QAS CLI requires two environment variables to function properly:

• QAS_TOKEN - QA Sphere API token
• QAS_URL - Base URL of your QA Sphere instance (e.g., https://qas.eu2.qasphere.com)

You can create your API key under the QA Sphere settings.

To do that:

1. Go to Settings by clicking the gear icon in the top right corner.
2. Click API Keys
3. Click Create API Key and generate one

These variables can be defined in several ways:

1. Environment Variables: Export them in your shell:

   export QAS_TOKEN=your_token_here
   export QAS_URL=your_qas_url

2. .env File: Place a .env file in the current working directory:

   # .env
   QAS_TOKEN=your_token
   QAS_URL=your_qas_url

3. Configuration File: Create a .qaspherecli file in your project directory or any parent directory:

   # .qaspherecli
   QAS_TOKEN=your_token
   QAS_URL=your_qas_url
   
   # Example with real values:
   # QAS_TOKEN=tst0000001.1CKCEtest_JYyckc3zYtest.dhhjYY3BYEoQH41e62itest
   # QAS_URL=https://qas.eu1.qasphere.com

Playwright Configuration

To generate JUnit XML reports from Playwright, configure the JUnit reporter in your playwright.config.js:

// playwright.config.js
const { defineConfig } = require('@playwright/test');

module.exports = defineConfig({
  testDir: './tests',
  timeout: 30000,
  
  // JUnit reporter for QA Sphere integration
  reporter: [
    ['list'],  // Console output
    ['junit', { outputFile: 'junit-results/results.xml' }]  // For QA Sphere
  ],
  
  use: {
    headless: true,
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
  
  projects: [
    { name: 'chromium', use: { browserName: 'chromium' } },
    { name: 'firefox', use: { browserName: 'firefox' } },
    { name: 'webkit', use: { browserName: 'webkit' } },
  ],
});

Key Configuration Points

• JUnit Reporter: The junit reporter generates XML files that QA Sphere can process
• Output File: Specify a consistent output path (e.g., junit-results/results.xml)
• Screenshots: Configure screenshot: 'only-on-failure' to capture screenshots for failed tests
• Videos: Use video: 'retain-on-failure' to keep videos of failed tests

Using the junit-upload Command

The junit-upload command is used to upload JUnit XML test results to QA Sphere. The command will automatically create a new test run if no run URL is provided, or upload to an existing run if a run URL is specified.

Basic Syntax

# Create new test run automatically
qasphere junit-upload <path-to-junit-xml>

# Upload to existing test run
qasphere junit-upload -r <run-url> <path-to-junit-xml>

Options

• -r, --run-url: Optional URL of the QA Sphere test run. If not provided, a new test run will be created automatically
• --attachments: Upload detected attachments with test results (screenshots, videos, traces)
• --force: Ignore errors and continue uploading

Playwright-Specific Examples

1. Create a new test run and upload results:

   # Run Playwright tests
   npx playwright test
   
   # Upload results to QA Sphere
   qasphere junit-upload junit-results/results.xml

2. Upload to an existing test run:

   qasphere junit-upload -r https://qas.eu1.qasphere.com/project/P1/run/23 junit-results/results.xml

3. Upload with attachments (screenshots, videos, traces):

   qasphere junit-upload --attachments junit-results/results.xml

4. Using NPX (no global installation needed):

   npx playwright test
   npx qas-cli junit-upload --attachments junit-results/results.xml

NPM Script Integration

Add a convenient script to your package.json:

{
  "scripts": {
    "test": "playwright test",
    "test:upload": "playwright test && npx qas-cli junit-upload --attachments junit-results/results.xml"
  }
}

Then run:

npm run test:upload

Why this approach?

• No version pinning: Using npx qas-cli ensures you always get the latest version
• Convenience: A simple npm run test:upload is easier to remember
• CI/CD friendly: The script can be easily integrated into GitHub Actions, GitLab CI, or other pipelines
• Team consistency: Everyone runs the same command with the same flags

Test Case Marker Requirements

To ensure successful mapping of test results to QA Sphere test cases, your Playwright tests should include a QA Sphere test case marker.

A marker uses the format PROJECT-SEQUENCE (e.g., PRJ-003), where:

• PROJECT is your QA Sphere project code.
• SEQUENCE is the unique sequence number of the test case within the project.

This marker allows the CLI tool to correctly associate an automated test result with its corresponding test case in QA Sphere. Markers are used for searching and are displayed prominently in the QA Sphere Web UI.

Playwright Test Examples with Markers

// tests/login.spec.ts
import { test, expect } from '@playwright/test';

test('PRJ-312: Login with valid credentials', async ({ page }) => {
  await page.goto('https://example.com/login');
  await page.fill('#username', '[email protected]');
  await page.fill('#password', 'password123');
  await page.click('button[type="submit"]');
  await expect(page).toHaveURL(/dashboard/);
});

test('PRJ-313: Login with invalid credentials', async ({ page }) => {
  await page.goto('https://example.com/login');
  await page.fill('#username', '[email protected]');
  await page.fill('#password', 'wrongpassword');
  await page.click('button[type="submit"]');
  await expect(page.locator('.error')).toBeVisible();
});

The marker should be included at the start of your test name. Examples of valid test case names:

• PRJ-312: Login with valid credentials
• PRJ-313: Login with invalid credentials

Attachment Support

Playwright automatically captures screenshots, videos, and traces for failed tests when configured. The --attachments flag will automatically detect and upload these files to QA Sphere.

Supported Attachments

• Screenshots: Captured on test failure (when screenshot: 'only-on-failure' is configured)
• Videos: Retained on failure (when video: 'retain-on-failure' is configured)
• Traces: Playwright trace files for debugging

Uploading Attachments

# Upload results with all attachments
qasphere junit-upload --attachments junit-results/results.xml

The CLI will automatically:

1. Detect attachment files referenced in the JUnit XML
2. Match them to the corresponding test cases
3. Upload them to QA Sphere alongside test results

Troubleshooting

Common Issues

1. JUnit XML not found

   Error: File not found: junit-results/results.xml

   Solution: Ensure Playwright is configured with the JUnit reporter and the output path matches your upload command.

2. Test cases not matching

   • Ensure your test names include the marker format (e.g., PRJ-312: Test name)
   • Verify the marker matches an existing test case in QA Sphere
   • Check that the project code in the marker matches your QA Sphere project

3. Attachments not uploading

   • Verify --attachments flag is included
   • Check that screenshots/videos are being generated (configure in playwright.config.js)
   • Ensure attachment paths in JUnit XML are relative to the working directory

Getting Help

For help with any command, use the -h flag:

qasphere junit-upload -h

Or with NPX:

npx qas-cli junit-upload -h

Complete Example Workflow

Here's a complete example of running Playwright tests and uploading to QA Sphere:

# 1. Set up environment variables (or use .env file)
export QAS_TOKEN=your_token_here
export QAS_URL=https://qas.eu1.qasphere.com

# 2. Run Playwright tests
npx playwright test

# 3. Upload results to QA Sphere
npx qas-cli junit-upload --attachments junit-results/results.xml

Or as a single command with an npm script:

npm run test:upload

By following these guidelines, you can effectively integrate your Playwright test results with QA Sphere, streamlining your test management process.