Skip to content

[Bug]: Report screenshots are completely black in headless Linux environment #2970

Description

@leiyaoyu187-del

Version

Environment
@midscene/core: 1.10.3
@midscene/web: 1.10.3
Playwright: 1.48.0
Node.js: 22.23.1
OS: Ubuntu 22.04 LTS (headless, no GPU)
Browser: Chromium (headless mode)
MIDSCENE_OUTPUT_FORMAT: single-html

Details

Describe the bug

When running Midscene.js with Playwright in a headless Linux environment (no GPU), the AI agent correctly identifies and interacts with page elements, but the screenshots saved in the HTML report are completely black.

Environment

  • @midscene/core: 1.10.3
  • @midscene/web: 1.10.3
  • Playwright: 1.48.0
  • Node.js: 22.23.1
  • OS: Ubuntu 22.04 LTS (headless, no GPU)
  • Browser: Chromium (headless mode)
  • MIDSCENE_OUTPUT_FORMAT: single-html

Steps to Reproduce

  1. Run a Midscene script on a Linux server with headless Chromium
  2. The target page has CSS animations and gradient backgrounds (a login page with animated background)
  3. Script uses agent.aiTap(), agent.aiAssert() etc.
  4. All AI actions succeed — the agent correctly locates buttons, input fields, and verifies page content
  5. Open the generated HTML report

Expected Behavior

The HTML report should show screenshots of the page at each step, matching what the AI "saw" during execution.

Actual Behavior

All screenshots in the report are completely black. The HTML file is 3-4MB (screenshots ARE embedded as base64 in single-html mode), but the image content is black.

Key Observation

The AI agent can correctly identify elements on the page (e.g., "账号输入框" at coordinates [1330, 346], "登录按钮", etc.). This means the screenshots taken for AI processing are rendered correctly. However, the screenshots saved to the report are black.

This suggests that the screenshots used for AI inference and the screenshots saved to the report are captured at different times or through different mechanisms, and the report screenshots fail to render properly.

What I've Tried

  1. --disable-gpu flag → screenshots still black
  2. --use-angle=swiftshader flag → screenshots completely broken (file not found)
  3. MIDSCENE_OUTPUT_FORMAT=single-html → screenshots embedded but still black
  4. MIDSCENE_OUTPUT_FORMAT=html-and-external-assets → external PNG files not generated at all
  5. Adding waitForTimeout(3000) before actions → no effect on report screenshots

Script Example

page.setViewportSize({ width: 1600, height: 900 });
await page.goto('xxxxx');
await page.waitForTimeout(3000);
await page.getByText('账号密码登录').click();
await page.waitForTimeout(1000);
await agent.aiTap('账号输入框');
await page.keyboard.type('xxx');
await agent.aiTap('密码输入框');
await page.keyboard.type('password');
await agent.aiTap('登录按钮');
await page.waitForTimeout(3000);

### Reproduce link

N/A

### Reproduce Steps

1. Set up a Linux server (Ubuntu 22.04) with Node.js 22+ and no GPU
2. Install Chromium via Playwright: npx playwright install chromium
3. Install Midscene: npm install @midscene/core @midscene/web
4. Create a script that uses PlaywrightAgent to interact with a page that has CSS animations/gradients:

const { chromium } = require('playwright');
const { PlaywrightAgent } = require('@midscene/web/playwright');

process.env.MIDSCENE_OUTPUT_FORMAT = 'single-html';
process.env.MIDSCENE_MODEL_NAME = 'your-model';
process.env.MIDSCENE_MODEL_API_KEY = 'your-key';
process.env.MIDSCENE_MODEL_FAMILY = 'your-family';

(async () => {
  const browser = await chromium.launch({ headless: true, args: ['--no-sandbox'] });
  const page = await browser.newPage({ viewport: { width: 1600, height: 900 } });
  const agent = new PlaywrightAgent(page);
  
  await page.goto('xxxx');
  await page.waitForTimeout(3000);
  await agent.aiTap('账号输入框');
  await page.keyboard.type('test');
  
  // Check the generated HTML report - screenshots will be black
  await browser.close();
})();

5. Open the generated HTML report in a browser
6. Observe that all screenshots in the report are completely black, even though AI correctly identified elements

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions