Spike Scout Bug Dedup

Spike: Deterministic Playwright Reproduction Synthesis & Bug Deduplication Pipeline

Executive Summary: This architectural spike investigates how an autonomous exploratory synthetic QA scout session (Epic #762) that encounters a runtime defect (console.error, uncaught pageerror, or HTTP $\ge 400$ response) reliably distills its exploratory session into a minimal, deterministic @playwright/test .spec.ts reproduction script (requiring zero LLM inference at playback), packages diagnostic artifacts (trace.zip, failure screenshot PNG, and HAR 1.2 network log), and files or updates a deduplicated GitHub Issue via a normalized SHA-256 signature (computeBugSignature).

---

1. Context & Problem Statement

1.1 Epic #762 Scope & The Exploratory-to-Deterministic Bridge

Synthetic QA scouts explore staging and preview storefronts using local vision/text models driving real headless Chromium browsers. While LLM-driven exploration excels at discovering broken user flows without brittle pre-written selectors, raw exploratory transcripts present three operational hazards for engineering triage:

1. Non-Deterministic Replay: Re-running a multi-turn local LLM loop to reproduce a bug is slow, non-deterministic, and expensive for human developers debugging locally.
2. Duplicate Issue Spam: Scheduled synthetic scouts running recurring sweeps against a broken staging endpoint will encounter the same defect on every run, flooding the repository issue tracker unless recurring failures are deterministically deduplicated.
3. Vague Bug Reports: Issue descriptions lacking concrete DOM selectors, chronological actions, HTTP response payloads, and downloadable Playwright traces cannot be acted upon by engineers or coding agents.

1.2 Spike Research Questions (#769)

1. Unknown Question: How can an autonomous exploratory agent session that encounters an anomaly reliably synthesize a minimal, deterministic Playwright test script, package reproduction artifacts (trace.zip, HAR, screenshot), and file a deduplicated GitHub issue without spamming duplicate reports?
2. Decision Blocked: Bug reporting pipeline design, artifact storage format, error signature hashing, and issue filing integration.
3. Hard Negative Constraint: No predetermined reporting architecture; test against synthetic injected bugs with real Playwright browser execution and real npx playwright test CLI verification.

---

2. Architecture Overview: End-to-End Bug Reporting Pipeline

flowchart TD
    subgraph Scout_Browser ["1. Exploratory Scout Session (Playwright Chromium)"]
        Page["Playwright Page + Context Tracing"]
        Recorder["ScoutAnomalyListener (Action & Anomaly Recorder)"]
        Page -->|"console.error / pageerror / HTTP >= 400"| Recorder
        Recorder -->|"goto / fill / click / press"| Page
    end

    subgraph Synthesis_Engine ["2. Deterministic Distillation (Zero-LLM Playback)"]
        Trace["AnomalyTraceSession"]
        Pruner["Action Log Pruner (Slice 0..actionIndexAtFailure)"]
        Hasher["computeBugSignature (Volatile Token Normalization + SHA-256)"]
        Codegen["synthesizePlaywrightSpec (.spec.ts Generator)"]
        MdGen["synthesizeReproduction (Non-Vague GitHub Issue Markdown)"]
    end

    subgraph Triage_Pipeline ["3. Deduplicating Issue Tracker"]
        Lookup{"Open Issue Matches SHA-256 Signature?"}
        Update["Increment occurrenceCount + Append Occurrence Comment"]
        Create["Create New GitHub Issue with Embedded .spec.ts & Artifact Manifest"]
    end

    Recorder -->|"finalizeSession()"| Trace
    Trace --> Pruner
    Trace --> Hasher
    Pruner --> Codegen
    Hasher --> Codegen
    Codegen --> MdGen
    MdGen --> Lookup
    Lookup -->|"Yes (Recurring Defect)"| Update
    Lookup -->|"No (First Occurrence)"| Create

---

3. Browser Anomaly Listener & Action Log Specification

Implemented in scripts/scout/dedup/listener.ts (ScoutAnomalyListener), the listener attaches directly to a Playwright Page and BrowserContext:

3.1 Monitored Browser & Network Anomaly Channels

1. Console Errors (console.error):
   • Listener: page.on('console', msg => msg.type() === 'error')
   • Captures application-level error logs along with source script URL, line number, and column offset while filtering redundant browser resource status noise already captured by the HTTP response listener.
2. Uncaught Page Exceptions (pageerror):
   • Listener: page.on('pageerror', err => ...)
   • Captures unhandled runtime exceptions (TypeError, ReferenceError, rejected promises) with full V8 stack traces and binds them to the most recent DOM selector interaction (lastSelector) and chronological step index (actionIndexAtFailure).
3. HTTP Error Responses (http_error):
   • Listener: page.on('response', res => res.status() >= 400)
   • Captures failing API and document responses (4xx / 5xx), recording HTTP method, URL, status code, MIME type, and a 500-character response body snippet into a structured HAR 1.2 log (session.har.json).

3.2 Deterministic Action Log Pruning

Every exploratory interaction (goto, fill, click, press, select, hover) is recorded chronologically before execution. When an anomaly occurs at actionIndexAtFailure, the synthesizer prunes all subsequent exploratory steps (actions.slice(0, actionIndexAtFailure + 1)), yielding a minimal reproduction sequence free of post-failure noise.

---

4. Error Normalization & SHA-256 Deduplication Hash Specification

To prevent volatile runtime tokens from causing hash drift across runs, computeBugSignature({ route, errorType, message, seamOrSelector }) in scripts/scout/dedup/dedup.ts normalizes all inputs prior to hashing.

4.1 Volatile Token Normalization Rules

Token Category	Raw Runtime Example	Normalized Representation	Purpose
Ephemeral Localhost Ports	http://127.0.0.1:61550/checkout	/checkout (or <LOCAL_ORIGIN>)	Prevents random OS ephemeral test ports from altering signatures
Volatile Query Nonces	?nonce=1773000500&_t=9911	Stripped (semantic params retained)	Eliminates cache-busting query timestamps and request tokens
UUIDv4 Identifiers	550e8400-e29b-41d4-a716-446655440000	<UUID>	Normalizes dynamic request IDs, session IDs, and entity UUIDs
Numeric Path IDs	/orders/987654/items/321	/orders/<ID>/items/<ID>	Collapses dynamic database record IDs into canonical route templates
ISO-8601 Timestamps	2026-11-10T22:04:10.123Z	<TIMESTAMP>	Removes wall-clock timestamps embedded in application error logs
UNIX Epoch Timestamps	1773000500123	<EPOCH>	Normalizes millisecond and second epoch counters
Hex Memory Addresses	0x7fff5fbff8c0	<ADDR>	Strips native/WASM memory pointers
Bundle Line:Column Offsets	bundle.js:142:29	bundle.js:<LINE>:<COL>	Stabilizes stack traces across minor bundle minification shifts

4.2 Canonical Preimage & SHA-256 Digest

The canonical preimage is constructed deterministically as:

route:${normalizedRoute}
type:${normalizedErrorType}
message:${normalizedMessage}
seam:${normalizedSeam}

The 64-character lowercase hexadecimal SHA-256 digest (crypto.createHash('sha256').update(canonicalPreimage, 'utf8').digest('hex')) is embedded at the top of every generated GitHub Issue body as an HTML comment marker:

<!-- scout-bug-signature: 79a35942f46c... -->

When a synthetic scout encounters a defect, the deduplicating tracker searches open issues for this marker. If an open issue already exists:

• Zero Duplicate Issues Created: No new issue is opened.
• Occurrence Telemetry Updated: occurrenceCount is incremented, lastSeenAt is updated, and a structured occurrence comment is appended linking the latest artifact bundle.

---

5. Artifact Packaging & Retention Strategy

Each finalized anomaly session produces a self-contained diagnostic bundle (ArtifactBundleManifest):

Artifact File	Format	Empirical Size (Checkout Defect)	Diagnostic Purpose & Retention Policy
trace.zip	Playwright Trace Archive	~28–33 KB	Full DOM snapshots, action timeline, console logs, and network waterfall; inspectable via npx playwright show-trace trace.zip. Retained for 7 days on ephemeral object storage (or attached to issue).
failure-screenshot.png	Full-Page PNG	~15.7 KB	Immediate visual state at the moment of failure for rapid human triage in GitHub Issues. Retained for 7 days via presigned URLs.
session.har.json	HAR 1.2 JSON	~0.9 KB	Structured HTTP request/response status codes, MIME types, and error body snippets for status >= 400 calls. Retained inline in issue markdown and for 7 days in artifact storage.
repro-scout-<sig>.spec.ts	@playwright/test TypeScript	47 lines (~1.5 KB)	Standalone deterministic Playwright test script embedded directly inside the GitHub Issue body (permanent retention in GitHub Issue).

---

6. Empirical Reproducibility Metrics & Two-Tier Verification

6.1 Tier 1: Machine Gate Verification

• Unit Test Suite (scripts/scout/dedup/bug-dedup.test.ts): 11/11 unit tests passing across 5 suites (volatile normalization, SHA-256 signature stability & collision separation, anomaly listener event capture, .spec.ts synthesis, deduplication tracker, and failure-mode guardrails).
• Architecture & Docs Taxonomy (pnpm run lint:arch & pnpm run check): 0 errors.

6.2 Tier 2: Behavioral Runtime Proof (scripts/scout/dedup/verify-bug-dedup.ts)

Executed against a live ephemeral node:http server (GET /checkout, POST /api/orders/submit returning HTTP 500, and GET /healthy) using a real headless Playwright Chromium browser and spawning real npx playwright test child processes:

Step	Scenario	Command / Action / Probe	Expected	Actual	Pass/Fail	Empirical Notes
1	Live Browser Anomaly Capture	ScoutAnomalyListener on headless Chromium /checkout click	pageerror, console.error, http_error	console.error, http_error, pageerror	🟢 PASS	Captured all 3 anomaly classes across 3 chronological actions
2	Diagnostic Artifact Packaging	listener.finalizeSession({ outputDir })	trace.zip, PNG, and HAR $> 0\text{ B}$ on disk	trace=27,838B, png=15,691B, har=871B	🟢 PASS	Verified real Playwright trace archive, screenshot PNG, and HAR 1.2 log on disk
3	Deterministic .spec.ts & Markdown Synthesis	synthesizeReproduction(session1)	Valid .spec.ts + 0 missing markdown sections	Valid (47 lines, sig=79a35942f46c)	🟢 PASS	Pure deterministic codegen with zero LLM invocation
4	Real CLI Reproduction (Assert-Defect Mode)	npx playwright test repro-scout-79a35942f46c.spec.ts	Exit code 0 (1 passed, defect reproduced)	Exit code 0 (1 passed)	🟢 PASS	Generated .spec.ts replayed steps and deterministically reproduced the TypeError
5	Real CLI Regression Gate (Assert-No-Error Mode)	npx playwright test repro-red-gate.spec.ts	Non-zero exit code (fails while bug is unfixed)	Exit code 1	🟢 PASS	Confirmed generated regression test fails against unfixed endpoint
6	Deduplication Pipeline Run #1	tracker.reportTraceSession(session1)	action=created_new, issueNumber=1, count=1	action=created_new, issueNumber=1, count=1	🟢 PASS	Filed initial Issue #1 with signature 79a35942f46c
7	Deduplication Pipeline Run #2 (Duplicate Spam Check)	tracker.reportTraceSession(session2) on recurring defect	action=updated_existing, issueNumber=1, totalIssues=1	action=updated_existing, issueNumber=1, totalIssues=1	🟢 PASS	Recurring defect with new UUID/timestamp/nonce updated Issue #1 (count=2), 0 duplicates created
8	Clean Bug-Free Session Guardrail	tracker.reportTraceSession(session3) on /healthy	reported=false, action=skipped_clean_session, totalIssues=1	reported=false, action=skipped_clean_session, totalIssues=1	🟢 PASS	Clean exploratory session produced zero bug reports and zero false-positive issues

• Empirical Reproducibility Rate: 100% (8/8 probes, 100% deterministic CLI playback).
• Duplicate Issue Spam Rate: 0% (0 duplicate issues across recurring runs).

---

7. Standalone siteswarm-scout Phase 2 Boundary & Mandatory scripts/scout/ Deletion

In accordance with ADR-0006 (docs/write-ups/adr-scout-repository-boundary.md, Issue #766):

1. Standalone Daemon Destination: In Phase 2, the anomaly listener, SHA-256 deduplication engine, and .spec.ts synthesizer prototyped in scripts/scout/dedup/ will be ported into the standalone siteswarm-scout daemon repository under siteswarm-scout/src/reporting/.
2. Mandatory scripts/scout/ Teardown Before Phase 2: All exploratory spike files in scripts/scout/ (including scripts/scout/dedup/, scripts/scout/edge/, scripts/scout/model/, and scripts/scout/topology/) are temporary Phase 1 research artifacts. Prior to Phase 2 production cutover, scripts/scout/ will be completely removed from the SiteSwarm monorepo (git rm -rf scripts/scout/) and its tooling test entries pruned from root package.json after distilling canonical specifications into docs/architecture/SCOUT_RUNNER.md and docs/runbooks/QA_TRIAGE.md.

---

8. Decisions Intentionally Deferred

Per SiteSwarm’s Minimal Viable Abstraction (MVA) invariant, the following concerns are intentionally deferred to Phase 2 implementation in siteswarm-scout:

1. Automated Delta-Debugging / Action Bisection: Currently, synthesizePlaywrightSpec prunes actions occurring after actionIndexAtFailure. Automated binary-search step bisection (iteratively removing intermediate steps to find the strict minimal causal subset on 50+ step journeys) is deferred to Phase 2.
2. Automatic Issue Auto-Closure on Green Verification: Automatically closing open scout-filed GitHub issues when subsequent scheduled sweeps pass is deferred until staging deployment webhook correlation is wired in siteswarm-scout.
3. Cross-Browser Engine Matrix Replay: Phase 1 synthesizes and verifies reproduction scripts against headless Chromium. Cross-browser reproduction sweeps (WebKit / Firefox) are deferred to the dedicated siteswarm-scout runner fleet.