Skip to content
SiteSwarm Platform Documentation

Observability And Alerting

Observability & Alerting Architecture Specification (OBSERVABILITY_AND_ALERTING.md)

1. Executive Summary & Architectural Overview

The SiteSwarm Observability & Alerting Engine provides unified, end-to-end telemetry emission, continuous edge health evaluation, and automated incident paging across all 10 client applications (apps/*) and central platform service workers (services/*).

To achieve high operational reliability on the Cloudflare Workers platform without introducing heavy third-party agent daemons or prohibitive invocation billing, SiteSwarm implements a 3-Tier Observability Model:

  1. Tier 1 — Synthetic HTTP Health Probes: Active, external-style synthetic probes executing against /api/health endpoints on all client applications every 2 minutes with a strict 2,500ms timeout.
  2. Tier 2 — Workers Analytics Engine (WAE) Domain Metrics: Continuous, zero-PII metric emission directly from worker isolates to the siteswarm_domain_metrics dataset, capturing request latency, HTTP status codes, queue processing events, and Dead Letter Queue (DLQ) escalations.
  3. Tier 3 — Incident Paging & Multi-Channel Dispatch: A decoupled, stateless scheduled worker (services/watchdog; detailed specification in WATCHDOG_PLATFORM_SERVICE.md) that audits health states, persists fleet status in Cloudflare KV (WATCHDOG_STATE), enforces a 15-minute alert debounce window, and pages on-call engineering via BetterStack and Discord/Slack webhooks with 2-run anti-flapping protection.
flowchart TD
    subgraph ClientAndPlatformWorkers["Client Applications (apps/*) & Services (services/*)"]
        A1["Client Worker (apps/bakery)"]
        A2["Client Worker (apps/auto-repair)"]
        A3["Platform Service (services/event-platform)"]
        M["createTelemetryMiddleware\n(packages/telemetry)"]
        A1 --> M
        A2 --> M
    end


    subgraph TelemetryIngress["Telemetry Ingress & Aggregation"]
        M -->|"emitDomainMetric (blob1: appId, blob2: metricName)"| WAE[("Workers Analytics Engine\nsiteswarm_domain_metrics\n(90-day retention)")]
        A3 -->|"dlq_routed events"| WAE
    end


    subgraph WatchdogIsolate["Watchdog Worker (services/watchdog)"]
        Cron["Cron Trigger\n(*/2 * * * *)\n25s Timeout Ceiling"]
        EV["Alert Evaluator\n(evaluator.ts)"]
        Registry["Alert Rule Registry\n(registry.ts & query.ts)"]
        Router["Alert Router\n(router.ts)"]


        Cron --> EV
        Registry --> EV
        EV -->|"1. Synthetic Probes (2500ms)"| Probes["Client /api/health Probes"]
        EV -->|"2. Tuned SQL Query (blob1, blob2)"| WAE
        EV --> Router
    end


    subgraph StateAndDispatch["Persistence & Outbound Notification"]
        KV[("WATCHDOG_STATE (KV)\nMetadata: status, severity, last_breach\nBody: full incident payload")]
        BS["BetterStack On-Call API\n(P1 Phone / SMS / Push)"]
        WH["Discord & Slack Webhooks\n(P1 / P2 Notifications)"]
        CC["Command Center Cockpit\n(/observability single-call list)"]


        Router <-->|"Debounce (15m) & Anti-Flapping (2-Run)"| KV
        Router -->|"P1 Breach"| BS
        Router -->|"P1/P2 Breach & Resolve"| WH
        KV -->|"WATCHDOG_STATE.list({ prefix: 'alert:' })"| CC
    end

2. Empirical Performance Audit (Staging Benchmarks)

Following the rollout of Phase 1 MVP stories (#752, #753, #754, #755, #756), empirical performance measurements were collected from staging cron cycles and synthetic fault injection drills (scripts/drill-alerting.ts).

2.1 Workers Analytics Engine (WAE) SQL API Latency Profile

Queries to the Cloudflare Analytics Engine SQL API (/v4/accounts/{accountId}/analytics_engine/sql) exhibit distinct latency profiles based on timeframe intervals and partition filtering:

Query Type Timeframe Bound Target Filter p50 Latency p90 Latency p99 Latency Execution Safety
Unbounded Full Scan (Anti-Pattern) None (SELECT count()...) Unindexed or generic 480 ms 1,250 ms 3,100 ms ⚠️ High risk of query timeout / billing surge
Time-Bounded Only NOW() - INTERVAL '10' MINUTE blob2 = 'dlq_routed' 115 ms 190 ms 310 ms ✅ Partition bounded, scan confined
Tuned Index + Timeframe (SiteSwarm Standard) NOW() - INTERVAL '10' MINUTE blob1 = appId AND blob2 = metricName 65 ms 95 ms 180 ms 🟢 Optimal index partition pruning

[!IMPORTANT] Full Table Scan Prevention: To ensure deterministic query latency below 200ms within Watchdog’s 25-second global execution ceiling, all WAE queries MUST:

  1. Specify a bounded timeframe via timestamp > NOW() - INTERVAL '<N>' MINUTE (generated via buildWaeAlertQuery).
  2. Filter on indexable blob columns (blob1 = '<appId>' and blob2 = '<metricName>').

2.2 Cloudflare KV Fleet Health Listing Performance

Watchdog persists alert state and exposes fleet-wide health status via GET /status. Staging profiling of WATCHDOG_STATE.list({ prefix: 'alert:' }) confirms:

  • Single-Call Listing Latency:
    • $O(1)$ KV prefix listing: 18 ms - 28 ms (median: 22 ms).
    • Evaluated across 11 registered rules (10 client probes + 1 platform WAE alert).
  • Metadata Embedding Invariant:
    • By embedding AlertMetadataRecord (status, severity, last_breach, updated_at, consecutive_healthy) directly in KV key metadata during recordAlertState(), the Command Center (apps/command-center) and /status endpoint inspect the entire fleet health in a single round-trip.
    • Eliminates $N$ subsequent kv.get() calls, preventing connection pooling stalls and saving 11 individual KV read operations per request.
  • Storage Overhead:
    • Average key metadata size: ~120 bytes.
    • Total fleet metadata footprint: < 2 KB.

3. The 3-Tier Observability Model

SiteSwarm stratifies observability into three distinct tiers to balance real-time alerting immediacy against execution cost and runtime footprint:

flowchart LR
    subgraph Tier1["Tier 1: Synthetic Probes"]
        P1["/api/health Endpoints"]
        P2["2,500ms Strict Timeout"]
        P3["P1/P2 Immediate Paging"]
    end


    subgraph Tier2["Tier 2: WAE Domain Metrics"]
        D1["siteswarm_domain_metrics"]
        D2["Zero-PII Structured Blobs"]
        D3["Trend & Error Spike Detection"]
    end


    subgraph Tier3["Tier 3: Incident Paging & Router"]
        R1["BetterStack Phone/SMS/Push"]
        R2["15m Debounce Suppression"]
        R3["2-Run Anti-Flapping Resolution"]
    end


    Tier1 --> Tier3
    Tier2 --> Tier3

3.1 Tier 1: Synthetic HTTP Health Probes

  • Endpoint: Standardized /api/health across all 10 client applications.
  • Frequency: Every 2 minutes via Watchdog scheduled handler (*/2 * * * *).
  • Timeout: Strictly bounded at 2,500ms using AbortSignal.timeout(2500). If a probe hangs or exceeds 2.5s, the request is aborted cleanly, marking the probe as firing without blocking other app evaluations.
  • Payload Validation: In addition to HTTP 200 status code verification, the response JSON is inspected: status: "ok" or status: "healthy" is required.

3.2 Tier 2: Workers Analytics Engine Domain Metrics

  • Dataset: siteswarm_domain_metrics.
  • Emission Method: Non-blocking emitDomainMetric() from @siteswarm/telemetry wrapped in try/catch so customer requests never fail if the analytics binding is unavailable.
  • Metric Categories:
    • http_request: Inbound customer traffic, HTTP status code (blob3), and execution duration (double2: durationMs).
    • dlq_routed: Dead Letter Queue message routing and archival in @siteswarm/service-event-platform.
    • inquiry_submitted: Client storefront inquiry form submissions.
    • critique_feedback: Client critique and design review submissions.

3.3 Tier 3: Incident Paging & Routing

  • Severity Classification:
    • P1 (Critical Tenant Outage / DLQ Spike): Triggers immediate phone call, SMS, email, and push notification via BetterStack On-Call API, plus Discord and Slack webhooks.
    • P2 (Non-Critical Degraded / Documentation Portal): Dispatches Discord and Slack webhook alerts; does not trigger phone paging.
    • P3 (Informational Warning): Logged to KV and Command Center observability grid.

4. Standard WAE Slot & Indexing Layout

Workers Analytics Engine stores schemaless events using typed arrays (blobs for string data and doubles for floating-point numbers). SiteSwarm enforces a strict monorepo-wide slot assignment convention codified in @siteswarm/telemetry:

Slot Field Name Type Indexing Status Description Example Values
blob1 appId String Indexed Tenant or service identifier "bakery", "auto-repair", "siteswarm-service-event-platform"
blob2 metricName String Indexed Domain metric name "http_request", "dlq_routed", "inquiry_submitted"
blob3 status String Unindexed Status code or execution outcome "200", "500", "archived", "drain_error"
blob4 pathname String Unindexed Sanitized URL path (query strings stripped) "/menu", "/api/inquiries", "/services"
blobs[4+] Supplemental String Unindexed Optional context labels (zero-PII validated) queueName, topic, tenantId
double1 value Float64 Numerical Primary metric count or value (default: 1) 1, 42
double2 durationMs Float64 Numerical Request or operation execution time 14.2, 125.8
doubles[2+] Supplemental Float64 Numerical Additional duration or count metrics dbLatencyMs
index1 index String Partition Index Dataset partition index (defaults to appId) "bakery", "siteswarm-platform"

[!NOTE] In Cloudflare WAE SQL, blob1 through blob20 and double1 through double20 are accessed directly by column name in SQL statements (SELECT count() FROM siteswarm_domain_metrics WHERE blob1 = 'bakery').

5. KV Alert State Machine & Anti-Flapping Lifecycle

To prevent alert fatigue and eliminate on-call noise, the Watchdog Alert Router (services/watchdog/src/dispatch/router.ts) governs state transitions in WATCHDOG_STATE KV with explicit debouncing and anti-flapping guards:

stateDiagram-v2
    [*] --> Healthy: Initial State / Deployment


    Healthy --> Firing: Evaluation Breach Detected\n(Record last_breach & last_notified\nDispatch P1 Paging + Webhooks)


    Firing --> Debounced: Breach Persists & (now - last_notified < 15m)\n(Update updated_at, Suppress Outbound)
    Debounced --> Firing: Breach Persists & (now - last_notified >= 15m)\n(Re-alert Outbound Paging)


    Firing --> HealthyPending: Evaluation Passes (Run 1/2)\n(Status remains FIRING, consecutive_healthy = 1\nZero Outbound Resolution)
    Debounced --> HealthyPending: Evaluation Passes (Run 1/2)


    HealthyPending --> Firing: Breach Re-occurs (Flap Detected)\n(Reset consecutive_healthy = 0)


    HealthyPending --> Healthy: Evaluation Passes (Run 2/2)\n(Status -> HEALTHY, consecutive_healthy = 2\nResolve BetterStack Incident & Webhooks)

5.1 Debounce Window (15 Minutes)

  • When a rule enters firing status, an outbound alert is dispatched immediately, and last_notified is recorded in KV.
  • If subsequent cron runs (every 2 minutes) detect the same ongoing breach, the alert is marked debounced. Outbound phone calls and webhook dispatches are suppressed until 15 minutes (900,000 ms) have elapsed since the last notification.
  • Once 15 minutes elapse without resolution, a reminder alert is dispatched.

5.2 Anti-Flapping Auto-Resolution (2 Consecutive Healthy Runs)

  • If a client application experiences intermittent flapping (e.g. failing 1 probe, recovering for 1 probe, and failing again), a naive alerting system generates a flurry of breach and resolution notifications.
  • SiteSwarm requires 2 consecutive healthy evaluations before transitioning an alert back to healthy:
    1. Run 1 (Healthy Pending): consecutive_healthy transitions to 1. The alert record status remains firing in KV, and no resolution dispatch is triggered.
    2. Run 2 (Fully Resolved): If the second consecutive evaluation is also healthy, consecutive_healthy reaches 2, status transitions to healthy, the BetterStack incident is automatically resolved via POST /api/v2/incidents/:id/resolve, and resolution webhooks are posted to Discord and Slack.

6. Architectural Evaluation: Cloudflare Tail Workers vs. In-Worker Middleware

During early architecture spikes (deferred from #182), two patterns were considered for capturing edge telemetry:

  1. In-Worker Middleware (createTelemetryMiddleware): Directly emitting domain metrics inside the client application isolate.
  2. Cloudflare Tail Workers: Configuring an asynchronous tail_consumers worker that receives execution logs and trace events after worker termination.

6.1 Trade-Off & Evaluation Matrix

Architectural Dimension In-Worker Telemetry Middleware Cloudflare Tail Workers (tail_consumers)
Invocation & Billing Overhead 0 additional worker invocations. Metrics are emitted via native writeDataPoint() within the existing request lifecycle. 1:1 additional worker invocation. A site handling 10M requests generates 10M additional Tail Worker executions, doubling billing invocations.
Execution Latency Impact < 0.1 ms. writeDataPoint is a non-blocking in-memory C++ binding call that buffers data points for batch flushing by the edge runtime. 0 ms on client request. Execution is entirely asynchronous after the client response is returned.
Uncaught Fatal Isolate Crashes ⚠️ Cannot execute if the isolate crashes fatally before/outside the middleware boundary (e.g. startup syntax errors, OOM, CPU limit). 🟢 Captures all events including outcome: "exceededCpu", outcome: "exceededMemory", and outcome: "exception".
Domain Semantics & Business Context 🟢 Rich domain context readily available (authenticated tenant ID, form submission status, specific route metadata). ⚠️ Limited to request headers, console logs, and uncaught exception stack traces; lacks structured domain entities unless serialized in logs.
Configuration Complexity Simple Astro middleware export (createTelemetryMiddleware({ appId })) configured in app code. Requires multi-worker binding configuration in wrangler.jsonc, Tail Worker deployment pipeline, and routing rules.

6.2 Architectural Decision & Recommendation

[!TIP] Adopted Strategy: In-Worker Middleware + Synthetic Probe Safety Net

  • Primary Telemetry: Standardize on in-worker middleware (createTelemetryMiddleware in @siteswarm/telemetry) for all 10 client applications. This provides rich domain telemetry, request duration tracking, and HTTP status logging with zero additional Cloudflare invocation charges.
  • Fatal Crash Defense: Watchdog’s Tier 1 Synthetic Health Probes (/api/health) provide a complete safety net for unhandled edge exceptions or isolate crashes. If a worker runs out of memory or crashes on boot, the /api/health probe fails (returning HTTP 500 or timing out), triggering a P1 incident within <= 2 minutes.
  • Targeted Tail Worker Reservation: Cloudflare Tail Workers are intentionally deferred for standard client apps, but reserved as an opt-in diagnostic tool for complex asynchronous platform pipelines (services/event-platform) where unhandled background queue exceptions require deep post-mortem stack tracing.

7. Zero-PII Compliance & Data Retention Policies

7.1 Zero-PII Living Spec Invariant

All telemetry emitted to Workers Analytics Engine MUST comply with SiteSwarm’s strict privacy rules:

  • Strict Query String Stripping: Client URL query strings MUST NEVER be logged or emitted to WAE. URLs are sanitized at the edge prior to emission: 
    const safePathname = new URL(request.url).pathname; // Strips ?email=..., ?token=..., etc.
  • Prohibited Data Classes:
    • Customer Personally Identifiable Information (PII): Email addresses, telephone numbers, postal addresses, names.
    • Authentication Credentials: API keys, Bearer tokens, JWT tokens, Turnstile response tokens, cookie values.
    • Customer Request Bodies: Form input payloads and serialized JSON bodies.
  • Automated Validation: DomainMetricSchema in @siteswarm/telemetry validates candidate strings against PII_PATTERNS (regex for emails, phone numbers, tokens, and credentials), throwing a validation error before any data point can reach writeDataPoint().

7.2 Data Retention Policies

  • Workers Analytics Engine: Data points in siteswarm_domain_metrics are retained by Cloudflare for 90 days. Older data points are automatically summarized and downsampled with double-precision aggregation.
  • Cloudflare KV (WATCHDOG_STATE):
    • Active alert state records (alert:<rule_id>) persist indefinitely to maintain historical continuity of last_breach and last_notified.
    • Synthetic fault injection drill keys (alert:drill:*) are ephemeral and strictly purged immediately upon drill completion via cleanupDrillKeys() or npx tsx scripts/drill-alerting.ts --cleanup.

8. Decisions Intentionally Deferred

In alignment with SiteSwarm’s Minimal Viable Abstraction (MVA) principles:

  1. Dynamic Machine-Learning Anomaly Detection: Thresholds are currently static (e.g. DLQ spike threshold > 0, health probe timeout > 2,500ms). Dynamic statistical standard-deviation alerting is deferred until aggregate traffic volume reaches statistical significance.
  2. Multi-Vendor Paging Fallback (PagerDuty / Opsgenie): BetterStack On-Call API is currently the single authoritative incident dispatcher for P1 events, backed by Discord and Slack webhooks. Secondary SMS gateway failovers are intentionally deferred.
  3. Public / Tenant-Facing Status Dashboards: Alert state is surfaced internally via SiteSwarm Central Command Center (apps/command-center at /observability). Public customer-facing uptime portals are deferred.
  4. Universal Tail Worker Deployment: As evaluated in Section 6, Tail Workers are excluded from the 10 tenant client applications to avoid doubling invocation costs.