dev-tools-observer

Zero-dependency TypeScript browser SDK that intercepts all outgoing HTTP requests (fetch + XHR), hashes sensitive values, deduplicates similar requests by endpoint shape, and batches structured data to a configurable reporting server. The goal is to automatically map which API endpoints a front-end uses — and in what order — so that end-to-end tests can be generated automatically.

Quick start

import{ObserverSDK}from'dev-tools-observer';ObserverSDK.init({endpoint: 'https://your-ingest-server.com/ingest',orgId: 'my-org',// optional, forwarded as-is to the serverflushInterval: 5000,// ms between automatic flushes, default 5000debug: false,// set true to console.log every captured request});

Call ObserverSDK.destroy() to remove interceptors and stop the flush timer (e.g. in test teardown).

Commands

npm run build # tsup → dist/ (ESM + CJS + IIFE bundles)
npm run observe # launch headed Chrome with SDK injected into every page
npm run serve:dev # start a local ingest server on http://localhost:3737/ingest
npm test# vitest run (jsdom environment)
npm run test:watch # vitest watch mode
npm run test:coverage # v8 coverage report
npm run typecheck # tsc --noEmit (strict, zero errors required)

npm run observe opens a Chrome window and injects the SDK into every page you visit. Captured batches are forwarded to DT_ENDPOINT (default http://localhost:3737/ingest) via a Playwright CDP bridge — no gzip or keepalive issues. Environment variables:

DT_ENDPOINT=http://localhost:3737/ingest # ingest server URL
DT_FLUSH_MS=5000 # ms between automatic flushes
DT_ORG_ID=default # X-Org-Id header sent with every batch

---

Architecture

fetch() / XHR call
│
▼
interceptor-fetch.ts / interceptor-xhr.ts
• blocks analytics domains + own endpoint (anti-recursion)
• reads request method, URL, headers, body
• reads response status, headers, body (via response.clone())
• fires onCapture(RawRequestData) in background void-promise
│
▼
sdk.ts — onCapture()
• parseUrl() → normalizes path params, extracts query keys
• hashQueryParams() → one hash per query param key
• hashRecord() → space-split hash for headers
• hashRecord() → space-split hash for path params
• hashBody() → JSON tree walk / form fields / text space-split
• builds CapturedRequest
│
▼
queue.ts — EventQueue.push()
• deduplicates by (method, domain, normalizedPath, sortedQueryKeyNames, graphqlOp)
• increments count on duplicate
│
▼
(every flushInterval ms, or ObserverSDK.flush())
sender.ts — sendBatch()
• tries navigator.sendBeacon with gzip-compressed Blob
• falls back to originalFetch POST with keepalive:true
• uses originalFetch (saved before interceptor was installed) to bypass interception

---

Source files

File	Responsibility
src/index.ts	Public re-exports
src/sdk.ts	ObserverSDK static class — init / flush / destroy
src/types.ts	All interfaces (SDKConfig, RawRequestData, CapturedRequest, …)
src/blocklist.ts	Analytics domain blocklist + isBlocked()
src/hasher.ts	SHA-256, space-split hashing, JSON tree walker
src/interceptor-fetch.ts	window.fetch monkey-patch
src/interceptor-xhr.ts	XMLHttpRequest.prototype monkey-patch
src/queue.ts	EventQueue with deduplication
src/sender.ts	sendBeacon + fetch fallback, gzip via CompressionStream
src/url-parser.ts	URL decomposition, path parameter normalization, makeDedupeKey()

---

Hashing

All values are hashed with truncated SHA-256 (first 16 hex chars) via crypto.subtle. No raw values leave the browser.

Space-split hashing (headers)

Header values are split on spaces before hashing. "Bearer eyJ..." becomes "<hash(Bearer)> <hash(eyJ...)>". This preserves structural shape (a 2-part Bearer token stays distinguishable from a 1-part API key) without revealing either value. The ingest server can later correlate matching hash segments across endpoints.

Field path in stored data: requestHeaders.authorization.1 refers to the second space-split token of the authorization header.

JSON tree hashing

Request and response bodies parsed as JSON have every leaf value hashed individually. Objects and arrays keep their structure; keys are preserved; only the string/number leaf values are replaced with their hashes. Nulls and booleans are passed through unchanged.

Query param hashing

Each query param key maps to a single hash. Multi-value params (?tag=a&tag=b) are joined with a comma ("a,b") before hashing, producing one hash per key. The stored value is queryParams.tag = hash("a,b").

Path param hashing

Detected ID segments in the URL path are extracted as rawPathParams before hashing. Each one is hashed via hashRecord (which calls hashSpaceSplit — since IDs contain no spaces, this is equivalent to hashToken). Stored as pathParams[':param0'] = hash("123").

---

Path parameter normalization

parseUrl() calls normalizePathSegments() on the URL pathname before returning. Any segment that looks like a database ID is replaced with :param0, :param1, etc.

A segment is considered an ID if it matches any of these patterns:

Pattern	Examples detected
Pure numeric	123, 0, 9999999
UUID v4/v7 (case-insensitive)	550e8400-e29b-41d4-a716-446655440000
MongoDB ObjectId (24 hex chars)	507f1f77bcf86cd799439011
Hex string ≥ 8 chars (lowercase)	deadbeef, cafebabe1234
ULID (26 uppercase Crockford base32 chars)	01ARZ3NDEKTSV4RRFFQ69G5FAV
Base64url ID (≥ 8 chars, [A-Za-z0-9_-], contains both an uppercase letter and a digit)	dQw4w9WgXcQ, usr_2cHqf3kJL9OFjl

Human-readable slugs like my-product, v2, api, 2024-01-15, and plain lowercase words are intentionally not normalized.

The normalized path is what gets stored and used in the deduplication key, so /api/products/123 and /api/products/456 are stored as one endpoint: /api/products/:param0.

The raw extracted values (before hashing) are held in rawPathParams:

rawPathParams = { ':param0': '123' }

---

Deduplication

Two requests are considered the same endpoint shape if they share:

• HTTP method (uppercased)
• Domain (including port if non-standard)
• Normalized path (after ID replacement)
• Sorted query parameter key names (not values)
• GraphQL operationName (if present)

/users?page=1 and /users?page=2 are the same endpoint shape — they deduplicate and increment a counter. /users?page=1 and /users?page=1&sort=name are different shapes (different key sets).

---

What is and is not captured

Captured

• Method, domain, normalized path, query param key names
• Hashed query param values, hashed path param values
• Hashed request and response headers (space-split)
• Hashed request and response bodies (JSON tree walk, form fields, text space-split)
• Response status code
• Timestamp and duration

Blocked (never captured)

• Requests to analytics/tracking domains (Google Analytics, Mixpanel, Segment, Amplitude, Sentry, PostHog, Datadog, HotJar, Intercom, LaunchDarkly, …)
• Requests to the SDK's own reporting endpoint (anti-recursion)

Known limitation

URL strings inside JSON response bodies are hashed as opaque strings. A HATEOAS-style response like { "next": "https://api.example.com/items?cursor=TOKEN" } hashes the entire URL — the cursor=TOKEN inside it is not parsed. This means the system cannot detect the structural edge from that response field to a subsequent request's query string when the caller follows the URL directly. Individual cursor/token values passed via normal JSON fields are detected.

---

CapturedRequest shape

interfaceCapturedRequest{method: string;// "GET", "POST", …protocol: string;// "https"domain: string;// "api.example.com" or "localhost:3000"path: string;// normalized, e.g. "/users/:param0"queryParams: Record<string,string>;// { page: hash("2") }pathParams: Record<string,string>;// { ':param0': hash("123") }requestHeaders: Record<string,string>;// space-split hashed, lowercase keysrequestBody: HashedBody|null;responseStatus: number;responseHeaders: Record<string,string>;responseBody: HashedBody|null;timestamp: number;duration: number;graphqlOperationName?: string;// preserved unhashed}

HashedBody is one of:

{ type: 'json'; data: unknown}// recursive hash tree{ type: 'graphql'; data: unknown; operationName: string|undefined}{ type: 'form'; data: Record<string,string>}// each field hashed{ type: 'text'; data: string}// space-split hashed{ type: 'binary'; data: null}// no hash possible

---

TypeScript gotchas

• @tsconfig/strictest + tsup: tsup's CJS resolver cannot resolve the package by name. All strictest settings are inlined directly in tsconfig.json.
• Header keys are lowercased: always access them as raw.requestHeaders['content-type'], never 'Content-Type'.
• Closure narrowing: TypeScript narrows let variables assigned only inside closures to null. Pattern: use a { current: T | null } ref object, copy to const before calling.
• noUncheckedIndexedAccess: array/record access returns T | undefined. Use ?. and ??.
• exactOptionalPropertyTypes: T | undefined is distinct from an absent optional property.
• XHR tests: do not use vi.stubGlobal('XMLHttpRequest', FakeClass) — jsdom validates instances. Replace prototype methods with vi.fn()before installing the interceptor.
• XMLHttpRequest.DONE: use the literal 4 in source code — static properties may be undefined when the prototype is replaced in tests.