chore(bots): regenerate registry for xAI-SearchBot and LinkupBot#39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cport1 merged 3 commits into from
Aug 25, 2026
Merged

chore(bots): regenerate registry for xAI-SearchBot and LinkupBot #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f059bb0
feat(sdk): OpenTelemetry spans, without depending on OpenTelemetry
cport1 Aug 22, 2026
dd6b897
chore(bots): regenerate registry for xAI-SearchBot and LinkupBot
cport1 Aug 25, 2026
bed3035
chore(bots): regenerate registry for the five newly identified crawlers
cport1 Aug 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **OpenTelemetry spans** around `protect()` and rule evaluation. Pass a tracer: `new WebDecoy({ tracer: trace.getTracer('webdecoy') })`. Injected rather than imported, so the package stays dependency-free and edge-safe — the `Tracer` type is a structural subset of OpenTelemetry's, so `trace.getTracer()` works with no adapter, and omitting it means no spans, no dependency and no behaviour change. Attributes cover the decision id (which joins a span to its dashboard row), the conclusion, the deciding rule, and whether the request cost a round trip to ingest. A tracer that throws cannot fail a request.

### Changed

- **One adapter core.** Express, Fastify, Next.js (middleware and Pages wrapper) and the fetch guard each carried their own copy of skip-path matching, the 429 and 403 payloads, and honeytoken arming — five copies of one set of decisions, and five places the next correction can fail to land. They now share `adapter-core.ts`; the framework-specific response mechanics are untouched, and every honeytoken-injection test passes unchanged. Fastify keeps its awaited arming, which has no window where early requests are served without the link.

## [0.13.0] - 2026-08-22

### Added
Expand Down
25 changes: 25 additions & 0 deletions README.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -512,6 +512,31 @@ new WebDecoy({ logger: fromPino(pino()) }); // pino's argument order is reverse
`fromPino()` exists because passing a pino instance directly type-checks and then
silently drops every structured field.

## Tracing

Pass an OpenTelemetry tracer and `protect()` emits a span, with a child span for
rule evaluation:

```typescript
import { trace } from '@opentelemetry/api';

new WebDecoy({ tracer: trace.getTracer('webdecoy') });
```

The tracer is **injected, not imported** — this package has no dependencies and
runs on Workers and Vercel Edge, where a stray transitive import is expensive.
The `Tracer` type is a structural subset of OpenTelemetry's, so
`trace.getTracer()` satisfies it with no adapter. Omit it and there are no spans,
no dependency, and no behaviour change.

Attributes are the questions an operator actually asks: `decision.id` (which
joins the span to the dashboard row), `decision.conclusion`, `decision.rule`,
`rules.evaluated`, and `webdecoy.remote` — whether the request cost a round trip
to ingest or was settled locally.

A tracer that throws cannot fail a request. Observability that can take the
request path down is worse than none.

## Examples

See [examples](./examples) for complete working setups — e.g. [express-basic](./examples/express-basic).
Expand Down
2 changes: 1 addition & 1 deletion packages/webdecoy/src/bots/bots.test.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ describe('matchUserAgent', () => {

describe('registry integrity', () => {
it('carries the full generated table', () => {
expect(BOT_REGISTRY.length).toBe(169);
expect(BOT_REGISTRY.length).toBe(175);
expect(BOT_CATEGORIES).toContain('training_crawler');
expect(BOT_CATEGORIES).toContain('search_crawler');
});
Expand Down
72 changes: 72 additions & 0 deletions packages/webdecoy/src/bots/parity-vectors.generated.json
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -575,6 +575,18 @@
"category": "seo_crawler",
"matched": true
},
{
"userAgent": "duckassistbot",
"id": "duckassistbot",
"category": "ai_search_crawler",
"matched": true
},
{
"userAgent": "Mozilla/5.0 (compatible; duckassistbot/1.0; +http://example.com/bot)",
"id": "duckassistbot",
"category": "ai_search_crawler",
"matched": true
},
{
"userAgent": "duckduckbot",
"id": "duckduckbot",
Expand DownExpand Up@@ -935,6 +947,18 @@
"category": "training_crawler",
"matched": true
},
{
"userAgent": "iboubot",
"id": "iboubot",
"category": "search_crawler",
"matched": true
},
{
"userAgent": "Mozilla/5.0 (compatible; iboubot/1.0; +http://example.com/bot)",
"id": "iboubot",
"category": "search_crawler",
"matched": true
},
{
"userAgent": "iframely",
"id": "iframely",
Expand DownExpand Up@@ -1055,6 +1079,18 @@
"category": "fetcher",
"matched": true
},
{
"userAgent": "linkupbot",
"id": "linkupbot",
"category": "ai_search_crawler",
"matched": true
},
{
"userAgent": "Mozilla/5.0 (compatible; linkupbot/1.0; +http://example.com/bot)",
"id": "linkupbot",
"category": "ai_search_crawler",
"matched": true
},
{
"userAgent": "mj12bot",
"id": "mj12bot",
Expand DownExpand Up@@ -1559,6 +1595,18 @@
"category": "fetcher",
"matched": true
},
{
"userAgent": "reflectionbot",
"id": "reflectionbot",
"category": "training_crawler",
"matched": true
},
{
"userAgent": "Mozilla/5.0 (compatible; reflectionbot/1.0; +http://example.com/bot)",
"id": "reflectionbot",
"category": "training_crawler",
"matched": true
},
{
"userAgent": "rogerbot",
"id": "rogerbot",
Expand DownExpand Up@@ -1787,6 +1835,18 @@
"category": "fetcher",
"matched": true
},
{
"userAgent": "sofyabot",
"id": "sofyabot",
"category": "ai_search_crawler",
"matched": true
},
{
"userAgent": "Mozilla/5.0 (compatible; sofyabot/1.0; +http://example.com/bot)",
"id": "sofyabot",
"category": "ai_search_crawler",
"matched": true
},
{
"userAgent": "sogou",
"id": "sogou",
Expand DownExpand Up@@ -2147,6 +2207,18 @@
"category": "generic_scraper",
"matched": true
},
{
"userAgent": "xai-searchbot",
"id": "xai-searchbot",
"category": "ai_search_crawler",
"matched": true
},
{
"userAgent": "Mozilla/5.0 (compatible; xai-searchbot/1.0; +http://example.com/bot)",
"id": "xai-searchbot",
"category": "ai_search_crawler",
"matched": true
},
{
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
"matched": false
Expand Down
8 changes: 7 additions & 1 deletion packages/webdecoy/src/bots/registry.generated.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@
* agents share User-Agent substrings, so sorting this array changes how real
* traffic is classified.
*
* 169 agents across 15 categories.
* 175 agents across 15 categories.
*/

/**
Expand DownExpand Up@@ -75,6 +75,7 @@ export const BOT_CATEGORIES: readonly BotCategory[] = [

/** The agent table, in match order. */
export const BOT_REGISTRY: readonly BotAgent[] = [
{ id: "reflectionbot", name: "Reflectionbot", category: "training_crawler", organization: "Reflection AI", baseScore: 70, respectsRobots: true, uaPatterns: ["reflectionbot"] },
{ id: "gptbot", name: "GPTBot", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["gptbot"] },
{ id: "chatgpt-user", name: "ChatGPT-User", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["chatgpt-user", "chatgpt"] },
{ id: "oai-searchbot", name: "OAI-SearchBot", category: "training_crawler", organization: "OpenAI", baseScore: 80, respectsRobots: true, uaPatterns: ["oai-searchbot"] },
Expand DownExpand Up@@ -103,6 +104,10 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
{ id: "velenpublicwebcrawler", name: "VelenPublicWebCrawler", category: "training_crawler", organization: "Velen", baseScore: 70, respectsRobots: false, uaPatterns: ["velenpublicwebcrawler"] },
{ id: "isscyberriskcrawler", name: "ISSCyberRiskCrawler", category: "training_crawler", organization: "ISS", baseScore: 65, respectsRobots: false, uaPatterns: ["isscyberriskcrawler"] },
{ id: "friendlycrawler", name: "FriendlyCrawler", category: "training_crawler", organization: "Unknown", baseScore: 65, respectsRobots: true, uaPatterns: ["friendlycrawler"] },
{ id: "duckassistbot", name: "DuckAssistBot", category: "ai_search_crawler", organization: "DuckDuckGo", baseScore: 65, respectsRobots: true, uaPatterns: ["duckassistbot"] },
{ id: "sofyabot", name: "SofyaBot", category: "ai_search_crawler", organization: "Sofya", baseScore: 65, respectsRobots: true, uaPatterns: ["sofyabot"] },
{ id: "xai-searchbot", name: "xAI-SearchBot", category: "ai_search_crawler", organization: "xAI", baseScore: 70, respectsRobots: true, uaPatterns: ["xai-searchbot"] },
{ id: "linkupbot", name: "LinkupBot", category: "ai_search_crawler", organization: "Linkup", baseScore: 65, respectsRobots: true, uaPatterns: ["linkupbot"] },
{ id: "searchgpt", name: "SearchGPT", category: "ai_search_crawler", organization: "OpenAI", baseScore: 75, respectsRobots: true, uaPatterns: ["searchgpt"] },
{ id: "phind", name: "Phind", category: "ai_search_crawler", organization: "Phind", baseScore: 70, respectsRobots: true, uaPatterns: ["phind"] },
{ id: "kagi", name: "Kagi", category: "ai_search_crawler", organization: "Kagi", baseScore: 65, respectsRobots: true, uaPatterns: ["kagi"] },
Expand All@@ -119,6 +124,7 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
{ id: "browser-use", name: "Browser-Use", category: "ai_agent", organization: "Open Source", baseScore: 70, respectsRobots: false, uaPatterns: ["browser-use"] },
{ id: "stagehand", name: "Stagehand", category: "ai_agent", organization: "Browserbase", baseScore: 70, respectsRobots: false, uaPatterns: ["stagehand"] },
{ id: "multion", name: "MultiOn", category: "ai_agent", organization: "MultiOn", baseScore: 70, respectsRobots: false, uaPatterns: ["multion"] },
{ id: "iboubot", name: "IbouBot", category: "search_crawler", organization: "Ibou", baseScore: 30, respectsRobots: true, uaPatterns: ["iboubot"] },
{ id: "googlebot", name: "Googlebot", category: "search_crawler", organization: "Google", baseScore: 30, respectsRobots: true, uaPatterns: ["googlebot"] },
{ id: "bingbot", name: "Bingbot", category: "search_crawler", organization: "Microsoft", baseScore: 30, respectsRobots: true, uaPatterns: ["bingbot"] },
{ id: "yandexbot", name: "YandexBot", category: "search_crawler", organization: "Yandex", baseScore: 35, respectsRobots: true, uaPatterns: ["yandexbot"] },
Expand Down
1 change: 1 addition & 0 deletions packages/webdecoy/src/index.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -84,6 +84,7 @@ export type { BotVerdict, BotAgent, BotCategory } from './bots';
// runtime with a fetch handler. `@webdecoy/hono` is a thin wrapper over it; Bun,
// Deno, Astro and Nitro need no package at all.
export { consoleLogger, silentLogger, fromPino } from './logger';
export type { Tracer, Span } from './tracing';
export type { Logger, LogFields } from './logger';

// Browser signals, joined to the requests that follow them. See client-signals.ts
Expand Down
51 changes: 49 additions & 2 deletions packages/webdecoy/src/sdk.ts
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -13,6 +13,8 @@ import { AgentVerifier } from './agent/verifier';
import type { AgentRequestInput, AgentVerdict } from './agent/types';
import { readEdgeVerdict } from './edge';
import { resolveLogger } from './logger';
import { startSpan, setAttribute, recordError, endSpan } from './tracing';
import type { Tracer } from './tracing';
import type { Logger } from './logger';
import { Decision, newDecisionId } from './decision';
import type { Conclusion } from './decision';
Expand All@@ -33,7 +35,7 @@ export class WebDecoy {
private client: WebDecoyClient | null;
private config: Omit<
Required<WebDecoyConfig>,
'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger'
'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger' | 'tracer'
> & {
apiKey?: string;
};
Expand All@@ -48,6 +50,8 @@ export class WebDecoy {
private readonly characteristics: readonly import('./characteristics').Characteristic[];
/** Where diagnostics go. Never console directly — see logger.ts. */
readonly log: Logger;
/** Optional OpenTelemetry tracer. Absent means no spans and no cost. */
private readonly tracer?: Tracer;
private readonly decisionCache: DecisionCache | null;

constructor(config: WebDecoyConfig) {
Expand All@@ -74,6 +78,7 @@ export class WebDecoy {
};

this.log = resolveLogger(config.logger, this.config.debug);
this.tracer = config.tracer;

// Initialize API client only when apiKey is provided
if (hasApiKey) {
Expand DownExpand Up@@ -253,7 +258,15 @@ export class WebDecoy {
/** Evaluate rules against a prepared context and report any violations. */
private runRules(context: RuleContext): RuleEngineResult | null {
if (!this.ruleEngine) return null;

const span = startSpan(this.tracer, 'webdecoy.rules');
const result = this.ruleEngine.evaluate(context);
setAttribute(span, 'webdecoy.rules.action', result.action);
setAttribute(span, 'webdecoy.rules.evaluated', result.results.length);
setAttribute(span, 'webdecoy.rules.violations', result.violations.length);
if (result.rule) setAttribute(span, 'webdecoy.rules.deciding', result.rule);
endSpan(span);

if (result.violations.length > 0 && this.violationReporter) {
this.violationReporter.report(result.violations);
}
Expand DownExpand Up@@ -335,7 +348,41 @@ export class WebDecoy {
// present on every outcome — and a per-return copy is a line someone would
// eventually forget on the branch that mattered.
const edge = readEdgeVerdict(metadata.headers);
return (await this.decide(metadata, options)).withEdge(edge);

const span = startSpan(this.tracer, 'webdecoy.protect');
try {
const decision = (await this.decide(metadata, options)).withEdge(edge);

// Attributes chosen so a trace answers the questions an operator actually
// asks: what did we decide, which rule decided it, and did this request
// cost a round trip to ingest. The decision id joins the span to the
// dashboard row.
setAttribute(span, 'webdecoy.decision.id', decision.id);
setAttribute(span, 'webdecoy.decision.conclusion', decision.conclusion);
setAttribute(span, 'webdecoy.decision.allowed', decision.allowed);
setAttribute(span, 'webdecoy.rules.evaluated', decision.results.length);
if (decision.ruleResult?.rule) {
setAttribute(span, 'webdecoy.decision.rule', decision.ruleResult.rule);
}
// A detection id that is not the decision id means the verdict came back
// from ingest rather than being settled locally.
setAttribute(
span,
'webdecoy.remote',
decision.detection.detection_id !== decision.id,
);
if (decision.error) {
setAttribute(span, 'webdecoy.error', decision.error);
}
return decision;
} catch (error) {
// decide() fails open rather than throwing, so this is a bug rather than
// a bad day — worth marking on the span rather than swallowing.
recordError(span, error);
throw error;
} finally {
endSpan(span);
}
}

private async decide(
Expand Down
Loading
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
chore(bots): regenerate registry for xAI-SearchBot and LinkupBot#39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

cport1
merged 3 commits into

from

Aug 25, 2026
 Merged


chore(bots): regenerate registry for xAI-SearchBot and LinkupBot
#39
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
f059bb0

feat(sdk): OpenTelemetry spans, without depending on OpenTelemetry


cport1 Aug 22, 2026
dd6b897

chore(bots): regenerate registry for xAI-SearchBot and LinkupBot


cport1 Aug 25, 2026
bed3035

chore(bots): regenerate registry for the five newly identified crawlers


cport1 Aug 25, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


8 changes: 8 additions & 0 deletions
CHANGELOG.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **OpenTelemetry spans** around `protect()` and rule evaluation. Pass a tracer: `new WebDecoy({ tracer: trace.getTracer('webdecoy') })`. Injected rather than imported, so the package stays dependency-free and edge-safe — the `Tracer` type is a structural subset of OpenTelemetry's, so `trace.getTracer()` works with no adapter, and omitting it means no spans, no dependency and no behaviour change. Attributes cover the decision id (which joins a span to its dashboard row), the conclusion, the deciding rule, and whether the request cost a round trip to ingest. A tracer that throws cannot fail a request.

### Changed

- **One adapter core.** Express, Fastify, Next.js (middleware and Pages wrapper) and the fetch guard each carried their own copy of skip-path matching, the 429 and 403 payloads, and honeytoken arming — five copies of one set of decisions, and five places the next correction can fail to land. They now share `adapter-core.ts`; the framework-specific response mechanics are untouched, and every honeytoken-injection test passes unchanged. Fastify keeps its awaited arming, which has no window where early requests are served without the link.

## [0.13.0] - 2026-08-22

### Added
Expand Down

25 changes: 25 additions & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -512,6 +512,31 @@ new WebDecoy({ logger: fromPino(pino()) }); // pino's argument order is reverse
`fromPino()` exists because passing a pino instance directly type-checks and then
silently drops every structured field.

## Tracing

Pass an OpenTelemetry tracer and `protect()` emits a span, with a child span for
rule evaluation:

```typescript
import { trace } from '@opentelemetry/api';

new WebDecoy({ tracer: trace.getTracer('webdecoy') });
```

The tracer is **injected, not imported** — this package has no dependencies and
runs on Workers and Vercel Edge, where a stray transitive import is expensive.
The `Tracer` type is a structural subset of OpenTelemetry's, so
`trace.getTracer()` satisfies it with no adapter. Omit it and there are no spans,
no dependency, and no behaviour change.

Attributes are the questions an operator actually asks: `decision.id` (which
joins the span to the dashboard row), `decision.conclusion`, `decision.rule`,
`rules.evaluated`, and `webdecoy.remote` — whether the request cost a round trip
to ingest or was settled locally.

A tracer that throws cannot fail a request. Observability that can take the
request path down is worse than none.

## Examples

See [examples](./examples) for complete working setups — e.g. [express-basic](./examples/express-basic).
Expand Down

2 changes: 1 addition & 1 deletion
packages/webdecoy/src/bots/bots.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ describe('matchUserAgent', () => {

describe('registry integrity', () => {
 it('carries the full generated table', () => {
 expect(BOT_REGISTRY.length).toBe(169);
 expect(BOT_REGISTRY.length).toBe(175);
 expect(BOT_CATEGORIES).toContain('training_crawler');
 expect(BOT_CATEGORIES).toContain('search_crawler');
 });
Expand Down

72 changes: 72 additions & 0 deletions
packages/webdecoy/src/bots/parity-vectors.generated.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -575,6 +575,18 @@
 "category": "seo_crawler",
 "matched": true
 },
 {
 "userAgent": "duckassistbot",
 "id": "duckassistbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; duckassistbot/1.0; +http://example.com/bot)",
 "id": "duckassistbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "duckduckbot",
 "id": "duckduckbot",
Expand DownExpand Up@@ -935,6 +947,18 @@
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "iboubot",
 "id": "iboubot",
 "category": "search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; iboubot/1.0; +http://example.com/bot)",
 "id": "iboubot",
 "category": "search_crawler",
 "matched": true
 },
 {
 "userAgent": "iframely",
 "id": "iframely",
Expand DownExpand Up@@ -1055,6 +1079,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "linkupbot",
 "id": "linkupbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; linkupbot/1.0; +http://example.com/bot)",
 "id": "linkupbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "mj12bot",
 "id": "mj12bot",
Expand DownExpand Up@@ -1559,6 +1595,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "reflectionbot",
 "id": "reflectionbot",
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; reflectionbot/1.0; +http://example.com/bot)",
 "id": "reflectionbot",
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "rogerbot",
 "id": "rogerbot",
Expand DownExpand Up@@ -1787,6 +1835,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "sofyabot",
 "id": "sofyabot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; sofyabot/1.0; +http://example.com/bot)",
 "id": "sofyabot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "sogou",
 "id": "sogou",
Expand DownExpand Up@@ -2147,6 +2207,18 @@
 "category": "generic_scraper",
 "matched": true
 },
 {
 "userAgent": "xai-searchbot",
 "id": "xai-searchbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; xai-searchbot/1.0; +http://example.com/bot)",
 "id": "xai-searchbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
 "matched": false
Expand Down

8 changes: 7 additions & 1 deletion
packages/webdecoy/src/bots/registry.generated.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@
 * agents share User-Agent substrings, so sorting this array changes how real
 * traffic is classified.
 *
 * 169 agents across 15 categories.
 * 175 agents across 15 categories.
 */

/**
Expand DownExpand Up@@ -75,6 +75,7 @@ export const BOT_CATEGORIES: readonly BotCategory[] = [

/** The agent table, in match order. */
export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "reflectionbot", name: "Reflectionbot", category: "training_crawler", organization: "Reflection AI", baseScore: 70, respectsRobots: true, uaPatterns: ["reflectionbot"] },
 { id: "gptbot", name: "GPTBot", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["gptbot"] },
 { id: "chatgpt-user", name: "ChatGPT-User", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["chatgpt-user", "chatgpt"] },
 { id: "oai-searchbot", name: "OAI-SearchBot", category: "training_crawler", organization: "OpenAI", baseScore: 80, respectsRobots: true, uaPatterns: ["oai-searchbot"] },
Expand DownExpand Up@@ -103,6 +104,10 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "velenpublicwebcrawler", name: "VelenPublicWebCrawler", category: "training_crawler", organization: "Velen", baseScore: 70, respectsRobots: false, uaPatterns: ["velenpublicwebcrawler"] },
 { id: "isscyberriskcrawler", name: "ISSCyberRiskCrawler", category: "training_crawler", organization: "ISS", baseScore: 65, respectsRobots: false, uaPatterns: ["isscyberriskcrawler"] },
 { id: "friendlycrawler", name: "FriendlyCrawler", category: "training_crawler", organization: "Unknown", baseScore: 65, respectsRobots: true, uaPatterns: ["friendlycrawler"] },
 { id: "duckassistbot", name: "DuckAssistBot", category: "ai_search_crawler", organization: "DuckDuckGo", baseScore: 65, respectsRobots: true, uaPatterns: ["duckassistbot"] },
 { id: "sofyabot", name: "SofyaBot", category: "ai_search_crawler", organization: "Sofya", baseScore: 65, respectsRobots: true, uaPatterns: ["sofyabot"] },
 { id: "xai-searchbot", name: "xAI-SearchBot", category: "ai_search_crawler", organization: "xAI", baseScore: 70, respectsRobots: true, uaPatterns: ["xai-searchbot"] },
 { id: "linkupbot", name: "LinkupBot", category: "ai_search_crawler", organization: "Linkup", baseScore: 65, respectsRobots: true, uaPatterns: ["linkupbot"] },
 { id: "searchgpt", name: "SearchGPT", category: "ai_search_crawler", organization: "OpenAI", baseScore: 75, respectsRobots: true, uaPatterns: ["searchgpt"] },
 { id: "phind", name: "Phind", category: "ai_search_crawler", organization: "Phind", baseScore: 70, respectsRobots: true, uaPatterns: ["phind"] },
 { id: "kagi", name: "Kagi", category: "ai_search_crawler", organization: "Kagi", baseScore: 65, respectsRobots: true, uaPatterns: ["kagi"] },
Expand All@@ -119,6 +124,7 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "browser-use", name: "Browser-Use", category: "ai_agent", organization: "Open Source", baseScore: 70, respectsRobots: false, uaPatterns: ["browser-use"] },
 { id: "stagehand", name: "Stagehand", category: "ai_agent", organization: "Browserbase", baseScore: 70, respectsRobots: false, uaPatterns: ["stagehand"] },
 { id: "multion", name: "MultiOn", category: "ai_agent", organization: "MultiOn", baseScore: 70, respectsRobots: false, uaPatterns: ["multion"] },
 { id: "iboubot", name: "IbouBot", category: "search_crawler", organization: "Ibou", baseScore: 30, respectsRobots: true, uaPatterns: ["iboubot"] },
 { id: "googlebot", name: "Googlebot", category: "search_crawler", organization: "Google", baseScore: 30, respectsRobots: true, uaPatterns: ["googlebot"] },
 { id: "bingbot", name: "Bingbot", category: "search_crawler", organization: "Microsoft", baseScore: 30, respectsRobots: true, uaPatterns: ["bingbot"] },
 { id: "yandexbot", name: "YandexBot", category: "search_crawler", organization: "Yandex", baseScore: 35, respectsRobots: true, uaPatterns: ["yandexbot"] },
Expand Down

1 change: 1 addition & 0 deletions
packages/webdecoy/src/index.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -84,6 +84,7 @@ export type { BotVerdict, BotAgent, BotCategory } from './bots';
// runtime with a fetch handler. `@webdecoy/hono` is a thin wrapper over it; Bun,
// Deno, Astro and Nitro need no package at all.
export { consoleLogger, silentLogger, fromPino } from './logger';
export type { Tracer, Span } from './tracing';
export type { Logger, LogFields } from './logger';

// Browser signals, joined to the requests that follow them. See client-signals.ts
Expand Down

51 changes: 49 additions & 2 deletions
packages/webdecoy/src/sdk.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -13,6 +13,8 @@ import { AgentVerifier } from './agent/verifier';
import type { AgentRequestInput, AgentVerdict } from './agent/types';
import { readEdgeVerdict } from './edge';
import { resolveLogger } from './logger';
import { startSpan, setAttribute, recordError, endSpan } from './tracing';
import type { Tracer } from './tracing';
import type { Logger } from './logger';
import { Decision, newDecisionId } from './decision';
import type { Conclusion } from './decision';
Expand All@@ -33,7 +35,7 @@ export class WebDecoy {
 private client: WebDecoyClient | null;
 private config: Omit<
 Required<WebDecoyConfig>,
 'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger'
 'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger' | 'tracer'
 > & {
 apiKey?: string;
 };
Expand All@@ -48,6 +50,8 @@ export class WebDecoy {
 private readonly characteristics: readonly import('./characteristics').Characteristic[];
 /** Where diagnostics go. Never console directly — see logger.ts. */
 readonly log: Logger;
 /** Optional OpenTelemetry tracer. Absent means no spans and no cost. */
 private readonly tracer?: Tracer;
 private readonly decisionCache: DecisionCache | null;

 constructor(config: WebDecoyConfig) {
Expand All@@ -74,6 +78,7 @@ export class WebDecoy {
 };

 this.log = resolveLogger(config.logger, this.config.debug);
 this.tracer = config.tracer;

 // Initialize API client only when apiKey is provided
 if (hasApiKey) {
Expand DownExpand Up@@ -253,7 +258,15 @@ export class WebDecoy {
 /** Evaluate rules against a prepared context and report any violations. */
 private runRules(context: RuleContext): RuleEngineResult | null {
 if (!this.ruleEngine) return null;

 const span = startSpan(this.tracer, 'webdecoy.rules');
 const result = this.ruleEngine.evaluate(context);
 setAttribute(span, 'webdecoy.rules.action', result.action);
 setAttribute(span, 'webdecoy.rules.evaluated', result.results.length);
 setAttribute(span, 'webdecoy.rules.violations', result.violations.length);
 if (result.rule) setAttribute(span, 'webdecoy.rules.deciding', result.rule);
 endSpan(span);

 if (result.violations.length > 0 && this.violationReporter) {
 this.violationReporter.report(result.violations);
 }
Expand DownExpand Up@@ -335,7 +348,41 @@ export class WebDecoy {
 // present on every outcome — and a per-return copy is a line someone would
 // eventually forget on the branch that mattered.
 const edge = readEdgeVerdict(metadata.headers);
 return (await this.decide(metadata, options)).withEdge(edge);

 const span = startSpan(this.tracer, 'webdecoy.protect');
 try {
 const decision = (await this.decide(metadata, options)).withEdge(edge);

 // Attributes chosen so a trace answers the questions an operator actually
 // asks: what did we decide, which rule decided it, and did this request
 // cost a round trip to ingest. The decision id joins the span to the
 // dashboard row.
 setAttribute(span, 'webdecoy.decision.id', decision.id);
 setAttribute(span, 'webdecoy.decision.conclusion', decision.conclusion);
 setAttribute(span, 'webdecoy.decision.allowed', decision.allowed);
 setAttribute(span, 'webdecoy.rules.evaluated', decision.results.length);
 if (decision.ruleResult?.rule) {
 setAttribute(span, 'webdecoy.decision.rule', decision.ruleResult.rule);
 }
 // A detection id that is not the decision id means the verdict came back
 // from ingest rather than being settled locally.
 setAttribute(
 span,
 'webdecoy.remote',
 decision.detection.detection_id !== decision.id,
 );
 if (decision.error) {
 setAttribute(span, 'webdecoy.error', decision.error);
 }
 return decision;
 } catch (error) {
 // decide() fails open rather than throwing, so this is a bug rather than
 // a bad day — worth marking on the span rather than swallowing.
 recordError(span, error);
 throw error;
 } finally {
 endSpan(span);
 }
 }

 private async decide(
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
chore(bots): regenerate registry for xAI-SearchBot and LinkupBot#39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

cport1
merged 3 commits into

from

Aug 25, 2026
 Merged


chore(bots): regenerate registry for xAI-SearchBot and LinkupBot
#39
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
f059bb0

feat(sdk): OpenTelemetry spans, without depending on OpenTelemetry


cport1 Aug 22, 2026
dd6b897

chore(bots): regenerate registry for xAI-SearchBot and LinkupBot


cport1 Aug 25, 2026
bed3035

chore(bots): regenerate registry for the five newly identified crawlers


cport1 Aug 25, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


8 changes: 8 additions & 0 deletions
CHANGELOG.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **OpenTelemetry spans** around `protect()` and rule evaluation. Pass a tracer: `new WebDecoy({ tracer: trace.getTracer('webdecoy') })`. Injected rather than imported, so the package stays dependency-free and edge-safe — the `Tracer` type is a structural subset of OpenTelemetry's, so `trace.getTracer()` works with no adapter, and omitting it means no spans, no dependency and no behaviour change. Attributes cover the decision id (which joins a span to its dashboard row), the conclusion, the deciding rule, and whether the request cost a round trip to ingest. A tracer that throws cannot fail a request.

### Changed

- **One adapter core.** Express, Fastify, Next.js (middleware and Pages wrapper) and the fetch guard each carried their own copy of skip-path matching, the 429 and 403 payloads, and honeytoken arming — five copies of one set of decisions, and five places the next correction can fail to land. They now share `adapter-core.ts`; the framework-specific response mechanics are untouched, and every honeytoken-injection test passes unchanged. Fastify keeps its awaited arming, which has no window where early requests are served without the link.

## [0.13.0] - 2026-08-22

### Added
Expand Down

25 changes: 25 additions & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -512,6 +512,31 @@ new WebDecoy({ logger: fromPino(pino()) }); // pino's argument order is reverse
`fromPino()` exists because passing a pino instance directly type-checks and then
silently drops every structured field.

## Tracing

Pass an OpenTelemetry tracer and `protect()` emits a span, with a child span for
rule evaluation:

```typescript
import { trace } from '@opentelemetry/api';

new WebDecoy({ tracer: trace.getTracer('webdecoy') });
```

The tracer is **injected, not imported** — this package has no dependencies and
runs on Workers and Vercel Edge, where a stray transitive import is expensive.
The `Tracer` type is a structural subset of OpenTelemetry's, so
`trace.getTracer()` satisfies it with no adapter. Omit it and there are no spans,
no dependency, and no behaviour change.

Attributes are the questions an operator actually asks: `decision.id` (which
joins the span to the dashboard row), `decision.conclusion`, `decision.rule`,
`rules.evaluated`, and `webdecoy.remote` — whether the request cost a round trip
to ingest or was settled locally.

A tracer that throws cannot fail a request. Observability that can take the
request path down is worse than none.

## Examples

See [examples](./examples) for complete working setups — e.g. [express-basic](./examples/express-basic).
Expand Down

2 changes: 1 addition & 1 deletion
packages/webdecoy/src/bots/bots.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ describe('matchUserAgent', () => {

describe('registry integrity', () => {
 it('carries the full generated table', () => {
 expect(BOT_REGISTRY.length).toBe(169);
 expect(BOT_REGISTRY.length).toBe(175);
 expect(BOT_CATEGORIES).toContain('training_crawler');
 expect(BOT_CATEGORIES).toContain('search_crawler');
 });
Expand Down

72 changes: 72 additions & 0 deletions
packages/webdecoy/src/bots/parity-vectors.generated.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -575,6 +575,18 @@
 "category": "seo_crawler",
 "matched": true
 },
 {
 "userAgent": "duckassistbot",
 "id": "duckassistbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; duckassistbot/1.0; +http://example.com/bot)",
 "id": "duckassistbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "duckduckbot",
 "id": "duckduckbot",
Expand DownExpand Up@@ -935,6 +947,18 @@
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "iboubot",
 "id": "iboubot",
 "category": "search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; iboubot/1.0; +http://example.com/bot)",
 "id": "iboubot",
 "category": "search_crawler",
 "matched": true
 },
 {
 "userAgent": "iframely",
 "id": "iframely",
Expand DownExpand Up@@ -1055,6 +1079,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "linkupbot",
 "id": "linkupbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; linkupbot/1.0; +http://example.com/bot)",
 "id": "linkupbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "mj12bot",
 "id": "mj12bot",
Expand DownExpand Up@@ -1559,6 +1595,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "reflectionbot",
 "id": "reflectionbot",
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; reflectionbot/1.0; +http://example.com/bot)",
 "id": "reflectionbot",
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "rogerbot",
 "id": "rogerbot",
Expand DownExpand Up@@ -1787,6 +1835,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "sofyabot",
 "id": "sofyabot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; sofyabot/1.0; +http://example.com/bot)",
 "id": "sofyabot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "sogou",
 "id": "sogou",
Expand DownExpand Up@@ -2147,6 +2207,18 @@
 "category": "generic_scraper",
 "matched": true
 },
 {
 "userAgent": "xai-searchbot",
 "id": "xai-searchbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; xai-searchbot/1.0; +http://example.com/bot)",
 "id": "xai-searchbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
 "matched": false
Expand Down

8 changes: 7 additions & 1 deletion
packages/webdecoy/src/bots/registry.generated.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@
 * agents share User-Agent substrings, so sorting this array changes how real
 * traffic is classified.
 *
 * 169 agents across 15 categories.
 * 175 agents across 15 categories.
 */

/**
Expand DownExpand Up@@ -75,6 +75,7 @@ export const BOT_CATEGORIES: readonly BotCategory[] = [

/** The agent table, in match order. */
export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "reflectionbot", name: "Reflectionbot", category: "training_crawler", organization: "Reflection AI", baseScore: 70, respectsRobots: true, uaPatterns: ["reflectionbot"] },
 { id: "gptbot", name: "GPTBot", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["gptbot"] },
 { id: "chatgpt-user", name: "ChatGPT-User", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["chatgpt-user", "chatgpt"] },
 { id: "oai-searchbot", name: "OAI-SearchBot", category: "training_crawler", organization: "OpenAI", baseScore: 80, respectsRobots: true, uaPatterns: ["oai-searchbot"] },
Expand DownExpand Up@@ -103,6 +104,10 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "velenpublicwebcrawler", name: "VelenPublicWebCrawler", category: "training_crawler", organization: "Velen", baseScore: 70, respectsRobots: false, uaPatterns: ["velenpublicwebcrawler"] },
 { id: "isscyberriskcrawler", name: "ISSCyberRiskCrawler", category: "training_crawler", organization: "ISS", baseScore: 65, respectsRobots: false, uaPatterns: ["isscyberriskcrawler"] },
 { id: "friendlycrawler", name: "FriendlyCrawler", category: "training_crawler", organization: "Unknown", baseScore: 65, respectsRobots: true, uaPatterns: ["friendlycrawler"] },
 { id: "duckassistbot", name: "DuckAssistBot", category: "ai_search_crawler", organization: "DuckDuckGo", baseScore: 65, respectsRobots: true, uaPatterns: ["duckassistbot"] },
 { id: "sofyabot", name: "SofyaBot", category: "ai_search_crawler", organization: "Sofya", baseScore: 65, respectsRobots: true, uaPatterns: ["sofyabot"] },
 { id: "xai-searchbot", name: "xAI-SearchBot", category: "ai_search_crawler", organization: "xAI", baseScore: 70, respectsRobots: true, uaPatterns: ["xai-searchbot"] },
 { id: "linkupbot", name: "LinkupBot", category: "ai_search_crawler", organization: "Linkup", baseScore: 65, respectsRobots: true, uaPatterns: ["linkupbot"] },
 { id: "searchgpt", name: "SearchGPT", category: "ai_search_crawler", organization: "OpenAI", baseScore: 75, respectsRobots: true, uaPatterns: ["searchgpt"] },
 { id: "phind", name: "Phind", category: "ai_search_crawler", organization: "Phind", baseScore: 70, respectsRobots: true, uaPatterns: ["phind"] },
 { id: "kagi", name: "Kagi", category: "ai_search_crawler", organization: "Kagi", baseScore: 65, respectsRobots: true, uaPatterns: ["kagi"] },
Expand All@@ -119,6 +124,7 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "browser-use", name: "Browser-Use", category: "ai_agent", organization: "Open Source", baseScore: 70, respectsRobots: false, uaPatterns: ["browser-use"] },
 { id: "stagehand", name: "Stagehand", category: "ai_agent", organization: "Browserbase", baseScore: 70, respectsRobots: false, uaPatterns: ["stagehand"] },
 { id: "multion", name: "MultiOn", category: "ai_agent", organization: "MultiOn", baseScore: 70, respectsRobots: false, uaPatterns: ["multion"] },
 { id: "iboubot", name: "IbouBot", category: "search_crawler", organization: "Ibou", baseScore: 30, respectsRobots: true, uaPatterns: ["iboubot"] },
 { id: "googlebot", name: "Googlebot", category: "search_crawler", organization: "Google", baseScore: 30, respectsRobots: true, uaPatterns: ["googlebot"] },
 { id: "bingbot", name: "Bingbot", category: "search_crawler", organization: "Microsoft", baseScore: 30, respectsRobots: true, uaPatterns: ["bingbot"] },
 { id: "yandexbot", name: "YandexBot", category: "search_crawler", organization: "Yandex", baseScore: 35, respectsRobots: true, uaPatterns: ["yandexbot"] },
Expand Down

1 change: 1 addition & 0 deletions
packages/webdecoy/src/index.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -84,6 +84,7 @@ export type { BotVerdict, BotAgent, BotCategory } from './bots';
// runtime with a fetch handler. `@webdecoy/hono` is a thin wrapper over it; Bun,
// Deno, Astro and Nitro need no package at all.
export { consoleLogger, silentLogger, fromPino } from './logger';
export type { Tracer, Span } from './tracing';
export type { Logger, LogFields } from './logger';

// Browser signals, joined to the requests that follow them. See client-signals.ts
Expand Down

51 changes: 49 additions & 2 deletions
packages/webdecoy/src/sdk.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -13,6 +13,8 @@ import { AgentVerifier } from './agent/verifier';
import type { AgentRequestInput, AgentVerdict } from './agent/types';
import { readEdgeVerdict } from './edge';
import { resolveLogger } from './logger';
import { startSpan, setAttribute, recordError, endSpan } from './tracing';
import type { Tracer } from './tracing';
import type { Logger } from './logger';
import { Decision, newDecisionId } from './decision';
import type { Conclusion } from './decision';
Expand All@@ -33,7 +35,7 @@ export class WebDecoy {
 private client: WebDecoyClient | null;
 private config: Omit<
 Required<WebDecoyConfig>,
 'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger'
 'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger' | 'tracer'
 > & {
 apiKey?: string;
 };
Expand All@@ -48,6 +50,8 @@ export class WebDecoy {
 private readonly characteristics: readonly import('./characteristics').Characteristic[];
 /** Where diagnostics go. Never console directly — see logger.ts. */
 readonly log: Logger;
 /** Optional OpenTelemetry tracer. Absent means no spans and no cost. */
 private readonly tracer?: Tracer;
 private readonly decisionCache: DecisionCache | null;

 constructor(config: WebDecoyConfig) {
Expand All@@ -74,6 +78,7 @@ export class WebDecoy {
 };

 this.log = resolveLogger(config.logger, this.config.debug);
 this.tracer = config.tracer;

 // Initialize API client only when apiKey is provided
 if (hasApiKey) {
Expand DownExpand Up@@ -253,7 +258,15 @@ export class WebDecoy {
 /** Evaluate rules against a prepared context and report any violations. */
 private runRules(context: RuleContext): RuleEngineResult | null {
 if (!this.ruleEngine) return null;

 const span = startSpan(this.tracer, 'webdecoy.rules');
 const result = this.ruleEngine.evaluate(context);
 setAttribute(span, 'webdecoy.rules.action', result.action);
 setAttribute(span, 'webdecoy.rules.evaluated', result.results.length);
 setAttribute(span, 'webdecoy.rules.violations', result.violations.length);
 if (result.rule) setAttribute(span, 'webdecoy.rules.deciding', result.rule);
 endSpan(span);

 if (result.violations.length > 0 && this.violationReporter) {
 this.violationReporter.report(result.violations);
 }
Expand DownExpand Up@@ -335,7 +348,41 @@ export class WebDecoy {
 // present on every outcome — and a per-return copy is a line someone would
 // eventually forget on the branch that mattered.
 const edge = readEdgeVerdict(metadata.headers);
 return (await this.decide(metadata, options)).withEdge(edge);

 const span = startSpan(this.tracer, 'webdecoy.protect');
 try {
 const decision = (await this.decide(metadata, options)).withEdge(edge);

 // Attributes chosen so a trace answers the questions an operator actually
 // asks: what did we decide, which rule decided it, and did this request
 // cost a round trip to ingest. The decision id joins the span to the
 // dashboard row.
 setAttribute(span, 'webdecoy.decision.id', decision.id);
 setAttribute(span, 'webdecoy.decision.conclusion', decision.conclusion);
 setAttribute(span, 'webdecoy.decision.allowed', decision.allowed);
 setAttribute(span, 'webdecoy.rules.evaluated', decision.results.length);
 if (decision.ruleResult?.rule) {
 setAttribute(span, 'webdecoy.decision.rule', decision.ruleResult.rule);
 }
 // A detection id that is not the decision id means the verdict came back
 // from ingest rather than being settled locally.
 setAttribute(
 span,
 'webdecoy.remote',
 decision.detection.detection_id !== decision.id,
 );
 if (decision.error) {
 setAttribute(span, 'webdecoy.error', decision.error);
 }
 return decision;
 } catch (error) {
 // decide() fails open rather than throwing, so this is a bug rather than
 // a bad day — worth marking on the span rather than swallowing.
 recordError(span, error);
 throw error;
 } finally {
 endSpan(span);
 }
 }

 private async decide(
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
chore(bots): regenerate registry for xAI-SearchBot and LinkupBot#39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

cport1
merged 3 commits into

from

Aug 25, 2026
 Merged


chore(bots): regenerate registry for xAI-SearchBot and LinkupBot
#39
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
f059bb0

feat(sdk): OpenTelemetry spans, without depending on OpenTelemetry


cport1 Aug 22, 2026
dd6b897

chore(bots): regenerate registry for xAI-SearchBot and LinkupBot


cport1 Aug 25, 2026
bed3035

chore(bots): regenerate registry for the five newly identified crawlers


cport1 Aug 25, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


8 changes: 8 additions & 0 deletions
CHANGELOG.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **OpenTelemetry spans** around `protect()` and rule evaluation. Pass a tracer: `new WebDecoy({ tracer: trace.getTracer('webdecoy') })`. Injected rather than imported, so the package stays dependency-free and edge-safe — the `Tracer` type is a structural subset of OpenTelemetry's, so `trace.getTracer()` works with no adapter, and omitting it means no spans, no dependency and no behaviour change. Attributes cover the decision id (which joins a span to its dashboard row), the conclusion, the deciding rule, and whether the request cost a round trip to ingest. A tracer that throws cannot fail a request.

### Changed

- **One adapter core.** Express, Fastify, Next.js (middleware and Pages wrapper) and the fetch guard each carried their own copy of skip-path matching, the 429 and 403 payloads, and honeytoken arming — five copies of one set of decisions, and five places the next correction can fail to land. They now share `adapter-core.ts`; the framework-specific response mechanics are untouched, and every honeytoken-injection test passes unchanged. Fastify keeps its awaited arming, which has no window where early requests are served without the link.

## [0.13.0] - 2026-08-22

### Added
Expand Down

25 changes: 25 additions & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -512,6 +512,31 @@ new WebDecoy({ logger: fromPino(pino()) }); // pino's argument order is reverse
`fromPino()` exists because passing a pino instance directly type-checks and then
silently drops every structured field.

## Tracing

Pass an OpenTelemetry tracer and `protect()` emits a span, with a child span for
rule evaluation:

```typescript
import { trace } from '@opentelemetry/api';

new WebDecoy({ tracer: trace.getTracer('webdecoy') });
```

The tracer is **injected, not imported** — this package has no dependencies and
runs on Workers and Vercel Edge, where a stray transitive import is expensive.
The `Tracer` type is a structural subset of OpenTelemetry's, so
`trace.getTracer()` satisfies it with no adapter. Omit it and there are no spans,
no dependency, and no behaviour change.

Attributes are the questions an operator actually asks: `decision.id` (which
joins the span to the dashboard row), `decision.conclusion`, `decision.rule`,
`rules.evaluated`, and `webdecoy.remote` — whether the request cost a round trip
to ingest or was settled locally.

A tracer that throws cannot fail a request. Observability that can take the
request path down is worse than none.

## Examples

See [examples](./examples) for complete working setups — e.g. [express-basic](./examples/express-basic).
Expand Down

2 changes: 1 addition & 1 deletion
packages/webdecoy/src/bots/bots.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ describe('matchUserAgent', () => {

describe('registry integrity', () => {
 it('carries the full generated table', () => {
 expect(BOT_REGISTRY.length).toBe(169);
 expect(BOT_REGISTRY.length).toBe(175);
 expect(BOT_CATEGORIES).toContain('training_crawler');
 expect(BOT_CATEGORIES).toContain('search_crawler');
 });
Expand Down

72 changes: 72 additions & 0 deletions
packages/webdecoy/src/bots/parity-vectors.generated.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -575,6 +575,18 @@
 "category": "seo_crawler",
 "matched": true
 },
 {
 "userAgent": "duckassistbot",
 "id": "duckassistbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; duckassistbot/1.0; +http://example.com/bot)",
 "id": "duckassistbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "duckduckbot",
 "id": "duckduckbot",
Expand DownExpand Up@@ -935,6 +947,18 @@
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "iboubot",
 "id": "iboubot",
 "category": "search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; iboubot/1.0; +http://example.com/bot)",
 "id": "iboubot",
 "category": "search_crawler",
 "matched": true
 },
 {
 "userAgent": "iframely",
 "id": "iframely",
Expand DownExpand Up@@ -1055,6 +1079,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "linkupbot",
 "id": "linkupbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; linkupbot/1.0; +http://example.com/bot)",
 "id": "linkupbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "mj12bot",
 "id": "mj12bot",
Expand DownExpand Up@@ -1559,6 +1595,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "reflectionbot",
 "id": "reflectionbot",
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; reflectionbot/1.0; +http://example.com/bot)",
 "id": "reflectionbot",
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "rogerbot",
 "id": "rogerbot",
Expand DownExpand Up@@ -1787,6 +1835,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "sofyabot",
 "id": "sofyabot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; sofyabot/1.0; +http://example.com/bot)",
 "id": "sofyabot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "sogou",
 "id": "sogou",
Expand DownExpand Up@@ -2147,6 +2207,18 @@
 "category": "generic_scraper",
 "matched": true
 },
 {
 "userAgent": "xai-searchbot",
 "id": "xai-searchbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; xai-searchbot/1.0; +http://example.com/bot)",
 "id": "xai-searchbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
 "matched": false
Expand Down

8 changes: 7 additions & 1 deletion
packages/webdecoy/src/bots/registry.generated.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@
 * agents share User-Agent substrings, so sorting this array changes how real
 * traffic is classified.
 *
 * 169 agents across 15 categories.
 * 175 agents across 15 categories.
 */

/**
Expand DownExpand Up@@ -75,6 +75,7 @@ export const BOT_CATEGORIES: readonly BotCategory[] = [

/** The agent table, in match order. */
export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "reflectionbot", name: "Reflectionbot", category: "training_crawler", organization: "Reflection AI", baseScore: 70, respectsRobots: true, uaPatterns: ["reflectionbot"] },
 { id: "gptbot", name: "GPTBot", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["gptbot"] },
 { id: "chatgpt-user", name: "ChatGPT-User", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["chatgpt-user", "chatgpt"] },
 { id: "oai-searchbot", name: "OAI-SearchBot", category: "training_crawler", organization: "OpenAI", baseScore: 80, respectsRobots: true, uaPatterns: ["oai-searchbot"] },
Expand DownExpand Up@@ -103,6 +104,10 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "velenpublicwebcrawler", name: "VelenPublicWebCrawler", category: "training_crawler", organization: "Velen", baseScore: 70, respectsRobots: false, uaPatterns: ["velenpublicwebcrawler"] },
 { id: "isscyberriskcrawler", name: "ISSCyberRiskCrawler", category: "training_crawler", organization: "ISS", baseScore: 65, respectsRobots: false, uaPatterns: ["isscyberriskcrawler"] },
 { id: "friendlycrawler", name: "FriendlyCrawler", category: "training_crawler", organization: "Unknown", baseScore: 65, respectsRobots: true, uaPatterns: ["friendlycrawler"] },
 { id: "duckassistbot", name: "DuckAssistBot", category: "ai_search_crawler", organization: "DuckDuckGo", baseScore: 65, respectsRobots: true, uaPatterns: ["duckassistbot"] },
 { id: "sofyabot", name: "SofyaBot", category: "ai_search_crawler", organization: "Sofya", baseScore: 65, respectsRobots: true, uaPatterns: ["sofyabot"] },
 { id: "xai-searchbot", name: "xAI-SearchBot", category: "ai_search_crawler", organization: "xAI", baseScore: 70, respectsRobots: true, uaPatterns: ["xai-searchbot"] },
 { id: "linkupbot", name: "LinkupBot", category: "ai_search_crawler", organization: "Linkup", baseScore: 65, respectsRobots: true, uaPatterns: ["linkupbot"] },
 { id: "searchgpt", name: "SearchGPT", category: "ai_search_crawler", organization: "OpenAI", baseScore: 75, respectsRobots: true, uaPatterns: ["searchgpt"] },
 { id: "phind", name: "Phind", category: "ai_search_crawler", organization: "Phind", baseScore: 70, respectsRobots: true, uaPatterns: ["phind"] },
 { id: "kagi", name: "Kagi", category: "ai_search_crawler", organization: "Kagi", baseScore: 65, respectsRobots: true, uaPatterns: ["kagi"] },
Expand All@@ -119,6 +124,7 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "browser-use", name: "Browser-Use", category: "ai_agent", organization: "Open Source", baseScore: 70, respectsRobots: false, uaPatterns: ["browser-use"] },
 { id: "stagehand", name: "Stagehand", category: "ai_agent", organization: "Browserbase", baseScore: 70, respectsRobots: false, uaPatterns: ["stagehand"] },
 { id: "multion", name: "MultiOn", category: "ai_agent", organization: "MultiOn", baseScore: 70, respectsRobots: false, uaPatterns: ["multion"] },
 { id: "iboubot", name: "IbouBot", category: "search_crawler", organization: "Ibou", baseScore: 30, respectsRobots: true, uaPatterns: ["iboubot"] },
 { id: "googlebot", name: "Googlebot", category: "search_crawler", organization: "Google", baseScore: 30, respectsRobots: true, uaPatterns: ["googlebot"] },
 { id: "bingbot", name: "Bingbot", category: "search_crawler", organization: "Microsoft", baseScore: 30, respectsRobots: true, uaPatterns: ["bingbot"] },
 { id: "yandexbot", name: "YandexBot", category: "search_crawler", organization: "Yandex", baseScore: 35, respectsRobots: true, uaPatterns: ["yandexbot"] },
Expand Down

1 change: 1 addition & 0 deletions
packages/webdecoy/src/index.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -84,6 +84,7 @@ export type { BotVerdict, BotAgent, BotCategory } from './bots';
// runtime with a fetch handler. `@webdecoy/hono` is a thin wrapper over it; Bun,
// Deno, Astro and Nitro need no package at all.
export { consoleLogger, silentLogger, fromPino } from './logger';
export type { Tracer, Span } from './tracing';
export type { Logger, LogFields } from './logger';

// Browser signals, joined to the requests that follow them. See client-signals.ts
Expand Down

51 changes: 49 additions & 2 deletions
packages/webdecoy/src/sdk.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -13,6 +13,8 @@ import { AgentVerifier } from './agent/verifier';
import type { AgentRequestInput, AgentVerdict } from './agent/types';
import { readEdgeVerdict } from './edge';
import { resolveLogger } from './logger';
import { startSpan, setAttribute, recordError, endSpan } from './tracing';
import type { Tracer } from './tracing';
import type { Logger } from './logger';
import { Decision, newDecisionId } from './decision';
import type { Conclusion } from './decision';
Expand All@@ -33,7 +35,7 @@ export class WebDecoy {
 private client: WebDecoyClient | null;
 private config: Omit<
 Required<WebDecoyConfig>,
 'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger'
 'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger' | 'tracer'
 > & {
 apiKey?: string;
 };
Expand All@@ -48,6 +50,8 @@ export class WebDecoy {
 private readonly characteristics: readonly import('./characteristics').Characteristic[];
 /** Where diagnostics go. Never console directly — see logger.ts. */
 readonly log: Logger;
 /** Optional OpenTelemetry tracer. Absent means no spans and no cost. */
 private readonly tracer?: Tracer;
 private readonly decisionCache: DecisionCache | null;

 constructor(config: WebDecoyConfig) {
Expand All@@ -74,6 +78,7 @@ export class WebDecoy {
 };

 this.log = resolveLogger(config.logger, this.config.debug);
 this.tracer = config.tracer;

 // Initialize API client only when apiKey is provided
 if (hasApiKey) {
Expand DownExpand Up@@ -253,7 +258,15 @@ export class WebDecoy {
 /** Evaluate rules against a prepared context and report any violations. */
 private runRules(context: RuleContext): RuleEngineResult | null {
 if (!this.ruleEngine) return null;

 const span = startSpan(this.tracer, 'webdecoy.rules');
 const result = this.ruleEngine.evaluate(context);
 setAttribute(span, 'webdecoy.rules.action', result.action);
 setAttribute(span, 'webdecoy.rules.evaluated', result.results.length);
 setAttribute(span, 'webdecoy.rules.violations', result.violations.length);
 if (result.rule) setAttribute(span, 'webdecoy.rules.deciding', result.rule);
 endSpan(span);

 if (result.violations.length > 0 && this.violationReporter) {
 this.violationReporter.report(result.violations);
 }
Expand DownExpand Up@@ -335,7 +348,41 @@ export class WebDecoy {
 // present on every outcome — and a per-return copy is a line someone would
 // eventually forget on the branch that mattered.
 const edge = readEdgeVerdict(metadata.headers);
 return (await this.decide(metadata, options)).withEdge(edge);

 const span = startSpan(this.tracer, 'webdecoy.protect');
 try {
 const decision = (await this.decide(metadata, options)).withEdge(edge);

 // Attributes chosen so a trace answers the questions an operator actually
 // asks: what did we decide, which rule decided it, and did this request
 // cost a round trip to ingest. The decision id joins the span to the
 // dashboard row.
 setAttribute(span, 'webdecoy.decision.id', decision.id);
 setAttribute(span, 'webdecoy.decision.conclusion', decision.conclusion);
 setAttribute(span, 'webdecoy.decision.allowed', decision.allowed);
 setAttribute(span, 'webdecoy.rules.evaluated', decision.results.length);
 if (decision.ruleResult?.rule) {
 setAttribute(span, 'webdecoy.decision.rule', decision.ruleResult.rule);
 }
 // A detection id that is not the decision id means the verdict came back
 // from ingest rather than being settled locally.
 setAttribute(
 span,
 'webdecoy.remote',
 decision.detection.detection_id !== decision.id,
 );
 if (decision.error) {
 setAttribute(span, 'webdecoy.error', decision.error);
 }
 return decision;
 } catch (error) {
 // decide() fails open rather than throwing, so this is a bug rather than
 // a bad day — worth marking on the span rather than swallowing.
 recordError(span, error);
 throw error;
 } finally {
 endSpan(span);
 }
 }

 private async decide(
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
chore(bots): regenerate registry for xAI-SearchBot and LinkupBot#39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

cport1
merged 3 commits into

from

Aug 25, 2026
 Merged


chore(bots): regenerate registry for xAI-SearchBot and LinkupBot
#39
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
f059bb0

feat(sdk): OpenTelemetry spans, without depending on OpenTelemetry


cport1 Aug 22, 2026
dd6b897

chore(bots): regenerate registry for xAI-SearchBot and LinkupBot


cport1 Aug 25, 2026
bed3035

chore(bots): regenerate registry for the five newly identified crawlers


cport1 Aug 25, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


8 changes: 8 additions & 0 deletions
CHANGELOG.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **OpenTelemetry spans** around `protect()` and rule evaluation. Pass a tracer: `new WebDecoy({ tracer: trace.getTracer('webdecoy') })`. Injected rather than imported, so the package stays dependency-free and edge-safe — the `Tracer` type is a structural subset of OpenTelemetry's, so `trace.getTracer()` works with no adapter, and omitting it means no spans, no dependency and no behaviour change. Attributes cover the decision id (which joins a span to its dashboard row), the conclusion, the deciding rule, and whether the request cost a round trip to ingest. A tracer that throws cannot fail a request.

### Changed

- **One adapter core.** Express, Fastify, Next.js (middleware and Pages wrapper) and the fetch guard each carried their own copy of skip-path matching, the 429 and 403 payloads, and honeytoken arming — five copies of one set of decisions, and five places the next correction can fail to land. They now share `adapter-core.ts`; the framework-specific response mechanics are untouched, and every honeytoken-injection test passes unchanged. Fastify keeps its awaited arming, which has no window where early requests are served without the link.

## [0.13.0] - 2026-08-22

### Added
Expand Down

25 changes: 25 additions & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -512,6 +512,31 @@ new WebDecoy({ logger: fromPino(pino()) }); // pino's argument order is reverse
`fromPino()` exists because passing a pino instance directly type-checks and then
silently drops every structured field.

## Tracing

Pass an OpenTelemetry tracer and `protect()` emits a span, with a child span for
rule evaluation:

```typescript
import { trace } from '@opentelemetry/api';

new WebDecoy({ tracer: trace.getTracer('webdecoy') });
```

The tracer is **injected, not imported** — this package has no dependencies and
runs on Workers and Vercel Edge, where a stray transitive import is expensive.
The `Tracer` type is a structural subset of OpenTelemetry's, so
`trace.getTracer()` satisfies it with no adapter. Omit it and there are no spans,
no dependency, and no behaviour change.

Attributes are the questions an operator actually asks: `decision.id` (which
joins the span to the dashboard row), `decision.conclusion`, `decision.rule`,
`rules.evaluated`, and `webdecoy.remote` — whether the request cost a round trip
to ingest or was settled locally.

A tracer that throws cannot fail a request. Observability that can take the
request path down is worse than none.

## Examples

See [examples](./examples) for complete working setups — e.g. [express-basic](./examples/express-basic).
Expand Down

2 changes: 1 addition & 1 deletion
packages/webdecoy/src/bots/bots.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ describe('matchUserAgent', () => {

describe('registry integrity', () => {
 it('carries the full generated table', () => {
 expect(BOT_REGISTRY.length).toBe(169);
 expect(BOT_REGISTRY.length).toBe(175);
 expect(BOT_CATEGORIES).toContain('training_crawler');
 expect(BOT_CATEGORIES).toContain('search_crawler');
 });
Expand Down

72 changes: 72 additions & 0 deletions
packages/webdecoy/src/bots/parity-vectors.generated.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -575,6 +575,18 @@
 "category": "seo_crawler",
 "matched": true
 },
 {
 "userAgent": "duckassistbot",
 "id": "duckassistbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; duckassistbot/1.0; +http://example.com/bot)",
 "id": "duckassistbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "duckduckbot",
 "id": "duckduckbot",
Expand DownExpand Up@@ -935,6 +947,18 @@
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "iboubot",
 "id": "iboubot",
 "category": "search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; iboubot/1.0; +http://example.com/bot)",
 "id": "iboubot",
 "category": "search_crawler",
 "matched": true
 },
 {
 "userAgent": "iframely",
 "id": "iframely",
Expand DownExpand Up@@ -1055,6 +1079,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "linkupbot",
 "id": "linkupbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; linkupbot/1.0; +http://example.com/bot)",
 "id": "linkupbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "mj12bot",
 "id": "mj12bot",
Expand DownExpand Up@@ -1559,6 +1595,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "reflectionbot",
 "id": "reflectionbot",
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; reflectionbot/1.0; +http://example.com/bot)",
 "id": "reflectionbot",
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "rogerbot",
 "id": "rogerbot",
Expand DownExpand Up@@ -1787,6 +1835,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "sofyabot",
 "id": "sofyabot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; sofyabot/1.0; +http://example.com/bot)",
 "id": "sofyabot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "sogou",
 "id": "sogou",
Expand DownExpand Up@@ -2147,6 +2207,18 @@
 "category": "generic_scraper",
 "matched": true
 },
 {
 "userAgent": "xai-searchbot",
 "id": "xai-searchbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; xai-searchbot/1.0; +http://example.com/bot)",
 "id": "xai-searchbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
 "matched": false
Expand Down

8 changes: 7 additions & 1 deletion
packages/webdecoy/src/bots/registry.generated.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@
 * agents share User-Agent substrings, so sorting this array changes how real
 * traffic is classified.
 *
 * 169 agents across 15 categories.
 * 175 agents across 15 categories.
 */

/**
Expand DownExpand Up@@ -75,6 +75,7 @@ export const BOT_CATEGORIES: readonly BotCategory[] = [

/** The agent table, in match order. */
export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "reflectionbot", name: "Reflectionbot", category: "training_crawler", organization: "Reflection AI", baseScore: 70, respectsRobots: true, uaPatterns: ["reflectionbot"] },
 { id: "gptbot", name: "GPTBot", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["gptbot"] },
 { id: "chatgpt-user", name: "ChatGPT-User", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["chatgpt-user", "chatgpt"] },
 { id: "oai-searchbot", name: "OAI-SearchBot", category: "training_crawler", organization: "OpenAI", baseScore: 80, respectsRobots: true, uaPatterns: ["oai-searchbot"] },
Expand DownExpand Up@@ -103,6 +104,10 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "velenpublicwebcrawler", name: "VelenPublicWebCrawler", category: "training_crawler", organization: "Velen", baseScore: 70, respectsRobots: false, uaPatterns: ["velenpublicwebcrawler"] },
 { id: "isscyberriskcrawler", name: "ISSCyberRiskCrawler", category: "training_crawler", organization: "ISS", baseScore: 65, respectsRobots: false, uaPatterns: ["isscyberriskcrawler"] },
 { id: "friendlycrawler", name: "FriendlyCrawler", category: "training_crawler", organization: "Unknown", baseScore: 65, respectsRobots: true, uaPatterns: ["friendlycrawler"] },
 { id: "duckassistbot", name: "DuckAssistBot", category: "ai_search_crawler", organization: "DuckDuckGo", baseScore: 65, respectsRobots: true, uaPatterns: ["duckassistbot"] },
 { id: "sofyabot", name: "SofyaBot", category: "ai_search_crawler", organization: "Sofya", baseScore: 65, respectsRobots: true, uaPatterns: ["sofyabot"] },
 { id: "xai-searchbot", name: "xAI-SearchBot", category: "ai_search_crawler", organization: "xAI", baseScore: 70, respectsRobots: true, uaPatterns: ["xai-searchbot"] },
 { id: "linkupbot", name: "LinkupBot", category: "ai_search_crawler", organization: "Linkup", baseScore: 65, respectsRobots: true, uaPatterns: ["linkupbot"] },
 { id: "searchgpt", name: "SearchGPT", category: "ai_search_crawler", organization: "OpenAI", baseScore: 75, respectsRobots: true, uaPatterns: ["searchgpt"] },
 { id: "phind", name: "Phind", category: "ai_search_crawler", organization: "Phind", baseScore: 70, respectsRobots: true, uaPatterns: ["phind"] },
 { id: "kagi", name: "Kagi", category: "ai_search_crawler", organization: "Kagi", baseScore: 65, respectsRobots: true, uaPatterns: ["kagi"] },
Expand All@@ -119,6 +124,7 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "browser-use", name: "Browser-Use", category: "ai_agent", organization: "Open Source", baseScore: 70, respectsRobots: false, uaPatterns: ["browser-use"] },
 { id: "stagehand", name: "Stagehand", category: "ai_agent", organization: "Browserbase", baseScore: 70, respectsRobots: false, uaPatterns: ["stagehand"] },
 { id: "multion", name: "MultiOn", category: "ai_agent", organization: "MultiOn", baseScore: 70, respectsRobots: false, uaPatterns: ["multion"] },
 { id: "iboubot", name: "IbouBot", category: "search_crawler", organization: "Ibou", baseScore: 30, respectsRobots: true, uaPatterns: ["iboubot"] },
 { id: "googlebot", name: "Googlebot", category: "search_crawler", organization: "Google", baseScore: 30, respectsRobots: true, uaPatterns: ["googlebot"] },
 { id: "bingbot", name: "Bingbot", category: "search_crawler", organization: "Microsoft", baseScore: 30, respectsRobots: true, uaPatterns: ["bingbot"] },
 { id: "yandexbot", name: "YandexBot", category: "search_crawler", organization: "Yandex", baseScore: 35, respectsRobots: true, uaPatterns: ["yandexbot"] },
Expand Down

1 change: 1 addition & 0 deletions
packages/webdecoy/src/index.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -84,6 +84,7 @@ export type { BotVerdict, BotAgent, BotCategory } from './bots';
// runtime with a fetch handler. `@webdecoy/hono` is a thin wrapper over it; Bun,
// Deno, Astro and Nitro need no package at all.
export { consoleLogger, silentLogger, fromPino } from './logger';
export type { Tracer, Span } from './tracing';
export type { Logger, LogFields } from './logger';

// Browser signals, joined to the requests that follow them. See client-signals.ts
Expand Down

51 changes: 49 additions & 2 deletions
packages/webdecoy/src/sdk.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -13,6 +13,8 @@ import { AgentVerifier } from './agent/verifier';
import type { AgentRequestInput, AgentVerdict } from './agent/types';
import { readEdgeVerdict } from './edge';
import { resolveLogger } from './logger';
import { startSpan, setAttribute, recordError, endSpan } from './tracing';
import type { Tracer } from './tracing';
import type { Logger } from './logger';
import { Decision, newDecisionId } from './decision';
import type { Conclusion } from './decision';
Expand All@@ -33,7 +35,7 @@ export class WebDecoy {
 private client: WebDecoyClient | null;
 private config: Omit<
 Required<WebDecoyConfig>,
 'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger'
 'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger' | 'tracer'
 > & {
 apiKey?: string;
 };
Expand All@@ -48,6 +50,8 @@ export class WebDecoy {
 private readonly characteristics: readonly import('./characteristics').Characteristic[];
 /** Where diagnostics go. Never console directly — see logger.ts. */
 readonly log: Logger;
 /** Optional OpenTelemetry tracer. Absent means no spans and no cost. */
 private readonly tracer?: Tracer;
 private readonly decisionCache: DecisionCache | null;

 constructor(config: WebDecoyConfig) {
Expand All@@ -74,6 +78,7 @@ export class WebDecoy {
 };

 this.log = resolveLogger(config.logger, this.config.debug);
 this.tracer = config.tracer;

 // Initialize API client only when apiKey is provided
 if (hasApiKey) {
Expand DownExpand Up@@ -253,7 +258,15 @@ export class WebDecoy {
 /** Evaluate rules against a prepared context and report any violations. */
 private runRules(context: RuleContext): RuleEngineResult | null {
 if (!this.ruleEngine) return null;

 const span = startSpan(this.tracer, 'webdecoy.rules');
 const result = this.ruleEngine.evaluate(context);
 setAttribute(span, 'webdecoy.rules.action', result.action);
 setAttribute(span, 'webdecoy.rules.evaluated', result.results.length);
 setAttribute(span, 'webdecoy.rules.violations', result.violations.length);
 if (result.rule) setAttribute(span, 'webdecoy.rules.deciding', result.rule);
 endSpan(span);

 if (result.violations.length > 0 && this.violationReporter) {
 this.violationReporter.report(result.violations);
 }
Expand DownExpand Up@@ -335,7 +348,41 @@ export class WebDecoy {
 // present on every outcome — and a per-return copy is a line someone would
 // eventually forget on the branch that mattered.
 const edge = readEdgeVerdict(metadata.headers);
 return (await this.decide(metadata, options)).withEdge(edge);

 const span = startSpan(this.tracer, 'webdecoy.protect');
 try {
 const decision = (await this.decide(metadata, options)).withEdge(edge);

 // Attributes chosen so a trace answers the questions an operator actually
 // asks: what did we decide, which rule decided it, and did this request
 // cost a round trip to ingest. The decision id joins the span to the
 // dashboard row.
 setAttribute(span, 'webdecoy.decision.id', decision.id);
 setAttribute(span, 'webdecoy.decision.conclusion', decision.conclusion);
 setAttribute(span, 'webdecoy.decision.allowed', decision.allowed);
 setAttribute(span, 'webdecoy.rules.evaluated', decision.results.length);
 if (decision.ruleResult?.rule) {
 setAttribute(span, 'webdecoy.decision.rule', decision.ruleResult.rule);
 }
 // A detection id that is not the decision id means the verdict came back
 // from ingest rather than being settled locally.
 setAttribute(
 span,
 'webdecoy.remote',
 decision.detection.detection_id !== decision.id,
 );
 if (decision.error) {
 setAttribute(span, 'webdecoy.error', decision.error);
 }
 return decision;
 } catch (error) {
 // decide() fails open rather than throwing, so this is a bug rather than
 // a bad day — worth marking on the span rather than swallowing.
 recordError(span, error);
 throw error;
 } finally {
 endSpan(span);
 }
 }

 private async decide(
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
chore(bots): regenerate registry for xAI-SearchBot and LinkupBot#39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

cport1
merged 3 commits into

from

Aug 25, 2026
 Merged


chore(bots): regenerate registry for xAI-SearchBot and LinkupBot
#39
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
f059bb0

feat(sdk): OpenTelemetry spans, without depending on OpenTelemetry


cport1 Aug 22, 2026
dd6b897

chore(bots): regenerate registry for xAI-SearchBot and LinkupBot


cport1 Aug 25, 2026
bed3035

chore(bots): regenerate registry for the five newly identified crawlers


cport1 Aug 25, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


8 changes: 8 additions & 0 deletions
CHANGELOG.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **OpenTelemetry spans** around `protect()` and rule evaluation. Pass a tracer: `new WebDecoy({ tracer: trace.getTracer('webdecoy') })`. Injected rather than imported, so the package stays dependency-free and edge-safe — the `Tracer` type is a structural subset of OpenTelemetry's, so `trace.getTracer()` works with no adapter, and omitting it means no spans, no dependency and no behaviour change. Attributes cover the decision id (which joins a span to its dashboard row), the conclusion, the deciding rule, and whether the request cost a round trip to ingest. A tracer that throws cannot fail a request.

### Changed

- **One adapter core.** Express, Fastify, Next.js (middleware and Pages wrapper) and the fetch guard each carried their own copy of skip-path matching, the 429 and 403 payloads, and honeytoken arming — five copies of one set of decisions, and five places the next correction can fail to land. They now share `adapter-core.ts`; the framework-specific response mechanics are untouched, and every honeytoken-injection test passes unchanged. Fastify keeps its awaited arming, which has no window where early requests are served without the link.

## [0.13.0] - 2026-08-22

### Added
Expand Down

25 changes: 25 additions & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -512,6 +512,31 @@ new WebDecoy({ logger: fromPino(pino()) }); // pino's argument order is reverse
`fromPino()` exists because passing a pino instance directly type-checks and then
silently drops every structured field.

## Tracing

Pass an OpenTelemetry tracer and `protect()` emits a span, with a child span for
rule evaluation:

```typescript
import { trace } from '@opentelemetry/api';

new WebDecoy({ tracer: trace.getTracer('webdecoy') });
```

The tracer is **injected, not imported** — this package has no dependencies and
runs on Workers and Vercel Edge, where a stray transitive import is expensive.
The `Tracer` type is a structural subset of OpenTelemetry's, so
`trace.getTracer()` satisfies it with no adapter. Omit it and there are no spans,
no dependency, and no behaviour change.

Attributes are the questions an operator actually asks: `decision.id` (which
joins the span to the dashboard row), `decision.conclusion`, `decision.rule`,
`rules.evaluated`, and `webdecoy.remote` — whether the request cost a round trip
to ingest or was settled locally.

A tracer that throws cannot fail a request. Observability that can take the
request path down is worse than none.

## Examples

See [examples](./examples) for complete working setups — e.g. [express-basic](./examples/express-basic).
Expand Down

2 changes: 1 addition & 1 deletion
packages/webdecoy/src/bots/bots.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ describe('matchUserAgent', () => {

describe('registry integrity', () => {
 it('carries the full generated table', () => {
 expect(BOT_REGISTRY.length).toBe(169);
 expect(BOT_REGISTRY.length).toBe(175);
 expect(BOT_CATEGORIES).toContain('training_crawler');
 expect(BOT_CATEGORIES).toContain('search_crawler');
 });
Expand Down

72 changes: 72 additions & 0 deletions
packages/webdecoy/src/bots/parity-vectors.generated.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -575,6 +575,18 @@
 "category": "seo_crawler",
 "matched": true
 },
 {
 "userAgent": "duckassistbot",
 "id": "duckassistbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; duckassistbot/1.0; +http://example.com/bot)",
 "id": "duckassistbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "duckduckbot",
 "id": "duckduckbot",
Expand DownExpand Up@@ -935,6 +947,18 @@
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "iboubot",
 "id": "iboubot",
 "category": "search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; iboubot/1.0; +http://example.com/bot)",
 "id": "iboubot",
 "category": "search_crawler",
 "matched": true
 },
 {
 "userAgent": "iframely",
 "id": "iframely",
Expand DownExpand Up@@ -1055,6 +1079,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "linkupbot",
 "id": "linkupbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; linkupbot/1.0; +http://example.com/bot)",
 "id": "linkupbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "mj12bot",
 "id": "mj12bot",
Expand DownExpand Up@@ -1559,6 +1595,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "reflectionbot",
 "id": "reflectionbot",
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; reflectionbot/1.0; +http://example.com/bot)",
 "id": "reflectionbot",
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "rogerbot",
 "id": "rogerbot",
Expand DownExpand Up@@ -1787,6 +1835,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "sofyabot",
 "id": "sofyabot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; sofyabot/1.0; +http://example.com/bot)",
 "id": "sofyabot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "sogou",
 "id": "sogou",
Expand DownExpand Up@@ -2147,6 +2207,18 @@
 "category": "generic_scraper",
 "matched": true
 },
 {
 "userAgent": "xai-searchbot",
 "id": "xai-searchbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; xai-searchbot/1.0; +http://example.com/bot)",
 "id": "xai-searchbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
 "matched": false
Expand Down

8 changes: 7 additions & 1 deletion
packages/webdecoy/src/bots/registry.generated.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@
 * agents share User-Agent substrings, so sorting this array changes how real
 * traffic is classified.
 *
 * 169 agents across 15 categories.
 * 175 agents across 15 categories.
 */

/**
Expand DownExpand Up@@ -75,6 +75,7 @@ export const BOT_CATEGORIES: readonly BotCategory[] = [

/** The agent table, in match order. */
export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "reflectionbot", name: "Reflectionbot", category: "training_crawler", organization: "Reflection AI", baseScore: 70, respectsRobots: true, uaPatterns: ["reflectionbot"] },
 { id: "gptbot", name: "GPTBot", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["gptbot"] },
 { id: "chatgpt-user", name: "ChatGPT-User", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["chatgpt-user", "chatgpt"] },
 { id: "oai-searchbot", name: "OAI-SearchBot", category: "training_crawler", organization: "OpenAI", baseScore: 80, respectsRobots: true, uaPatterns: ["oai-searchbot"] },
Expand DownExpand Up@@ -103,6 +104,10 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "velenpublicwebcrawler", name: "VelenPublicWebCrawler", category: "training_crawler", organization: "Velen", baseScore: 70, respectsRobots: false, uaPatterns: ["velenpublicwebcrawler"] },
 { id: "isscyberriskcrawler", name: "ISSCyberRiskCrawler", category: "training_crawler", organization: "ISS", baseScore: 65, respectsRobots: false, uaPatterns: ["isscyberriskcrawler"] },
 { id: "friendlycrawler", name: "FriendlyCrawler", category: "training_crawler", organization: "Unknown", baseScore: 65, respectsRobots: true, uaPatterns: ["friendlycrawler"] },
 { id: "duckassistbot", name: "DuckAssistBot", category: "ai_search_crawler", organization: "DuckDuckGo", baseScore: 65, respectsRobots: true, uaPatterns: ["duckassistbot"] },
 { id: "sofyabot", name: "SofyaBot", category: "ai_search_crawler", organization: "Sofya", baseScore: 65, respectsRobots: true, uaPatterns: ["sofyabot"] },
 { id: "xai-searchbot", name: "xAI-SearchBot", category: "ai_search_crawler", organization: "xAI", baseScore: 70, respectsRobots: true, uaPatterns: ["xai-searchbot"] },
 { id: "linkupbot", name: "LinkupBot", category: "ai_search_crawler", organization: "Linkup", baseScore: 65, respectsRobots: true, uaPatterns: ["linkupbot"] },
 { id: "searchgpt", name: "SearchGPT", category: "ai_search_crawler", organization: "OpenAI", baseScore: 75, respectsRobots: true, uaPatterns: ["searchgpt"] },
 { id: "phind", name: "Phind", category: "ai_search_crawler", organization: "Phind", baseScore: 70, respectsRobots: true, uaPatterns: ["phind"] },
 { id: "kagi", name: "Kagi", category: "ai_search_crawler", organization: "Kagi", baseScore: 65, respectsRobots: true, uaPatterns: ["kagi"] },
Expand All@@ -119,6 +124,7 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "browser-use", name: "Browser-Use", category: "ai_agent", organization: "Open Source", baseScore: 70, respectsRobots: false, uaPatterns: ["browser-use"] },
 { id: "stagehand", name: "Stagehand", category: "ai_agent", organization: "Browserbase", baseScore: 70, respectsRobots: false, uaPatterns: ["stagehand"] },
 { id: "multion", name: "MultiOn", category: "ai_agent", organization: "MultiOn", baseScore: 70, respectsRobots: false, uaPatterns: ["multion"] },
 { id: "iboubot", name: "IbouBot", category: "search_crawler", organization: "Ibou", baseScore: 30, respectsRobots: true, uaPatterns: ["iboubot"] },
 { id: "googlebot", name: "Googlebot", category: "search_crawler", organization: "Google", baseScore: 30, respectsRobots: true, uaPatterns: ["googlebot"] },
 { id: "bingbot", name: "Bingbot", category: "search_crawler", organization: "Microsoft", baseScore: 30, respectsRobots: true, uaPatterns: ["bingbot"] },
 { id: "yandexbot", name: "YandexBot", category: "search_crawler", organization: "Yandex", baseScore: 35, respectsRobots: true, uaPatterns: ["yandexbot"] },
Expand Down

1 change: 1 addition & 0 deletions
packages/webdecoy/src/index.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -84,6 +84,7 @@ export type { BotVerdict, BotAgent, BotCategory } from './bots';
// runtime with a fetch handler. `@webdecoy/hono` is a thin wrapper over it; Bun,
// Deno, Astro and Nitro need no package at all.
export { consoleLogger, silentLogger, fromPino } from './logger';
export type { Tracer, Span } from './tracing';
export type { Logger, LogFields } from './logger';

// Browser signals, joined to the requests that follow them. See client-signals.ts
Expand Down

51 changes: 49 additions & 2 deletions
packages/webdecoy/src/sdk.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -13,6 +13,8 @@ import { AgentVerifier } from './agent/verifier';
import type { AgentRequestInput, AgentVerdict } from './agent/types';
import { readEdgeVerdict } from './edge';
import { resolveLogger } from './logger';
import { startSpan, setAttribute, recordError, endSpan } from './tracing';
import type { Tracer } from './tracing';
import type { Logger } from './logger';
import { Decision, newDecisionId } from './decision';
import type { Conclusion } from './decision';
Expand All@@ -33,7 +35,7 @@ export class WebDecoy {
 private client: WebDecoyClient | null;
 private config: Omit<
 Required<WebDecoyConfig>,
 'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger'
 'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger' | 'tracer'
 > & {
 apiKey?: string;
 };
Expand All@@ -48,6 +50,8 @@ export class WebDecoy {
 private readonly characteristics: readonly import('./characteristics').Characteristic[];
 /** Where diagnostics go. Never console directly — see logger.ts. */
 readonly log: Logger;
 /** Optional OpenTelemetry tracer. Absent means no spans and no cost. */
 private readonly tracer?: Tracer;
 private readonly decisionCache: DecisionCache | null;

 constructor(config: WebDecoyConfig) {
Expand All@@ -74,6 +78,7 @@ export class WebDecoy {
 };

 this.log = resolveLogger(config.logger, this.config.debug);
 this.tracer = config.tracer;

 // Initialize API client only when apiKey is provided
 if (hasApiKey) {
Expand DownExpand Up@@ -253,7 +258,15 @@ export class WebDecoy {
 /** Evaluate rules against a prepared context and report any violations. */
 private runRules(context: RuleContext): RuleEngineResult | null {
 if (!this.ruleEngine) return null;

 const span = startSpan(this.tracer, 'webdecoy.rules');
 const result = this.ruleEngine.evaluate(context);
 setAttribute(span, 'webdecoy.rules.action', result.action);
 setAttribute(span, 'webdecoy.rules.evaluated', result.results.length);
 setAttribute(span, 'webdecoy.rules.violations', result.violations.length);
 if (result.rule) setAttribute(span, 'webdecoy.rules.deciding', result.rule);
 endSpan(span);

 if (result.violations.length > 0 && this.violationReporter) {
 this.violationReporter.report(result.violations);
 }
Expand DownExpand Up@@ -335,7 +348,41 @@ export class WebDecoy {
 // present on every outcome — and a per-return copy is a line someone would
 // eventually forget on the branch that mattered.
 const edge = readEdgeVerdict(metadata.headers);
 return (await this.decide(metadata, options)).withEdge(edge);

 const span = startSpan(this.tracer, 'webdecoy.protect');
 try {
 const decision = (await this.decide(metadata, options)).withEdge(edge);

 // Attributes chosen so a trace answers the questions an operator actually
 // asks: what did we decide, which rule decided it, and did this request
 // cost a round trip to ingest. The decision id joins the span to the
 // dashboard row.
 setAttribute(span, 'webdecoy.decision.id', decision.id);
 setAttribute(span, 'webdecoy.decision.conclusion', decision.conclusion);
 setAttribute(span, 'webdecoy.decision.allowed', decision.allowed);
 setAttribute(span, 'webdecoy.rules.evaluated', decision.results.length);
 if (decision.ruleResult?.rule) {
 setAttribute(span, 'webdecoy.decision.rule', decision.ruleResult.rule);
 }
 // A detection id that is not the decision id means the verdict came back
 // from ingest rather than being settled locally.
 setAttribute(
 span,
 'webdecoy.remote',
 decision.detection.detection_id !== decision.id,
 );
 if (decision.error) {
 setAttribute(span, 'webdecoy.error', decision.error);
 }
 return decision;
 } catch (error) {
 // decide() fails open rather than throwing, so this is a bug rather than
 // a bad day — worth marking on the span rather than swallowing.
 recordError(span, error);
 throw error;
 } finally {
 endSpan(span);
 }
 }

 private async decide(
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
chore(bots): regenerate registry for xAI-SearchBot and LinkupBot#39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

cport1
merged 3 commits into

from

Aug 25, 2026
 Merged


chore(bots): regenerate registry for xAI-SearchBot and LinkupBot
#39
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
f059bb0

feat(sdk): OpenTelemetry spans, without depending on OpenTelemetry


cport1 Aug 22, 2026
dd6b897

chore(bots): regenerate registry for xAI-SearchBot and LinkupBot


cport1 Aug 25, 2026
bed3035

chore(bots): regenerate registry for the five newly identified crawlers


cport1 Aug 25, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


8 changes: 8 additions & 0 deletions
CHANGELOG.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **OpenTelemetry spans** around `protect()` and rule evaluation. Pass a tracer: `new WebDecoy({ tracer: trace.getTracer('webdecoy') })`. Injected rather than imported, so the package stays dependency-free and edge-safe — the `Tracer` type is a structural subset of OpenTelemetry's, so `trace.getTracer()` works with no adapter, and omitting it means no spans, no dependency and no behaviour change. Attributes cover the decision id (which joins a span to its dashboard row), the conclusion, the deciding rule, and whether the request cost a round trip to ingest. A tracer that throws cannot fail a request.

### Changed

- **One adapter core.** Express, Fastify, Next.js (middleware and Pages wrapper) and the fetch guard each carried their own copy of skip-path matching, the 429 and 403 payloads, and honeytoken arming — five copies of one set of decisions, and five places the next correction can fail to land. They now share `adapter-core.ts`; the framework-specific response mechanics are untouched, and every honeytoken-injection test passes unchanged. Fastify keeps its awaited arming, which has no window where early requests are served without the link.

## [0.13.0] - 2026-08-22

### Added
Expand Down

25 changes: 25 additions & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -512,6 +512,31 @@ new WebDecoy({ logger: fromPino(pino()) }); // pino's argument order is reverse
`fromPino()` exists because passing a pino instance directly type-checks and then
silently drops every structured field.

## Tracing

Pass an OpenTelemetry tracer and `protect()` emits a span, with a child span for
rule evaluation:

```typescript
import { trace } from '@opentelemetry/api';

new WebDecoy({ tracer: trace.getTracer('webdecoy') });
```

The tracer is **injected, not imported** — this package has no dependencies and
runs on Workers and Vercel Edge, where a stray transitive import is expensive.
The `Tracer` type is a structural subset of OpenTelemetry's, so
`trace.getTracer()` satisfies it with no adapter. Omit it and there are no spans,
no dependency, and no behaviour change.

Attributes are the questions an operator actually asks: `decision.id` (which
joins the span to the dashboard row), `decision.conclusion`, `decision.rule`,
`rules.evaluated`, and `webdecoy.remote` — whether the request cost a round trip
to ingest or was settled locally.

A tracer that throws cannot fail a request. Observability that can take the
request path down is worse than none.

## Examples

See [examples](./examples) for complete working setups — e.g. [express-basic](./examples/express-basic).
Expand Down

2 changes: 1 addition & 1 deletion
packages/webdecoy/src/bots/bots.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ describe('matchUserAgent', () => {

describe('registry integrity', () => {
 it('carries the full generated table', () => {
 expect(BOT_REGISTRY.length).toBe(169);
 expect(BOT_REGISTRY.length).toBe(175);
 expect(BOT_CATEGORIES).toContain('training_crawler');
 expect(BOT_CATEGORIES).toContain('search_crawler');
 });
Expand Down

72 changes: 72 additions & 0 deletions
packages/webdecoy/src/bots/parity-vectors.generated.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -575,6 +575,18 @@
 "category": "seo_crawler",
 "matched": true
 },
 {
 "userAgent": "duckassistbot",
 "id": "duckassistbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; duckassistbot/1.0; +http://example.com/bot)",
 "id": "duckassistbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "duckduckbot",
 "id": "duckduckbot",
Expand DownExpand Up@@ -935,6 +947,18 @@
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "iboubot",
 "id": "iboubot",
 "category": "search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; iboubot/1.0; +http://example.com/bot)",
 "id": "iboubot",
 "category": "search_crawler",
 "matched": true
 },
 {
 "userAgent": "iframely",
 "id": "iframely",
Expand DownExpand Up@@ -1055,6 +1079,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "linkupbot",
 "id": "linkupbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; linkupbot/1.0; +http://example.com/bot)",
 "id": "linkupbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "mj12bot",
 "id": "mj12bot",
Expand DownExpand Up@@ -1559,6 +1595,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "reflectionbot",
 "id": "reflectionbot",
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; reflectionbot/1.0; +http://example.com/bot)",
 "id": "reflectionbot",
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "rogerbot",
 "id": "rogerbot",
Expand DownExpand Up@@ -1787,6 +1835,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "sofyabot",
 "id": "sofyabot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; sofyabot/1.0; +http://example.com/bot)",
 "id": "sofyabot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "sogou",
 "id": "sogou",
Expand DownExpand Up@@ -2147,6 +2207,18 @@
 "category": "generic_scraper",
 "matched": true
 },
 {
 "userAgent": "xai-searchbot",
 "id": "xai-searchbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; xai-searchbot/1.0; +http://example.com/bot)",
 "id": "xai-searchbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
 "matched": false
Expand Down

8 changes: 7 additions & 1 deletion
packages/webdecoy/src/bots/registry.generated.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@
 * agents share User-Agent substrings, so sorting this array changes how real
 * traffic is classified.
 *
 * 169 agents across 15 categories.
 * 175 agents across 15 categories.
 */

/**
Expand DownExpand Up@@ -75,6 +75,7 @@ export const BOT_CATEGORIES: readonly BotCategory[] = [

/** The agent table, in match order. */
export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "reflectionbot", name: "Reflectionbot", category: "training_crawler", organization: "Reflection AI", baseScore: 70, respectsRobots: true, uaPatterns: ["reflectionbot"] },
 { id: "gptbot", name: "GPTBot", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["gptbot"] },
 { id: "chatgpt-user", name: "ChatGPT-User", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["chatgpt-user", "chatgpt"] },
 { id: "oai-searchbot", name: "OAI-SearchBot", category: "training_crawler", organization: "OpenAI", baseScore: 80, respectsRobots: true, uaPatterns: ["oai-searchbot"] },
Expand DownExpand Up@@ -103,6 +104,10 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "velenpublicwebcrawler", name: "VelenPublicWebCrawler", category: "training_crawler", organization: "Velen", baseScore: 70, respectsRobots: false, uaPatterns: ["velenpublicwebcrawler"] },
 { id: "isscyberriskcrawler", name: "ISSCyberRiskCrawler", category: "training_crawler", organization: "ISS", baseScore: 65, respectsRobots: false, uaPatterns: ["isscyberriskcrawler"] },
 { id: "friendlycrawler", name: "FriendlyCrawler", category: "training_crawler", organization: "Unknown", baseScore: 65, respectsRobots: true, uaPatterns: ["friendlycrawler"] },
 { id: "duckassistbot", name: "DuckAssistBot", category: "ai_search_crawler", organization: "DuckDuckGo", baseScore: 65, respectsRobots: true, uaPatterns: ["duckassistbot"] },
 { id: "sofyabot", name: "SofyaBot", category: "ai_search_crawler", organization: "Sofya", baseScore: 65, respectsRobots: true, uaPatterns: ["sofyabot"] },
 { id: "xai-searchbot", name: "xAI-SearchBot", category: "ai_search_crawler", organization: "xAI", baseScore: 70, respectsRobots: true, uaPatterns: ["xai-searchbot"] },
 { id: "linkupbot", name: "LinkupBot", category: "ai_search_crawler", organization: "Linkup", baseScore: 65, respectsRobots: true, uaPatterns: ["linkupbot"] },
 { id: "searchgpt", name: "SearchGPT", category: "ai_search_crawler", organization: "OpenAI", baseScore: 75, respectsRobots: true, uaPatterns: ["searchgpt"] },
 { id: "phind", name: "Phind", category: "ai_search_crawler", organization: "Phind", baseScore: 70, respectsRobots: true, uaPatterns: ["phind"] },
 { id: "kagi", name: "Kagi", category: "ai_search_crawler", organization: "Kagi", baseScore: 65, respectsRobots: true, uaPatterns: ["kagi"] },
Expand All@@ -119,6 +124,7 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "browser-use", name: "Browser-Use", category: "ai_agent", organization: "Open Source", baseScore: 70, respectsRobots: false, uaPatterns: ["browser-use"] },
 { id: "stagehand", name: "Stagehand", category: "ai_agent", organization: "Browserbase", baseScore: 70, respectsRobots: false, uaPatterns: ["stagehand"] },
 { id: "multion", name: "MultiOn", category: "ai_agent", organization: "MultiOn", baseScore: 70, respectsRobots: false, uaPatterns: ["multion"] },
 { id: "iboubot", name: "IbouBot", category: "search_crawler", organization: "Ibou", baseScore: 30, respectsRobots: true, uaPatterns: ["iboubot"] },
 { id: "googlebot", name: "Googlebot", category: "search_crawler", organization: "Google", baseScore: 30, respectsRobots: true, uaPatterns: ["googlebot"] },
 { id: "bingbot", name: "Bingbot", category: "search_crawler", organization: "Microsoft", baseScore: 30, respectsRobots: true, uaPatterns: ["bingbot"] },
 { id: "yandexbot", name: "YandexBot", category: "search_crawler", organization: "Yandex", baseScore: 35, respectsRobots: true, uaPatterns: ["yandexbot"] },
Expand Down

1 change: 1 addition & 0 deletions
packages/webdecoy/src/index.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -84,6 +84,7 @@ export type { BotVerdict, BotAgent, BotCategory } from './bots';
// runtime with a fetch handler. `@webdecoy/hono` is a thin wrapper over it; Bun,
// Deno, Astro and Nitro need no package at all.
export { consoleLogger, silentLogger, fromPino } from './logger';
export type { Tracer, Span } from './tracing';
export type { Logger, LogFields } from './logger';

// Browser signals, joined to the requests that follow them. See client-signals.ts
Expand Down

51 changes: 49 additions & 2 deletions
packages/webdecoy/src/sdk.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -13,6 +13,8 @@ import { AgentVerifier } from './agent/verifier';
import type { AgentRequestInput, AgentVerdict } from './agent/types';
import { readEdgeVerdict } from './edge';
import { resolveLogger } from './logger';
import { startSpan, setAttribute, recordError, endSpan } from './tracing';
import type { Tracer } from './tracing';
import type { Logger } from './logger';
import { Decision, newDecisionId } from './decision';
import type { Conclusion } from './decision';
Expand All@@ -33,7 +35,7 @@ export class WebDecoy {
 private client: WebDecoyClient | null;
 private config: Omit<
 Required<WebDecoyConfig>,
 'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger'
 'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger' | 'tracer'
 > & {
 apiKey?: string;
 };
Expand All@@ -48,6 +50,8 @@ export class WebDecoy {
 private readonly characteristics: readonly import('./characteristics').Characteristic[];
 /** Where diagnostics go. Never console directly — see logger.ts. */
 readonly log: Logger;
 /** Optional OpenTelemetry tracer. Absent means no spans and no cost. */
 private readonly tracer?: Tracer;
 private readonly decisionCache: DecisionCache | null;

 constructor(config: WebDecoyConfig) {
Expand All@@ -74,6 +78,7 @@ export class WebDecoy {
 };

 this.log = resolveLogger(config.logger, this.config.debug);
 this.tracer = config.tracer;

 // Initialize API client only when apiKey is provided
 if (hasApiKey) {
Expand DownExpand Up@@ -253,7 +258,15 @@ export class WebDecoy {
 /** Evaluate rules against a prepared context and report any violations. */
 private runRules(context: RuleContext): RuleEngineResult | null {
 if (!this.ruleEngine) return null;

 const span = startSpan(this.tracer, 'webdecoy.rules');
 const result = this.ruleEngine.evaluate(context);
 setAttribute(span, 'webdecoy.rules.action', result.action);
 setAttribute(span, 'webdecoy.rules.evaluated', result.results.length);
 setAttribute(span, 'webdecoy.rules.violations', result.violations.length);
 if (result.rule) setAttribute(span, 'webdecoy.rules.deciding', result.rule);
 endSpan(span);

 if (result.violations.length > 0 && this.violationReporter) {
 this.violationReporter.report(result.violations);
 }
Expand DownExpand Up@@ -335,7 +348,41 @@ export class WebDecoy {
 // present on every outcome — and a per-return copy is a line someone would
 // eventually forget on the branch that mattered.
 const edge = readEdgeVerdict(metadata.headers);
 return (await this.decide(metadata, options)).withEdge(edge);

 const span = startSpan(this.tracer, 'webdecoy.protect');
 try {
 const decision = (await this.decide(metadata, options)).withEdge(edge);

 // Attributes chosen so a trace answers the questions an operator actually
 // asks: what did we decide, which rule decided it, and did this request
 // cost a round trip to ingest. The decision id joins the span to the
 // dashboard row.
 setAttribute(span, 'webdecoy.decision.id', decision.id);
 setAttribute(span, 'webdecoy.decision.conclusion', decision.conclusion);
 setAttribute(span, 'webdecoy.decision.allowed', decision.allowed);
 setAttribute(span, 'webdecoy.rules.evaluated', decision.results.length);
 if (decision.ruleResult?.rule) {
 setAttribute(span, 'webdecoy.decision.rule', decision.ruleResult.rule);
 }
 // A detection id that is not the decision id means the verdict came back
 // from ingest rather than being settled locally.
 setAttribute(
 span,
 'webdecoy.remote',
 decision.detection.detection_id !== decision.id,
 );
 if (decision.error) {
 setAttribute(span, 'webdecoy.error', decision.error);
 }
 return decision;
 } catch (error) {
 // decide() fails open rather than throwing, so this is a bug rather than
 // a bad day — worth marking on the span rather than swallowing.
 recordError(span, error);
 throw error;
 } finally {
 endSpan(span);
 }
 }

 private async decide(
Expand Down
Loading
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
chore(bots): regenerate registry for xAI-SearchBot and LinkupBot#39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

cport1
merged 3 commits into

from

Aug 25, 2026
 Merged


chore(bots): regenerate registry for xAI-SearchBot and LinkupBot
#39
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


3 commits


Select commit
Hold shift + click to select a range
f059bb0

feat(sdk): OpenTelemetry spans, without depending on OpenTelemetry


cport1 Aug 22, 2026
dd6b897

chore(bots): regenerate registry for xAI-SearchBot and LinkupBot


cport1 Aug 25, 2026
bed3035

chore(bots): regenerate registry for the five newly identified crawlers


cport1 Aug 25, 2026

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


8 changes: 8 additions & 0 deletions
CHANGELOG.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added

- **OpenTelemetry spans** around `protect()` and rule evaluation. Pass a tracer: `new WebDecoy({ tracer: trace.getTracer('webdecoy') })`. Injected rather than imported, so the package stays dependency-free and edge-safe — the `Tracer` type is a structural subset of OpenTelemetry's, so `trace.getTracer()` works with no adapter, and omitting it means no spans, no dependency and no behaviour change. Attributes cover the decision id (which joins a span to its dashboard row), the conclusion, the deciding rule, and whether the request cost a round trip to ingest. A tracer that throws cannot fail a request.

### Changed

- **One adapter core.** Express, Fastify, Next.js (middleware and Pages wrapper) and the fetch guard each carried their own copy of skip-path matching, the 429 and 403 payloads, and honeytoken arming — five copies of one set of decisions, and five places the next correction can fail to land. They now share `adapter-core.ts`; the framework-specific response mechanics are untouched, and every honeytoken-injection test passes unchanged. Fastify keeps its awaited arming, which has no window where early requests are served without the link.

## [0.13.0] - 2026-08-22

### Added
Expand Down

25 changes: 25 additions & 0 deletions
README.md

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -512,6 +512,31 @@ new WebDecoy({ logger: fromPino(pino()) }); // pino's argument order is reverse
`fromPino()` exists because passing a pino instance directly type-checks and then
silently drops every structured field.

## Tracing

Pass an OpenTelemetry tracer and `protect()` emits a span, with a child span for
rule evaluation:

```typescript
import { trace } from '@opentelemetry/api';

new WebDecoy({ tracer: trace.getTracer('webdecoy') });
```

The tracer is **injected, not imported** — this package has no dependencies and
runs on Workers and Vercel Edge, where a stray transitive import is expensive.
The `Tracer` type is a structural subset of OpenTelemetry's, so
`trace.getTracer()` satisfies it with no adapter. Omit it and there are no spans,
no dependency, and no behaviour change.

Attributes are the questions an operator actually asks: `decision.id` (which
joins the span to the dashboard row), `decision.conclusion`, `decision.rule`,
`rules.evaluated`, and `webdecoy.remote` — whether the request cost a round trip
to ingest or was settled locally.

A tracer that throws cannot fail a request. Observability that can take the
request path down is worse than none.

## Examples

See [examples](./examples) for complete working setups — e.g. [express-basic](./examples/express-basic).
Expand Down

2 changes: 1 addition & 1 deletion
packages/webdecoy/src/bots/bots.test.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -60,7 +60,7 @@ describe('matchUserAgent', () => {

describe('registry integrity', () => {
 it('carries the full generated table', () => {
 expect(BOT_REGISTRY.length).toBe(169);
 expect(BOT_REGISTRY.length).toBe(175);
 expect(BOT_CATEGORIES).toContain('training_crawler');
 expect(BOT_CATEGORIES).toContain('search_crawler');
 });
Expand Down

72 changes: 72 additions & 0 deletions
packages/webdecoy/src/bots/parity-vectors.generated.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -575,6 +575,18 @@
 "category": "seo_crawler",
 "matched": true
 },
 {
 "userAgent": "duckassistbot",
 "id": "duckassistbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; duckassistbot/1.0; +http://example.com/bot)",
 "id": "duckassistbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "duckduckbot",
 "id": "duckduckbot",
Expand DownExpand Up@@ -935,6 +947,18 @@
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "iboubot",
 "id": "iboubot",
 "category": "search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; iboubot/1.0; +http://example.com/bot)",
 "id": "iboubot",
 "category": "search_crawler",
 "matched": true
 },
 {
 "userAgent": "iframely",
 "id": "iframely",
Expand DownExpand Up@@ -1055,6 +1079,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "linkupbot",
 "id": "linkupbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; linkupbot/1.0; +http://example.com/bot)",
 "id": "linkupbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "mj12bot",
 "id": "mj12bot",
Expand DownExpand Up@@ -1559,6 +1595,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "reflectionbot",
 "id": "reflectionbot",
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; reflectionbot/1.0; +http://example.com/bot)",
 "id": "reflectionbot",
 "category": "training_crawler",
 "matched": true
 },
 {
 "userAgent": "rogerbot",
 "id": "rogerbot",
Expand DownExpand Up@@ -1787,6 +1835,18 @@
 "category": "fetcher",
 "matched": true
 },
 {
 "userAgent": "sofyabot",
 "id": "sofyabot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; sofyabot/1.0; +http://example.com/bot)",
 "id": "sofyabot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "sogou",
 "id": "sogou",
Expand DownExpand Up@@ -2147,6 +2207,18 @@
 "category": "generic_scraper",
 "matched": true
 },
 {
 "userAgent": "xai-searchbot",
 "id": "xai-searchbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (compatible; xai-searchbot/1.0; +http://example.com/bot)",
 "id": "xai-searchbot",
 "category": "ai_search_crawler",
 "matched": true
 },
 {
 "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
 "matched": false
Expand Down

8 changes: 7 additions & 1 deletion
packages/webdecoy/src/bots/registry.generated.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@
 * agents share User-Agent substrings, so sorting this array changes how real
 * traffic is classified.
 *
 * 169 agents across 15 categories.
 * 175 agents across 15 categories.
 */

/**
Expand DownExpand Up@@ -75,6 +75,7 @@ export const BOT_CATEGORIES: readonly BotCategory[] = [

/** The agent table, in match order. */
export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "reflectionbot", name: "Reflectionbot", category: "training_crawler", organization: "Reflection AI", baseScore: 70, respectsRobots: true, uaPatterns: ["reflectionbot"] },
 { id: "gptbot", name: "GPTBot", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["gptbot"] },
 { id: "chatgpt-user", name: "ChatGPT-User", category: "training_crawler", organization: "OpenAI", baseScore: 85, respectsRobots: true, uaPatterns: ["chatgpt-user", "chatgpt"] },
 { id: "oai-searchbot", name: "OAI-SearchBot", category: "training_crawler", organization: "OpenAI", baseScore: 80, respectsRobots: true, uaPatterns: ["oai-searchbot"] },
Expand DownExpand Up@@ -103,6 +104,10 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "velenpublicwebcrawler", name: "VelenPublicWebCrawler", category: "training_crawler", organization: "Velen", baseScore: 70, respectsRobots: false, uaPatterns: ["velenpublicwebcrawler"] },
 { id: "isscyberriskcrawler", name: "ISSCyberRiskCrawler", category: "training_crawler", organization: "ISS", baseScore: 65, respectsRobots: false, uaPatterns: ["isscyberriskcrawler"] },
 { id: "friendlycrawler", name: "FriendlyCrawler", category: "training_crawler", organization: "Unknown", baseScore: 65, respectsRobots: true, uaPatterns: ["friendlycrawler"] },
 { id: "duckassistbot", name: "DuckAssistBot", category: "ai_search_crawler", organization: "DuckDuckGo", baseScore: 65, respectsRobots: true, uaPatterns: ["duckassistbot"] },
 { id: "sofyabot", name: "SofyaBot", category: "ai_search_crawler", organization: "Sofya", baseScore: 65, respectsRobots: true, uaPatterns: ["sofyabot"] },
 { id: "xai-searchbot", name: "xAI-SearchBot", category: "ai_search_crawler", organization: "xAI", baseScore: 70, respectsRobots: true, uaPatterns: ["xai-searchbot"] },
 { id: "linkupbot", name: "LinkupBot", category: "ai_search_crawler", organization: "Linkup", baseScore: 65, respectsRobots: true, uaPatterns: ["linkupbot"] },
 { id: "searchgpt", name: "SearchGPT", category: "ai_search_crawler", organization: "OpenAI", baseScore: 75, respectsRobots: true, uaPatterns: ["searchgpt"] },
 { id: "phind", name: "Phind", category: "ai_search_crawler", organization: "Phind", baseScore: 70, respectsRobots: true, uaPatterns: ["phind"] },
 { id: "kagi", name: "Kagi", category: "ai_search_crawler", organization: "Kagi", baseScore: 65, respectsRobots: true, uaPatterns: ["kagi"] },
Expand All@@ -119,6 +124,7 @@ export const BOT_REGISTRY: readonly BotAgent[] = [
 { id: "browser-use", name: "Browser-Use", category: "ai_agent", organization: "Open Source", baseScore: 70, respectsRobots: false, uaPatterns: ["browser-use"] },
 { id: "stagehand", name: "Stagehand", category: "ai_agent", organization: "Browserbase", baseScore: 70, respectsRobots: false, uaPatterns: ["stagehand"] },
 { id: "multion", name: "MultiOn", category: "ai_agent", organization: "MultiOn", baseScore: 70, respectsRobots: false, uaPatterns: ["multion"] },
 { id: "iboubot", name: "IbouBot", category: "search_crawler", organization: "Ibou", baseScore: 30, respectsRobots: true, uaPatterns: ["iboubot"] },
 { id: "googlebot", name: "Googlebot", category: "search_crawler", organization: "Google", baseScore: 30, respectsRobots: true, uaPatterns: ["googlebot"] },
 { id: "bingbot", name: "Bingbot", category: "search_crawler", organization: "Microsoft", baseScore: 30, respectsRobots: true, uaPatterns: ["bingbot"] },
 { id: "yandexbot", name: "YandexBot", category: "search_crawler", organization: "Yandex", baseScore: 35, respectsRobots: true, uaPatterns: ["yandexbot"] },
Expand Down

1 change: 1 addition & 0 deletions
packages/webdecoy/src/index.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -84,6 +84,7 @@ export type { BotVerdict, BotAgent, BotCategory } from './bots';
// runtime with a fetch handler. `@webdecoy/hono` is a thin wrapper over it; Bun,
// Deno, Astro and Nitro need no package at all.
export { consoleLogger, silentLogger, fromPino } from './logger';
export type { Tracer, Span } from './tracing';
export type { Logger, LogFields } from './logger';

// Browser signals, joined to the requests that follow them. See client-signals.ts
Expand Down

51 changes: 49 additions & 2 deletions
packages/webdecoy/src/sdk.ts

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -13,6 +13,8 @@ import { AgentVerifier } from './agent/verifier';
import type { AgentRequestInput, AgentVerdict } from './agent/types';
import { readEdgeVerdict } from './edge';
import { resolveLogger } from './logger';
import { startSpan, setAttribute, recordError, endSpan } from './tracing';
import type { Tracer } from './tracing';
import type { Logger } from './logger';
import { Decision, newDecisionId } from './decision';
import type { Conclusion } from './decision';
Expand All@@ -33,7 +35,7 @@ export class WebDecoy {
 private client: WebDecoyClient | null;
 private config: Omit<
 Required<WebDecoyConfig>,
 'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger'
 'apiKey' | 'rules' | 'webBotAuth' | 'characteristics' | 'decisionCache' | 'logger' | 'tracer'
 > & {
 apiKey?: string;
 };
Expand All@@ -48,6 +50,8 @@ export class WebDecoy {
 private readonly characteristics: readonly import('./characteristics').Characteristic[];
 /** Where diagnostics go. Never console directly — see logger.ts. */
 readonly log: Logger;
 /** Optional OpenTelemetry tracer. Absent means no spans and no cost. */
 private readonly tracer?: Tracer;
 private readonly decisionCache: DecisionCache | null;

 constructor(config: WebDecoyConfig) {
Expand All@@ -74,6 +78,7 @@ export class WebDecoy {
 };

 this.log = resolveLogger(config.logger, this.config.debug);
 this.tracer = config.tracer;

 // Initialize API client only when apiKey is provided
 if (hasApiKey) {
Expand DownExpand Up@@ -253,7 +258,15 @@ export class WebDecoy {
 /** Evaluate rules against a prepared context and report any violations. */
 private runRules(context: RuleContext): RuleEngineResult | null {
 if (!this.ruleEngine) return null;

 const span = startSpan(this.tracer, 'webdecoy.rules');
 const result = this.ruleEngine.evaluate(context);
 setAttribute(span, 'webdecoy.rules.action', result.action);
 setAttribute(span, 'webdecoy.rules.evaluated', result.results.length);
 setAttribute(span, 'webdecoy.rules.violations', result.violations.length);
 if (result.rule) setAttribute(span, 'webdecoy.rules.deciding', result.rule);
 endSpan(span);

 if (result.violations.length > 0 && this.violationReporter) {
 this.violationReporter.report(result.violations);
 }
Expand DownExpand Up@@ -335,7 +348,41 @@ export class WebDecoy {
 // present on every outcome — and a per-return copy is a line someone would
 // eventually forget on the branch that mattered.
 const edge = readEdgeVerdict(metadata.headers);
 return (await this.decide(metadata, options)).withEdge(edge);

 const span = startSpan(this.tracer, 'webdecoy.protect');
 try {
 const decision = (await this.decide(metadata, options)).withEdge(edge);

 // Attributes chosen so a trace answers the questions an operator actually
 // asks: what did we decide, which rule decided it, and did this request
 // cost a round trip to ingest. The decision id joins the span to the
 // dashboard row.
 setAttribute(span, 'webdecoy.decision.id', decision.id);
 setAttribute(span, 'webdecoy.decision.conclusion', decision.conclusion);
 setAttribute(span, 'webdecoy.decision.allowed', decision.allowed);
 setAttribute(span, 'webdecoy.rules.evaluated', decision.results.length);
 if (decision.ruleResult?.rule) {
 setAttribute(span, 'webdecoy.decision.rule', decision.ruleResult.rule);
 }
 // A detection id that is not the decision id means the verdict came back
 // from ingest rather than being settled locally.
 setAttribute(
 span,
 'webdecoy.remote',
 decision.detection.detection_id !== decision.id,
 );
 if (decision.error) {
 setAttribute(span, 'webdecoy.error', decision.error);
 }
 return decision;
 } catch (error) {
 // decide() fails open rather than throwing, so this is a bug rather than
 // a bad day — worth marking on the span rather than swallowing.
 recordError(span, error);
 throw error;
 } finally {
 endSpan(span);
 }
 }

 private async decide(
Expand Down
Loading
Loading