feat(indexing): port extension→language detection from semble

[amondnet]

Port of `src/semble/index/files.py` → `src/indexing/files.ts`. This is Unit 4 of the parallel TS port effort.

What ships

• `EXTENSION_TO_LANGUAGE` — full 350+ entry record, no entries skipped (including the commented-out `.txt` mapping which is preserved as a TS comment).
• `DOC_LANGUAGES`, `CONFIG_LANGUAGES`, `DATA_LANGUAGES`, `ALL_LANGUAGES` as `ReadonlySet` (mirroring Python `frozenset`).
• `detectLanguage(fileName: string): string | undefined` — mirrors Python's `Path(name).suffix.lower()` lookup. Case-insensitive on the suffix; for dotfiles like `.gitignore` it returns `undefined` (Python's `Path('.gitignore').suffix` is `''`).
• `getExtensions(types, extensions)` — resolves `ContentType[]` to a sorted, deduplicated extension list, optionally unioning user-provided extensions.
• Inline `ContentType = 'code' | 'docs' | 'config'` (will be re-exported from `src/types.ts` once that lands).

Conventions preserved

• The internal `CODE_LANGUAGES` set is computed as `ALL − DOCS − CONFIG − DATA`, matching the upstream definition. `CONTENT_TYPE_LANGUAGES` keys map to `code`/`docs`/`config` for parity with `ContentType.CODE | DOCS | CONFIG`.
• Sort uses default lexicographic comparison to match Python's `sorted(set)`.

Tests

`bun test src/indexing/files.test.ts` — 22 pass, 0 fail. Cases:

• Common extensions (`.ts`, `.tsx`, `.py`, `.md`).
• Case-insensitivity (`Foo.TS` → `typescript`).
• Multi-dot filenames (`foo.bar.ts` → `typescript`).
• Path with directory separators (`src/indexing/files.ts`).
• Files without an extension (`Makefile` → undefined).
• Dotfile semantics (`.gitignore` → undefined, matching Python).
• `getExtensions` for each content type, with and without user extensions, including union across types.
• All exported sets are non-empty, and every value in `EXTENSION_TO_LANGUAGE` is in `ALL_LANGUAGES`.

Out of scope

• No changes to `package.json`.
• No e2e tests (per the unit brief).

---

Summary by cubic

Ports src/semble/index/files.py to TypeScript as src/indexing/files.ts for consistent extension→language detection in indexing. Mirrors Python behavior and ships with 22 unit tests.

• New Features
  • EXTENSION_TO_LANGUAGE mapping (350+ entries).
  • DOC_LANGUAGES, CONFIG_LANGUAGES, DATA_LANGUAGES, ALL_LANGUAGES as ReadonlySet<string>.
  • detectLanguage(fileName) with case-insensitive suffix, dotfile-safe handling, and POSIX/Windows path support.
  • getExtensions(types, extensions) returns a sorted, deduped union; includes inline ContentType = 'code' | 'docs' | 'config'.

^{Written for commit 7d4a229. Summary will update on new commits.}

[gemini-code-assist]

Code Review

This pull request introduces file extension mapping and language detection utilities in src/indexing/files.ts, along with comprehensive tests in src/indexing/files.test.ts. Feedback highlights a cross-platform compatibility issue in detectLanguage where Windows backslash path separators (\) are not handled, potentially leading to incorrect extension parsing. It is recommended to support both path separators and add corresponding test cases to prevent regressions.

[cubic-dev-ai]

1 issue found across 2 files

Architecture diagram

sequenceDiagram
participant Caller as Caller (e.g., indexer)
participant Detect as detectLanguage(fileName)
participant ExtMap as EXTENSION_TO_LANGUAGE
participant GetExt as getExtensions(types, extras)
participant TypeSets as CONTENT_TYPE_LANGUAGES
Note over Caller,TypeSets: Language Detection Flow (NEW)
Caller->>Detect: detectLanguage("foo.ts")
Detect->>Detect: Extract suffix (".ts"), lowercase
Detect->>ExtMap: Lookup ".ts"
ExtMap-->>Detect: "typescript"
Detect-->>Caller: "typescript"
alt Unknown extension
Caller->>Detect: detectLanguage("foo.xyz")
Detect->>Detect: Extract suffix ".xyz", lowercase
Detect->>ExtMap: Lookup ".xyz"
ExtMap-->>Detect: undefined
Detect-->>Caller: undefined
end
alt Dotfile (e.g., ".gitignore")
Caller->>Detect: detectLanguage(".gitignore")
Detect->>Detect: Extract suffix "" (empty)
Detect->>ExtMap: Lookup ""
ExtMap-->>Detect: undefined
Detect-->>Caller: undefined
end
Note over Caller,TypeSets: Extension Resolution Flow (NEW)
Caller->>GetExt: getExtensions(["code"], undefined)
GetExt->>TypeSets: Lookup "code" set
TypeSets-->>GetExt: CODE_LANGUAGES
GetExt->>GetExt: Map languages to extensions via LANGUAGE_TO_EXTENSIONS
GetExt->>GetExt: Sort and deduplicate
GetExt-->>Caller: [".ts", ".py", ".go", ...]
alt With user-provided extensions
Caller->>GetExt: getExtensions(["config"], [".custom"])
GetExt->>TypeSets: Lookup "config" set
TypeSets-->>GetExt: CONFIG_LANGUAGES
GetExt->>GetExt: Merge ".custom" into extension list
GetExt->>GetExt: Sort and deduplicate
GetExt-->>Caller: [".custom", ".toml", ".yaml", ...]
end
alt Union multiple content types
Caller->>GetExt: getExtensions(["code", "docs"], undefined)
GetExt->>TypeSets: Lookup "code" and "docs" sets
TypeSets-->>GetExt: CODE_LANGUAGES, DOC_LANGUAGES
GetExt->>GetExt: Union language sets, map to extensions
GetExt->>GetExt: Sort and deduplicate
GetExt-->>Caller: [".md", ".py", ".ts", ...]
end
Note over Caller,TypeSets: Internal Set Computation (at module init)
Note over TypeSets: CODE_LANGUAGES = ALL_LANGUAGES \ DOC \ CONFIG \ DATA

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[amondnet]

Applied 3 (Windows path separator support in detectLanguage + matching tests), deferred 0. All 3 bot comments addressed the same root cause: detectLanguage only handled '/' and would misclassify Windows-style paths. Fix uses Math.max(lastIndexOf('/'), lastIndexOf('\')) so .gitignore behind 'dir\' resolves to undefined and 'C:\Users\me\foo.py' returns 'python', mirroring pathlib.Path on Windows. 23/23 tests pass.