Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions .changeset/tap-thread-method.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
'@toolpath/tool-support': minor
'@toolpath/tool-scraper': minor
---

Record whether a tap cuts its thread or forms it.

`Tool` takes an optional `threadMethod`, `'cutting' | 'forming'`, beside `form`
rather than as new `TOOL_FORMS` values — the form vocabulary stays Fusion's, and
Fusion has no form-tap type. `ToolRecord.threadMethod` carries the same value on
a tap and `null` on every other kind; `toolRecord` refuses a tap without one and
a non-tap with one.

Every tap family now states it as a cited fact: Kennametal's three from its
`newTapType` facet, EMUGE's `FG01` from the category it titles `Machine taps`.
And EMUGE's cold-forming taps are scraped for the first time —
`emuge_form_taps.csv`, category `FG02`, 1,432 parts — so `forming` is a value the
catalog actually holds rather than one only the type admits.
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,7 @@ judgment rule starts being violated, give it a check rather than restating it he
| A scraper vendor adapter imports no other vendor | `pnpm test` (`vendor-boundary`) |
| Only a composition root reaches into `src/vendors/` | `pnpm test` (`vendor-boundary`) |
| Every scraper vendor directory has a `scrape.ts` | `pnpm test` (`vendor-boundary`) |
| Every tap family states whether it cuts its thread or forms it | `pnpm test` (`tap-method`) |
| `@toolpath/tool-support` imports nothing and declares no dependency | `pnpm test` (`boundary`) |
| `@toolpath/app-support`'s root entry imports no React | `pnpm test` (`boundary`) |
| `@toolpath/app-support` never imports `@toolpath/ui` | `pnpm test` (`boundary`) |
Expand Down
26 changes: 18 additions & 8 deletions packages/tool-scraper/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,14 @@ pnpm add @toolpath/tool-scraper

## Vendors

| Vendor | Transport | What it publishes |
| ------------------ | ------------------------------------------------------- | ------------------------------------ |
| Kennametal / WIDIA | AEM variant-table GET, parsed with `htmlparser2` | tools and toolholding |
| REGO-FIX | Elasticsearch proxy POST + per-part DIN 4000 XML | toolholding |
| Destiny Tool | Firestore REST, paginated | solid end mills |
| Harvey Tool | inline JS literal on a product page, plus its `<thead>` | miniature end mills, keyseat cutters |
| MariTool | osCommerce category listings, then one page per part | toolholding |
| EMUGE-FRANKEN | SAP Commerce JSON API: grouped, variant, batched detail | end mills, twist drills, taps |
| Vendor | Transport | What it publishes |
| ------------------ | ------------------------------------------------------- | ------------------------------------------------- |
| Kennametal / WIDIA | AEM variant-table GET, parsed with `htmlparser2` | tools and toolholding |
| REGO-FIX | Elasticsearch proxy POST + per-part DIN 4000 XML | toolholding |
| Destiny Tool | Firestore REST, paginated | solid end mills |
| Harvey Tool | inline JS literal on a product page, plus its `<thead>` | miniature end mills, keyseat cutters |
| MariTool | osCommerce category listings, then one page per part | toolholding |
| EMUGE-FRANKEN | SAP Commerce JSON API: grouped, variant, batched detail | end mills, twist drills, taps cutting and forming |

## Two entry points

Expand Down Expand Up @@ -77,6 +77,7 @@ toolpath-scrape harvey --catalog # what the four category trees
toolpath-scrape maritool maritool_cat40_holders.csv # its leaf categories come from its config
toolpath-scrape maritool --catalog # what the five taper trees hold today
toolpath-scrape emuge emuge_drills.csv # its category and unit come from its config
toolpath-scrape emuge emuge_form_taps.csv # FG02, the cold-forming taps
```

`toolpath-scrape --help` lists the rest.
Expand All @@ -99,6 +100,15 @@ here. Every Harvey record is `unspecified`: Harvey's material index is published
than in a variant table, and a scrape cannot reach it — see
[`docs/HARVEY_PRODUCT_TABLE.md`](docs/HARVEY_PRODUCT_TABLE.md) §1.5.1.

`threadMethod` is `cutting` or `forming` on a tap and `null` on everything else,
because the question does not apply to a drill. It is the one field that
separates a thread former from a cut tap: the two are the same `DC`, `TP`,
`SFDM`, `OAL` and `LCF`, often the same substrate and coating, and a shop drills
a larger hole before one than before the other. Neither vendor publishes it in a
variant table — Kennametal states it in a `newTapType` facet and EMUGE in the
split between its `FG01` and `FG02` categories — so it is a per-family fact, and
each family cites the index it was read off.

Every per-family constant no vendor table states carries its provenance — whether it was
vendor-stated, derived or assumed, and by whom on what date. The types enforce it: an assumed fact
without a note, a date and initials does not compile.
Expand Down
46 changes: 44 additions & 2 deletions packages/tool-scraper/docs/EMUGE_FRANKEN_COMMERCE_API.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@ so batching per variant is what a record needs.
| `FF01` | End mill cutters | 554 | 7,021 |
| `FB01` | Twist drills | 17 | 2,670 |
| `FG01` | Machine taps | 414 | 11,566 |
| `FG02` | Cold forming tap | 137 | 1,432 |
| `FF02` | Tool holders and accessories | 80 | 625 |

`FF02` is not scraped — this package takes EMUGE's cutting tools only.
Expand All @@ -164,9 +165,49 @@ Milling is split by unit system with the vendor's own facet,
and `AMM_EINHS_M` metric (5,189). Drilling and tapping have no such facet and
need none — every drill and every tap is published in millimetres.

Roughly 1,700 requests cover the three cutting-tool categories: the group
## 4b. Tapping is two categories, and that is the form/cut split

The vendor files its taps as `FG01` — which it titles **Machine taps** — and
`FG02`, **Cold forming tap**. A tap in the first cuts its thread away; one in the
second displaces material into it. They are the same three calls against the
same column labels, including `length of cutting edge l₂`, which EMUGE keeps on
a tool that has no cutting edge.

`FG02` went unscraped until 2026-09-07. Until then `emuge_taps.csv` was this
package's whole tapping corpus and every row in it was a cutting tap, with
nothing on the record saying so.

**The parts agree with the categories, independently.** Every grouped product
carries a flat `technicalDetails` list, and a tap's names the entry geometry —
a chamfer where it cuts, a lead taper where it forms:

| Category | Property stated | Groups | Groups stating the other |
| --- | --- | --- | --- |
| `FG01` | `chamfer form` | 414 of 414 | 0 |
| `FG02` | `lead taper form` | 137 of 137 | 0 |

Mutually exclusive, both directions, at full coverage (JG 2026-09-07). That is
what `families/emuge.ts` cites for each family's `threadMethod`, and
`tests/emuge-corpus.test.ts` re-checks it against a real scrape — a fact rather
than a mapped column cannot otherwise be contradicted by one.

**Two things that look like the discriminator and are not:**

- **`Geometry` is not one.** `FG02`'s eight values — `AL`, `GAL`, `H`, `MULTI`,
`SPEED`, `STEEL`, `VA`, `Z` — are all `FG01` values too, and mean different
products: a `Z`-geometry former is InnoForm where a `Z`-geometry cutting tap
is Rekord B-Z. This is why `PRODUCT_LINES` has no `FG02` table and the codes
pass through verbatim.
- **`flute characteristic: without` is not one.** The 189 `FG01` variants
carrying it are six EMUGE *Robust* groups — reinforced cutting taps — and
every one of them still states a `chamfer form`.

## 4c. Request cost

Roughly 1,900 requests cover the four cutting-tool categories: the group
listings, one variant call per group, and `ceil(variants / 30)` detail calls.
At the package's 400 ms pacing that is about twelve minutes.
At the package's 400 ms pacing that is about thirteen minutes; `FG02` is about
190 of those requests on its own.

## 4a. The product line, and the facet that partitions each category

Expand All @@ -178,6 +219,7 @@ checked at group level on 2026-09-01 rather than by summing counts:
| `FF01` | `AMM_PROG_LINIE` | `product line` | 15 | 554 of 554 | 2 |
| `FB01` | `HYB_BAM_SB_GT` | `Geometry` | 4 | 17 of 17 | 0 |
| `FG01` | `HYB_BAM_SB_GT` | `Geometry` | 17 | 414 of 414 | 0 |
| `FG02` | `HYB_BAM_SB_GT` | `Geometry` | 8 | 137 of 137 | 0 |

All three reach the CSV without a request being added: `Geometry` is on the
grouped product's `technicalDetails` and `product line` on the per-part detail
Expand Down
50 changes: 49 additions & 1 deletion packages/tool-scraper/docs/MARITOOL_CATALOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,55 @@ calibration/ATC tooling and coolant tubes — 731 further parts under `c23`.
holders. ISO30 has three ER parts and none of them publishes a spec table, so
the family would be a receipt of nothing.

### 2.3 The roster count
### 2.3 The taps, and how they classify themselves

Not scraped. Recorded here because MariTool's tree already answers the question
`ToolRecord.threadMethod` exists for, and because the answer wants writing down
while somebody has read the pages.

Under `c78_148` (`Cutting Tools / Taps`), read 2026-09-07:

| Leaf | cPath | Method |
| --- | --- | --- |
| Thread Forming Taps | `78_148_274` | `forming` |
| — Plug Form Tap | `78_148_274_275` | `forming` |
| — Bottoming Form Tap | `78_148_274_276` | `forming` |
| Spiral Flute Taps | `78_148_149` | `cutting` |
| — Spiral Flute Plug Taps, Spiral Flute Bottoming | | `cutting` |
| Spiral Point Taps | `78_148_224` | `cutting` |
| Taps for Aluminum | `78_148_271` | `cutting` |
| — Sprial Point Plug Taps *(the vendor's typo)*, Spiral Flute Semi Bottoming | | `cutting` |
| Taper Pipe Taps | `78_148_283` | `cutting` |
| DIN Length HPT Taps | `78_148_284` | `cutting` |
| — Plug Style DIN Taps, Bottoming Style DIN Taps | | `cutting` |

**Only the forming branch carries the word**, in the leaf name and in every
product title under it — `Plug Style Thread Forming Tap Bright Finish 10-24 H4`.
So `form`/`forming` in the name is the rule, applied when the table is built and
checked into it, never run as a filter at scrape time. §2.2 already gives the
reason: three ER leaves are named `Collet Chucks` rather than `ER Collet Chucks`
and a name match would drop all three silently.

**What a tap family still needs decided.** A product page publishes enough for a
record — `Shank Size` → `SFDM`, `Total Length` → `OAL`, `Thread Length` → `LCF`,
`Size` + `Pitch Diameter` → a designation `thread.ts` already parses, and no
`DC`, which is the Kennametal case. What it also does is state a **metric thread
on an inch body**:

```
Size: M3 Pitch Diameter: 0.5 # millimetres
Shank Size: 0.141 Total Length: 1 15/16 # inches
Thread Limit: D5 Finish: TiN
```

`Pitch Diameter` is the vendor's label for what is a pitch on a metric tap and a
thread count on an inch one (`Size: 10`, `Pitch Diameter: 24`). No family in
this package is mixed that way and `ToolRecord.unit` is one value per record, so
this is its own decision — and it is the hazard
`tool-support/src/geometry.ts`'s `TP` docstring already records rather than
resolves.

### 2.4 The roster count

`Displaying 1 to 30 (of 51 products)`, and `Result Pages` paging on `page=N`.
30 products per page; the 41 leaves in scope need **47 listing pages**.
Expand Down
127 changes: 112 additions & 15 deletions packages/tool-scraper/src/families/emuge.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
/**
* EMUGE-FRANKEN's families.
*
* Four, one per catalog category crossed with the unit system the category is
* Five, one per catalog category crossed with the unit system the category is
* published in. That is coarser than the vendor's own marketing, which splits
* end mills fifteen ways by product line — TOP-Cut, Hard-Cut, Alu-Cut — and it
* is deliberate: EMUGE states the product line, the cutting material, the
Expand All @@ -12,8 +12,22 @@
* `PRODUCT_LINE_COLUMNS` reads one per part onto `ToolRecord.productLine`, from
* a column every scrape already writes.
*
* So the only fact three of these four state is `unit`, and the fourth adds the
* two a drill record cannot be built without.
* So the only fact the two milling families state is `unit`; the drill family
* adds the two a drill record cannot be built without, and each tap family adds
* how its taps make a thread — see `MACHINE_TAP` and `COLD_FORMING_TAP`.
*
* ## The tapping split, which is not a unit split
*
* Tapping is the one category EMUGE publishes as **two**: `FG01`, which it
* titles `Machine taps`, and `FG02`, `Cold forming tap`. They are the same
* three calls against the same column labels and differ only in what the tools
* do — one cuts the thread away, the other displaces material into it — so
* they are two families and not two adapters, and the category is what each
* one's `threadMethod` fact cites.
*
* `FG02` went unscraped until 2026-09-07, which meant `emuge_taps.csv` was the
* whole of this package's tapping corpus and every row in it was a cutting tap
* with nothing recording that it was.
*
* ## `rows`
*
Expand All @@ -35,7 +49,10 @@
* system to read, because the vendor states one system for all of them.
*/

import type { UnitSystem } from '../conventions.js'
import type { FamilyDefinition } from '../family.js'
import type { Fact } from '../provenance.js'
import type { ThreadMethod } from '../records.js'

/**
* The facet EMUGE indexes milling variants by unit system under.
Expand Down Expand Up @@ -78,8 +95,80 @@ export const SCRAPE_TARGETS = {
},
'emuge_drills.csv': { category: 'FB01' },
'emuge_taps.csv': { category: 'FG01' },
'emuge_form_taps.csv': { category: 'FG02' },
} as const satisfies Record<string, Target>

/**
* Tapping geometry, identical either side of the cutting/forming split.
*
* **EMUGE labels a cold-forming tap's lead `length of cutting edge l₂` too**,
* and it reaches the CSV under that label, because a scraped column keeps the
* vendor's own name — `conventions.ts` opens on the reason. A former has no
* cutting edge and the label is wrong about it; renaming it here would swap one
* vendor's inaccuracy for this package's invention, and the record's `LCF` is
* the canonical name either way.
*/
const TAP_COLUMNS = {
DC: 'nominal diameter d₁',
SFDM: 'Shank diameter d₂',
OAL: 'Overall length l₁',
LCF: 'length of cutting edge l₂',
TP: 'pitch',
} as const

/**
* Which of EMUGE's two tap categories a family scrapes, and how it is known.
*
* The vendor splits its taps into two categories and scraping either one is
* already the answer — but the *parts* say so as well, independently, which is
* what makes these `vendor-stated` rather than a claim about a URL. Every
* grouped product carries a flat `technicalDetails` list, and:
*
* - all 414 `FG01` groups state `chamfer form` and none states `lead taper
* form`;
* - all 137 `FG02` groups state `lead taper form` and none states `chamfer
* form`.
*
* Mutually exclusive, both directions, at full coverage (JG 2026-09-07). A
* chamfer is ground onto a tap that cuts and a lead taper is rolled onto one
* that forms, so the vendor is naming the same distinction twice.
* `tests/emuge-corpus.test.ts` re-checks that agreement against a real scrape,
* which is what keeps these two constants honest without either family reading
* a column.
*
* **Two things that look like this discriminator and are not**, recorded so
* nobody reaches for them: `Geometry` — the column `vendors/emuge/records.ts`
* reads as the product line — takes `AL`, `GAL`, `H`, `MULTI`, `SPEED`,
* `STEEL`, `VA` and `Z` in *both* categories; and the 189 `FG01` variants whose
* `flute characteristic` is `without` are six EMUGE *Robust* groups, reinforced
* cutting taps that still state a `chamfer form`.
*/
/**
* The unit both tap families are published in.
*
* Shared because the vendor's rule is about tapping and not about either
* category: a `#4-40 UNC` tap states millimetres whichever of the two it sits
* in, which is the same thing the module note says about there being no unit
* facet on tapping at all.
*/
const TAP_MILLIMETERS = {
value: 'millimeters',
source: 'vendor-stated',
cite: 'every tap dimension is published in millimetres whatever the thread standard — a `#4-40 UNC` tap states `nominal diameter d₁ [mm]` as `2.845 mm`, `pitch [mm]` as `0.635 mm`, and its shank and lengths in `mm` — with `thread symbol`, `nominal size` and `threads per inch` carrying the inch designation beside them',
} as const satisfies Fact<UnitSystem>

const MACHINE_TAP = {
value: 'cutting',
source: 'vendor-stated',
cite: "the vendor's own category `FG01`, which it titles `Machine taps`; and independently every one of its 414 grouped products states a `chamfer form` and none states a `lead taper form` (JG 2026-09-07)",
} as const satisfies Fact<ThreadMethod>

const COLD_FORMING_TAP = {
value: 'forming',
source: 'vendor-stated',
cite: "the vendor's own category `FG02`, which it titles `Cold forming tap`; and independently every one of its 137 grouped products states a `lead taper form` and none states a `chamfer form` (JG 2026-09-07)",
} as const satisfies Fact<ThreadMethod>

/** Milling geometry, identical either side of the unit split. */
const MILLING_COLUMNS = {
DC: 'cutting diameter Ød₁',
Expand Down Expand Up @@ -171,20 +260,28 @@ export const FAMILIES = {
// `records.DIMENSIONAL_COLUMNS` excluding it means: the vendor publishes
// `pitch [mm]` and this family is millimetres, so the column is already in
// the record's native unit.
columns: {
DC: 'nominal diameter d₁',
SFDM: 'Shank diameter d₂',
OAL: 'Overall length l₁',
LCF: 'length of cutting edge l₂',
TP: 'pitch',
},
columns: TAP_COLUMNS,
rows: 11566,
facts: {
unit: {
value: 'millimeters',
source: 'vendor-stated',
cite: 'every tap dimension is published in millimetres whatever the thread standard — a `#4-40 UNC` tap states `nominal diameter d₁ [mm]` as `2.845 mm`, `pitch [mm]` as `0.635 mm`, and its shank and lengths in `mm` — with `thread symbol`, `nominal size` and `threads per inch` carrying the inch designation beside them',
},
unit: TAP_MILLIMETERS,
threadMethod: MACHINE_TAP,
},
},
// The second half of the vendor's own tapping catalog, and the only forming
// taps this package reaches. It scrapes the same three calls as `FG01`
// against the same column labels — see `TAP_COLUMNS` — so it is a family
// rather than an adapter: what differs is the category, and what the
// category settles.
'emuge_form_taps.csv': {
id: 'form-taps',
brand: 'emuge',
kind: 'tap',
familyCode: 'FG02',
columns: TAP_COLUMNS,
rows: 1432,
facts: {
unit: TAP_MILLIMETERS,
threadMethod: COLD_FORMING_TAP,
},
},
} as const satisfies Record<string, FamilyDefinition>
Loading
Loading