Goal
Extend the online-data USB-VID aggregator (the pipeline introduced in #718 / #719) with per-board VID:PID coverage by ingesting structured board-manifest data from upstream vendors. Today the aggregator carries:
- Vendor names (via the merged
usb-vid.json — text DBs + the curated vendor_names_inlined.py overlay covering 253 newer VIDs). - PlatformIO board catalog (
pio-boards.json — names + MCU family but no VID:PID). - MCU → VID heuristics (
mcu_to_vid.json — confidence-scored, not per-board).
What's still missing for the headline VID:PID → board lookup: a precise per-board VID:PID table that says e.g. "0x303A:0x4002 is an ESP32-S3" rather than "any ESP32-S3-family board on VID 0x303A."
Authoritative public sources (in priority order)
| Tier | Source | Format | Coverage | Type |
|---|
| 1 | raspberrypi/usb-pid → Readme.md | Markdown table | VID 0x2E8A PID allocations (Pico, Pimoroni, Cytron, WIZnet, TI USB-TMC etc.) | PID allocation registry |
| 1 | espressif/usb-pids → allocated-pids.txt + allocated-pids-espressif-devboards.txt | Plain text (VVVV:PPPP description) | VID 0x303A customer & devboard PIDs (TinyS2, LILYGO, ATMegaZero, ESP32-S3-DevKitC, ESP32-P4 etc.) | PID allocation registry |
| 2 | arduino/ArduinoCore-avr → boards.txt | boardname.vid.N=0xVVVV / pid.N=0xPPPP | Leonardo, Micro, Esplora, Yún (VID 0x2341, 0x2A03) | Board-manifest |
| 2 | arduino/ArduinoCore-samd → boards.txt | Same | Zero, MKR family, Nano 33 IoT (VID 0x03EB, 0x2341) | Board-manifest |
| 2 | espressif/arduino-esp32 → boards.txt | Same | ESP32-S2/S3/C3 family + partner boards (UM TinyS2, LOLIN S2 Mini, etc.) | Board-manifest |
| 2 | adafruit/Adafruit_nRF52_Arduino → boards.txt | Same | Feather nRF52840 Express/Sense, ItsyBitsy, Circuit Playground Bluefruit, CLUE (VID 0x239A) | Board-manifest |
| 2 | SiliconLabs/arduino → boards.txt | Same | Nano Matter, Seeed XIAO MG24 (VID 0x2341, 0x2886) | Board-manifest (official) |
| 3 | stm32duino/Arduino_Core_STM32 → boards.txt | Same | Nucleo, Discovery, Eval, generic STM32 (VID 0x0483, PIDs 0x5740 / 0x3744 / 0x3748 / 0x374B / 0x3753) | Board-manifest (community, not ST-official) |
| 4 | NordicSemiconductor/nrf-udev → 71-nrf.rules | udev rules | VID 0x1915 (all Nordic devices) — vendor-level only, no per-product | Auxiliary |
| 4 | NordicSemiconductor/nrf-device-setup-js (archived) → README | Prose | Single explicit pair: 0x1915:0x521F PCA10059 bootloader | Auxiliary, archived 2022-01-10 |
Gaps confirmed by the research
- NXP (MCUXpresso org): no public PID allocation file. Per-example USB descriptors in
nxp-mcuxpresso/mcux-sdk-middleware-usb only. - Microchip (Harmony USB): same shape — per-example USB descriptors, no centralized registry.
- TI: no maintained list.
energia/Energia is the closest but explicitly unmaintained. - STM32: no ST-owned official list.
stm32duino/Arduino_Core_STM32 is community-run.
Proposed implementation
Add a new fetch+parse step per source under online-data-tools/, modeled on the existing fetch_gowdy_supplement.py + overlay_usb_vid.py pattern:
fetch_rpi_pid_registry.py — parses raspberrypi/usb-pid/Readme.md markdown table → {vidpid: product_name} JSON. Tier-1 (most authoritative).fetch_espressif_pid_registry.py — parses espressif/usb-pids/allocated-pids*.txt (two files) → same shape. Tier-1.fetch_boards_txt.py — generic parser for Arduino-style boards.txt. Walks boardname.vid.N + pid.N pairs and emits {vidpid: f"{board_friendly_name}"}. Used for the 6 tier-2/3 board-manifest repos.overlay_per_board_pids.py — merges all per-board sources into a NEW online-data/data/usb-board-pids.json keyed by 8-hex-digit VVVVPPPP, with {vendor, board, source} per entry. Doesn't disturb the existing usb-vid.json.- SQLite consumer —
build_sqlite.py populates vidpid from BOTH the existing usb_product rows AND the new per-board source, with the per-board source winning on conflict (it's the authoritative consumer-facing name). - Workflow wiring — add the new fetches to
update-data.yml with continue-on-error: true (same fault-tolerance pattern as the existing tier-1/2/3 sources).
Output schema sketch
// online-data/data/usb-board-pids.json
{
"2e8a0003": {"vendor": "Raspberry Pi", "board": "RP2040 Boot", "source": "raspberrypi/usb-pid"},
"2e8a000b": {"vendor": "Raspberry Pi", "board": "Pico CircuitPython", "source": "raspberrypi/usb-pid"},
"303a8001": {"vendor": "Unexpected Maker", "board": "TinyS2 Arduino", "source": "espressif/usb-pids"},
"303a7012": {"vendor": "Espressif", "board": "ESP32-P4 Function EV", "source": "espressif/usb-pids"},
"23418057": {"vendor": "Arduino", "board": "Nano 33 IoT", "source": "arduino/ArduinoCore-samd"},
"239a8029": {"vendor": "Adafruit", "board": "Feather nRF52840 Express","source":"adafruit/Adafruit_nRF52_Arduino"},
"23410072": {"vendor": "Arduino", "board": "Nano Matter", "source": "SiliconLabs/arduino"}
}Acceptance criteria
Out of scope (defer to follow-ups)
- Scraping vendor SDK examples for embedded USB descriptors (NXP / Microchip / TI). Would require navigating each vendor's GitHub forest and is a much larger investment.
- Cross-vendor PID-collision detection (e.g. when an OEM ships under multiple VIDs). Add as
usb-board-pid-conflicts.json once the basic ingestion is live. - Adding the new dataset to the embedded
usb-vendors.tar.zst archive in fbuild. The archive is intentionally small (vendor-only); per-PID data stays in the runtime overlay + the www SQLite-over-HTTP backend.
Sub-issue checklist
Per-vendor ingest sub-issues. This meta auto-closes when all are CLOSED.
Tier-1 PID registries (most authoritative)
Tier-2 board-manifests (Arduino-core boards.txt)
End-to-end verification (blocked on all of #723–#739)
References
Goal
Extend the
online-dataUSB-VID aggregator (the pipeline introduced in #718 / #719) with per-board VID:PID coverage by ingesting structured board-manifest data from upstream vendors. Today the aggregator carries:usb-vid.json— text DBs + the curatedvendor_names_inlined.pyoverlay covering 253 newer VIDs).pio-boards.json— names + MCU family but no VID:PID).mcu_to_vid.json— confidence-scored, not per-board).What's still missing for the headline VID:PID → board lookup: a precise per-board VID:PID table that says e.g. "0x303A:0x4002 is an ESP32-S3" rather than "any ESP32-S3-family board on VID 0x303A."
Authoritative public sources (in priority order)
raspberrypi/usb-pid→Readme.mdespressif/usb-pids→allocated-pids.txt+allocated-pids-espressif-devboards.txtVVVV:PPPP description)arduino/ArduinoCore-avr→boards.txtboardname.vid.N=0xVVVV/pid.N=0xPPPParduino/ArduinoCore-samd→boards.txtespressif/arduino-esp32→boards.txtadafruit/Adafruit_nRF52_Arduino→boards.txtSiliconLabs/arduino→boards.txtstm32duino/Arduino_Core_STM32→boards.txtNordicSemiconductor/nrf-udev→71-nrf.rulesNordicSemiconductor/nrf-device-setup-js(archived) →READMEGaps confirmed by the research
nxp-mcuxpresso/mcux-sdk-middleware-usbonly.energia/Energiais the closest but explicitly unmaintained.stm32duino/Arduino_Core_STM32is community-run.Proposed implementation
Add a new fetch+parse step per source under
online-data-tools/, modeled on the existingfetch_gowdy_supplement.py+overlay_usb_vid.pypattern:fetch_rpi_pid_registry.py— parsesraspberrypi/usb-pid/Readme.mdmarkdown table →{vidpid: product_name}JSON. Tier-1 (most authoritative).fetch_espressif_pid_registry.py— parsesespressif/usb-pids/allocated-pids*.txt(two files) → same shape. Tier-1.fetch_boards_txt.py— generic parser for Arduino-styleboards.txt. Walksboardname.vid.N+pid.Npairs and emits{vidpid: f"{board_friendly_name}"}. Used for the 6 tier-2/3 board-manifest repos.overlay_per_board_pids.py— merges all per-board sources into a NEWonline-data/data/usb-board-pids.jsonkeyed by 8-hex-digitVVVVPPPP, with{vendor, board, source}per entry. Doesn't disturb the existingusb-vid.json.build_sqlite.pypopulatesvidpidfrom BOTH the existingusb_productrows AND the new per-board source, with the per-board source winning on conflict (it's the authoritative consumer-facing name).update-data.ymlwithcontinue-on-error: true(same fault-tolerance pattern as the existing tier-1/2/3 sources).Output schema sketch
Acceptance criteria
update-data.yml.online-data/data/usb-board-pids.jsoncommitted with at least 200 per-board entries on first successful run.manifest.jsonadvertises the new dataset (alongsideusb-vid,pio-boards,vendor_boards).vidpidtable is populated from this new source, so a canned-queryvidpid='2e8a000b'returns "Pico CircuitPython" not just "Pico SDK CDC UART".fbuild-core::usb::resolver::pretty(0x303A, 0x4002)resolves through the overlay (tier-2) and returns "Unexpected Maker TinyS2 Arduino" instead of the synthetic"Device 0x4002"placeholder.online-data-tools/README.md.Out of scope (defer to follow-ups)
usb-board-pid-conflicts.jsononce the basic ingestion is live.usb-vendors.tar.zstarchive in fbuild. The archive is intentionally small (vendor-only); per-PID data stays in the runtime overlay + the www SQLite-over-HTTP backend.Sub-issue checklist
Per-vendor ingest sub-issues. This meta auto-closes when all are CLOSED.
Tier-1 PID registries (most authoritative)
Tier-2 board-manifests (Arduino-core boards.txt)
End-to-end verification (blocked on all of #723–#739)
References
dump_usb_idsprotection policy).