Skip to content

Commit ef0a912

Browse files
coryrylancoryrylan
committed
chore(ci): enhance ci performance profiling and add new scripts
- Introduced a new CI performance audit document to analyze and optimize the CI pipeline. - Added a `ci:profile` script to facilitate performance profiling. - Updated `package.json` and `pnpm-lock.yaml` to include new dependencies and scripts. - Enhanced existing scripts for better performance tracking and reporting. - Updated various project configurations to improve build and test processes. Signed-off-by: Cory Rylan <crylan@nvidia.com>
1 parent f9f1e27 commit ef0a912

19 files changed

Lines changed: 2831 additions & 1350 deletions

File tree

‎.agents/skills/profile-ci/SKILL.md‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
---
2+
name: profile-ci
3+
description: Run and analyze Elements cold CI performance profiles and propose evidence-backed build, test, lint, and dependency-graph improvements. Use whenever asked to profile or benchmark pnpm run ci, rerun ci:profile, create a CI performance audit, compare CI timings, find bottlenecks or the completion path, investigate a CI performance regression, or recommend measured CI/build optimizations.
4+
---
5+
6+
# Profile CI
7+
8+
Produce a repeatable cold-CI profile, explain what controls wall-clock completion, and turn the evidence into prioritized, testable recommendations.
9+
10+
## Required context
11+
12+
1. Read the repository `AGENTS.md`.
13+
2. Read `projects/internals/BUILD.md`.
14+
3. Inspect the current root `ci:profile` script and `projects/internals/ci/ci-profile.js`. Treat the script as the profiling source of truth.
15+
4. Read a project's `DEVELOPMENT.md` before running project-specific commands when that file exists.
16+
17+
Do not copy profiling logic into this skill. Update the repository profiler when its behavior needs to change.
18+
19+
## Choose the workflow
20+
21+
- Run a new profile when the user asks to rerun, benchmark current changes, refresh an audit, or verify an optimization.
22+
- Analyze existing `.metrics/ci-profile.{json,md}` artifacts without rerunning when the user asks only for interpretation and the artifacts match the intended commit and worktree state.
23+
- Create recommendations without implementing them unless the user also asks for the changes.
24+
25+
## Profile safely
26+
27+
Run all repository commands through mise.
28+
29+
1. Inspect `git status --short`.
30+
2. Preview ignored files that reset would delete with `git clean -ndX`.
31+
3. Stop and ask before profiling if that preview includes user data, local assets, secrets, or other non-reproducible files. The profiler runs `pnpm run ci:reset`, which deletes ignored files and reinstalls dependencies before every sample.
32+
4. Never stash, commit, discard, or clean tracked changes merely to make the profiler accept the worktree.
33+
5. Use the clean command when the worktree is clean:
34+
35+
```shell
36+
mise exec -- pnpm run ci:profile
37+
```
38+
39+
6. When the dirty changes are the intentional subject of the audit, preserve them and run:
40+
41+
```shell
42+
CI_PROFILE_ALLOW_DIRTY=1 mise exec -- pnpm run ci:profile
43+
```
44+
45+
Record the dirty-worktree condition in the report. Do not use this override for unrelated or unexplained changes.
46+
47+
Allow all three cold samples to finish. Dependency installation or browser setup can require network access. If a run fails, inspect the copied `.metrics/ci-profile-run-*.log` and reset logs, report the incomplete profile, and do not invent missing samples.
48+
49+
## Verify the artifacts
50+
51+
The profiler writes ignored artifacts under `.metrics/`:
52+
53+
-`ci-profile.json`: structured metadata, run durations, and per-script samples
54+
-`ci-profile.md`: generated method and top-ten summary
55+
-`ci-profile-run-{1,2,3}.log`: complete CI logs
56+
-`ci-profile-reset-{1,2,3}.log`: reset and install logs
57+
58+
Before analysis, confirm:
59+
60+
- all three runs exist;
61+
- every run has zero incomplete scripts;
62+
- each ranked script has three samples;
63+
- commit, dirty state, tool versions, CPU, and memory describe the intended environment;
64+
- the generated Markdown agrees with the JSON.
65+
66+
Use JSON as the numeric source of truth. Keep full precision during calculations and round only for presentation.
67+
68+
## Analyze wall-clock relevance
69+
70+
The slowest command is not automatically the critical path.
71+
72+
1. Rank leaf scripts by median duration from `ci-profile.json`.
73+
2. Inspect the end of each run log to identify the scripts that consistently complete last.
74+
3. Trace those scripts through their Wireit `dependencies` in the relevant `package.json` files.
75+
4. Separate:
76+
- final or near-final dependency branches;
77+
- long parallel work that consumes CPU, memory, filesystem, or browser capacity;
78+
- upstream work that delays a final branch;
79+
- composite commands whose phases need separate timing.
80+
5. Inspect each candidate's command, configuration, file count, output, output consumers, and serialization settings before suggesting a change.
81+
6. Compare with the existing audit only when its environment and method are compatible. Describe before/after changes as concurrent cold-CI measurements, not isolated causal proof.
82+
83+
Read [the optimization playbook](references/optimization-playbook.md) when generating recommendations or designing follow-up experiments.
84+
85+
## Create the audit
86+
87+
Create or refresh `projects/internals/ci/CI-PERFORMANCE.md`. Use `.metrics/ci-profile.md` as generated evidence, not as the finished audit.
88+
89+
Include:
90+
91+
1. Frontmatter and generation context
92+
2. Executive summary with median CI time and comparison when available
93+
3. Method, environment, and all run results
94+
4. Current top-ten scripts
95+
5. Completion-path analysis
96+
6. Numbered open findings
97+
7. Verified completed changes when a prior audit exists
98+
8. Recommended experiment order
99+
9. Raw artifact location
100+
101+
For every finding, provide:
102+
103+
-**Evidence:** measurements plus current configuration or dependency facts
104+
-**Recommendation:** one bounded change or experiment
105+
-**Validation:** output-equivalence checks, metrics to compare, and resource checks
106+
-**Confidence:** confidence in the diagnosis and in the proposed approach
107+
108+
Keep completed work out of the open priority list. Mark an item verified only when the configuration changed and a comparable full profile confirms the result.
109+
110+
## Recommendation guardrails
111+
112+
- Do not add CI-share percentages together; scripts overlap.
113+
- Do not claim a critical path from rankings alone.
114+
- Do not recommend more concurrency without checking isolation, shared state, ports, output paths, report merging, memory, and machine-wide contention.
115+
- Do not remove a quality check unless the same coverage remains in another required workflow.
116+
- Prefer one-variable experiments and compare at least three samples when variance matters.
117+
- Distinguish targeted project benchmarks from full concurrent CI results.
118+
- Treat cache optimization separately from this cold profile because the profiler disables Wireit caching.
119+
- Label estimates and inferred causes. Do not present them as measurements.
120+
- Keep dependency upgrades separate from performance changes unless the upgrade is the explicit experiment.
121+
122+
## Check the report
123+
124+
Run:
125+
126+
```shell
127+
mise exec -- pnpm exec prettier --write projects/internals/ci/CI-PERFORMANCE.md
128+
mise exec -- pnpm exec vale --config .vale.ini projects/internals/ci/CI-PERFORMANCE.md
129+
git diff --check
130+
```
131+
132+
If `projects/internals/ci/ci-profile.test.js` exists, also run:
133+
134+
```shell
135+
mise exec -- node --test projects/internals/ci/ci-profile.test.js
136+
```
137+
138+
Do not rerun the full CI merely to validate the report: the profiler already completed it three times. Report the median, comparison, top remaining opportunities, artifact paths, and validation results to the user.

‎.agents/skills/profile-ci/references/optimization-playbook.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# CI optimization playbook
2+
3+
Use this reference after collecting a valid profile. Select approaches from current evidence; do not list every approach in every audit.
4+
5+
## Selection order
6+
7+
### Remove unconsumed or duplicate work
8+
9+
Look for bundle visualizers, duplicate test suites, unused report formats, repeated downloads, redundant generation, and work repeated in both a main job and a dedicated job.
10+
11+
Verify all output consumers before removal. Preserve required test, release, documentation, and deployment coverage.
12+
13+
### Improve the completion branch
14+
15+
Trace the scripts that finish last through Wireit dependencies. Consider:
16+
17+
- starting independent prerequisites earlier;
18+
- removing dependencies that do not represent real input requirements;
19+
- narrowing broad dependency fan-in;
20+
- reducing work in the final command;
21+
- avoiding a second copy, transform, or package pass.
22+
23+
Do not decouple a dependency until output and runtime behavior prove that the downstream task does not need it.
24+
25+
### Split composite commands
26+
27+
Split sequential phases when one command hides attribution or forces broad cache invalidation. Give each phase accurate Wireit files, outputs, dependencies, and environment.
28+
29+
Splitting improves observability and caching. It does not reduce cold wall time by itself unless the graph can safely overlap phases.
30+
31+
### Add controlled parallelism
32+
33+
Look for one-worker test suites, disabled file parallelism, sequential linting, and independent generated targets.
34+
35+
Prefer process-level shards when tools share browser state or globals. Give every shard separate ports, browser profiles, coverage directories, screenshots, JUnit/JSON output, and temporary files. Merge reports deterministically and verify identical totals and thresholds.
36+
37+
Measure peak memory and full CI time. A faster package command can slow the pipeline by starving sibling work.
38+
39+
### Reduce transformed or generated inputs
40+
41+
Use module counts, page counts, plugin timings, and repeated transforms to find large input surfaces. Consider:
42+
43+
- deduplicating identical generated modules;
44+
- externalizing safe shared assets;
45+
- omitting development-only modules from production;
46+
- avoiding repeated full-document scans;
47+
- narrowing entry points and globs;
48+
- generating shared metadata once.
49+
50+
Require output-equivalence checks for routes, bundles, screenshots, metadata, and runtime behavior.
51+
52+
### Improve cache boundaries
53+
54+
Audit Wireit `files`, `output`, `dependencies`, `cascade`, environment variables, and package-lock inputs. Split tasks whose unrelated inputs invalidate expensive work.
55+
56+
Measure cache-hit behavior separately. The cold profiler sets `WIREIT_CACHE=none`, so cache changes cannot explain its results.
57+
58+
### Use tool-specific diagnostics
59+
60+
Inspect the installed tool version and local configuration before choosing flags.
61+
62+
- Wireit: use command start/finish order and dependency declarations.
63+
- Vite or Rolldown: inspect transformed module counts and plugin timing warnings.
64+
- Eleventy: compare written files, templates, transforms, and Vite phases.
65+
- Vitest: inspect file/test totals, workers, concurrency, isolation, retries, browser providers, and reporters.
66+
- ESLint: inspect rule timing, file counts, cache state, and supported concurrency modes.
67+
- Coverage: identify report consumers and the cost of collection, transformation, serialization, and report generation.
68+
69+
## Experiment record
70+
71+
For each experiment, capture:
72+
73+
- hypothesis;
74+
- exact configuration difference;
75+
- baseline and candidate commands;
76+
- sample count and environment;
77+
- median, minimum, maximum, and variance;
78+
- output-equivalence checks;
79+
- test and coverage totals;
80+
- peak memory or other resource constraints;
81+
- targeted timing and full-CI timing;
82+
- decision and confidence.
83+
84+
Change one performance variable at a time where practical.
85+
86+
## Confidence rubric
87+
88+
-**High diagnosis confidence:** repeated measurements and configuration directly explain the serialization or work.
89+
-**Medium diagnosis confidence:** evidence is consistent, but concurrent contention or hidden tool behavior remains.
90+
-**Low diagnosis confidence:** the signal is noisy, appears only once, or lacks configuration support.
91+
-**High approach confidence:** a low-risk removal or configuration change preserves verified outputs.
92+
-**Medium approach confidence:** the approach needs isolation, report merging, or dependency validation.
93+
-**Low approach confidence:** the expected gain is speculative or depends on undocumented behavior.
94+
95+
State diagnosis confidence separately from approach confidence when they differ.
96+
97+
## Reject weak recommendations
98+
99+
Do not recommend:
100+
101+
- deleting caches to make a warm build faster;
102+
- adding all script durations to estimate total CI time;
103+
- increasing every worker count to the CPU count;
104+
- removing tests solely because they are slow;
105+
- moving a check to another workflow without confirming that required status checks include it;
106+
- claiming an isolated benchmark equals full CI savings;
107+
- upgrading dependencies and changing orchestration in the same performance experiment;
108+
- optimizing a top-ten parallel script while ignoring the branch that completes last.

‎config/vale/styles/config/vocabularies/Elements/accept.txt‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ ESLint
99
Wireit
1010
pnpm
1111
Vite
12+
Rolldown
1213
Vitest
1314
WCAG
1415
ARIA
@@ -30,8 +31,10 @@ Chromium
3031
Playwright
3132
OWASP
3233
CDN
34+
CPUs
3335
polyfill
3436
polyfills
37+
profiler
3538
entrypoint
3639
entrypoints
3740
postcss
@@ -72,15 +75,19 @@ popover
7275
popovers
7376
shortcode
7477
shortcodes
78+
passthrough
7579
server-driven
7680
single-column
7781
multi-column
7882
Heatmap
7983
heatmap
8084
heatmaps
85+
hotspot
8186
datalist
8287
debounce
8388
debounced
89+
deduplicating
90+
deduplication
8491
dropdown
8592
dropdowns
8693
checkbox
@@ -124,8 +131,10 @@ dev
124131
async
125132
Rollup
126133
Webpack
134+
Turbopack
127135
subheader
128136
tsconfig
137+
typecheck
129138
tsx
130139
TSX
131140
jsx
@@ -147,6 +156,7 @@ svg
147156
SVG
148157
scrollable
149158
parallelization
159+
sharding
150160
minimizable
151161
datagrid
152162
ESBuild
@@ -174,6 +184,7 @@ inlined
174184
Inlined
175185
globals
176186
gitignored
187+
untracked
177188
exportparts
178189
downleveling
179190
downlevel

‎package.json‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
"ci:all": "PAGES_BASE_URL=/elements/ pnpm run ci && PAGES_BASE_URL=/elements/ node ./projects/internals/ci/cache-validate.js ci",
2121
"ci:reset": "pnpm run clean && pnpm i --frozen-lockfile --prefer-offline",
2222
"ci:validate": "wireit",
23+
"ci:profile": "node ./projects/internals/ci/ci-profile.js",
2324
"lighthouse": "wireit",
2425
"clean": "git clean -dfX",
2526
"format": "wireit",
@@ -102,6 +103,7 @@
102103
"./projects/starters/vue:ci",
103104
"./projects/styles:ci",
104105
"./projects/themes:ci",
106+
"./projects/internals/ci:ci",
105107
"./projects/internals/design:ci",
106108
"./projects/internals/metadata:ci",
107109
"./projects/internals/patterns:ci",

0 commit comments

Comments
 (0)