Skip to content

fix(registry): mirror repo layout in cache so cross-workflow refs resolve - #194

Merged
Jason Robert (jrob5756) merged 1 commit into
mainfrom
fix/registry-cache-layout
May 15, 2026
Merged

fix(registry): mirror repo layout in cache so cross-workflow refs resolve#194
Jason Robert (jrob5756) merged 1 commit into
mainfrom
fix/registry-cache-layout

Conversation

@jrob5756

Copy link
Copy Markdown
Collaborator

Bug

The registry cache stored each workflow under
<base>/<registry>/<workflow_name>/<sha[:12]>/<filename>, isolating each
workflow in its own subtree. Repo-relative references between workflows in the
same registry repo broke at resolve time:

sdd-plan/plan.yaml # references ../document-review/workflow.yaml
document-review/workflow.yaml

The .. from <cache>/<registry>/sdd-plan/<sha>/plan.yaml resolved to
<cache>/<registry>/sdd-plan/document-review/workflow.yaml — a path that never
existed. Users had to manually copy files into the expected slots as a
workaround.

Fix

Cache layout

Workflows from the same registry+SHA now share a per-SHA root that mirrors the
source repository:

<base>/<registry>/<sha[:12]>/<repo_path> # mirrored repo files
<base>/<registry>/_meta/<sha[:12]>/source.json # cache metadata
<base>/<registry>/_meta/<sha[:12]>/index.yaml # cached registry index
<base>/<registry>/_meta/<sha[:12]>/<workflow>.complete # readiness sentinel

Adhoc:

<base>/_adhoc/<owner>/<repo>/<sha[:12]>/<repo_path>
<base>/_adhoc/<owner>/<repo>/_meta/<sha[:12]>/...

Metadata lives outside the SHA tree so it can never collide with real repo
paths (e.g. a repo's own .conductor/ directory).

Safety + correctness

  • Per-workflow readiness sentinel written last so concurrent or
    interrupted fetches never expose a partially populated workflow.
  • Atomic per-file os.replace() from a staging dir under _meta/<sha>/,
    intra-filesystem so renames are atomic. Files in the SHA mirror are
    content-addressed by the immutable SHA; concurrent promotions of identical
    content are idempotent.
  • _safe_repo_path() rejects .. segments, absolute paths (POSIX +
    Windows), NUL bytes, and empty paths from any index/sibling entry.
  • _resolve_within() defense-in-depth ensures resolved targets stay under
    the SHA root after disk resolution.
  • source.json carries cache_layout_version, registry_type, source,
    and full_sha; cache hits require all four to match the current registry
    entry. Stale metadata triggers a re-fetch.
  • The registry index is cached on disk (_meta/<sha>/index.yaml) so cache
    hits avoid a network round-trip.

Auto-fetch sub-workflows from same registry

When a sub-workflow ref like ../document-review/workflow.yaml resolves to a
path inside a registry SHA cache but the file isn't yet present,
_resolve_subworkflow_path calls a new auto_fetch_relative_workflow() which:

  1. Detects the (registry, sha) from the candidate path.
  2. Validates the cached metadata (layout version, registry type, SHA prefix
    match, valid GitHub source).
  3. Looks up a workflow in the cached index by repo-relative path.
  4. Fetches it, populating the shared SHA mirror.

The hook is gated to candidates that look like a file path (separators or
.yaml/.yml extension) and not a registry ref (no @), so registry-style
refs continue through the existing resolver path.

Reserved registry names

add_registry() now rejects names containing /, \, the empty string, or
the reserved _adhoc / _meta namespaces, preventing configurations that
would silently corrupt cache-layout detection.

Backward compatibility

Old per-workflow caches are simply ignored — never read by the new code. They
become wasted disk that conductor registry clear removes. No automatic
migration; users re-fetch on next use.

Out of scope

Cross-directory !file tag references (e.g.
instructions: !file ../other-workflow/prompt.txt) are unchanged. The loader
resolves !file during YAML load before _resolve_subworkflow_path runs;
supporting that case requires hooking the loader and tracking registry context
there. Same-directory !file refs continue to work because siblings are still
fetched alongside the workflow.

Testing

  • Full suite: 2633 passed, 11 skipped.
  • New cache tests: layout, sentinel-based hit detection, cached-index reuse,
    stale-metadata re-fetch, path-traversal rejection, cross-workflow
    auto-fetch, find_registry_cache_location, metadata-mismatch rejection.
  • New end-to-end test (TestCrossWorkflowRegistryRef) reproduces the original
    bug with a parent workflow that references a sibling via
    ../other/workflow.yaml and verifies the auto-fetch hook populates the
    cache and runs the sub-workflow.
  • make lint ✅ — make typecheck ✅ (one pre-existing unrelated warning)

…olve
Previously the cache stored each workflow under
`<base>/<registry>/<workflow_name>/<sha[:12]>/<filename>`. With each
workflow isolated in its own subtree, repo-relative references between
workflows in the same registry repo broke at resolve time:
sdd-plan/plan.yaml # references ../document-review/workflow.yaml
document-review/workflow.yaml
The relative `..` resolved to a path that never existed in the cache,
forcing manual workarounds (copying files into the expected slots).
Cache layout
============
Workflows from the same registry+SHA now share a per-SHA root that
mirrors the source repository:
<base>/<registry>/<sha[:12]>/<repo_path> # mirrored repo files
<base>/<registry>/_meta/<sha[:12]>/source.json # cache metadata
<base>/<registry>/_meta/<sha[:12]>/index.yaml # cached registry index
<base>/<registry>/_meta/<sha[:12]>/<workflow>.complete # readiness sentinel
Adhoc:
<base>/_adhoc/<owner>/<repo>/<sha[:12]>/<repo_path>
<base>/_adhoc/<owner>/<repo>/_meta/<sha[:12]>/...
Metadata lives outside the SHA tree so it can never collide with real
repo paths (e.g. a repo's own `.conductor/` directory).
Safety + correctness
====================
* Per-workflow readiness sentinel written **last**: prevents readers
from observing a partially populated workflow during a concurrent or
interrupted fetch.
* Atomic per-file `os.replace()` from a staging dir under `_meta/<sha>/`,
staying intra-filesystem so the rename is atomic. Files in the SHA
mirror are content-addressed by the immutable SHA, so concurrent
promotions of identical content are idempotent.
* `_safe_repo_path()` rejects `..` segments, absolute paths (POSIX +
Windows), NUL bytes, and empty paths from any index/sibling entry.
* `_resolve_within()` provides defense-in-depth, asserting that
resolved targets stay under the SHA root.
* `source.json` carries `cache_layout_version`, `registry_type`,
`source`, and `full_sha`; cache hits require all four to match the
current registry entry. Stale metadata triggers a re-fetch.
* The registry index is cached on disk (`_meta/<sha>/index.yaml`) so
cache hits avoid a network round-trip.
Auto-fetch sub-workflows from same registry
===========================================
When a sub-workflow ref like `../document-review/workflow.yaml` resolves
to a path inside a registry SHA cache but the file is not yet present,
`_resolve_subworkflow_path` now calls
`auto_fetch_relative_workflow()`, which:
1. Detects the `(registry, sha)` from the candidate path.
2. Validates the cached metadata (layout version, registry type, SHA
prefix match, valid GitHub source).
3. Looks up a workflow in the cached index by repo-relative path.
4. Fetches it, populating the shared SHA mirror.
The hook is gated to candidates that look like a file path (separators
or `.yaml`/`.yml` extension) and not a registry ref (no `@`), so
registry-style references continue through the existing resolver.
Reserved registry names
=======================
`add_registry()` now rejects names containing `/`, `\\`, the empty
string, or the reserved `_adhoc` / `_meta` namespaces, preventing
configurations that would silently corrupt cache-layout detection.
Backward compatibility
======================
Old per-workflow caches are simply ignored — never read by the new
code. They become wasted disk that `conductor registry clear` removes.
No automatic migration; users re-fetch on next use.
Out of scope
============
Cross-directory `!file` tag references (e.g.
`instructions: !file ../other-workflow/prompt.txt`) are unchanged.
The loader resolves `!file` during YAML load before
`_resolve_subworkflow_path` runs; supporting that case requires hooking
the loader and tracking registry context there. Same-directory `!file`
refs continue to work because siblings are still fetched alongside the
workflow.
Testing
=======
* Full test suite: 2633 passed.
* New cache tests: layout, sentinel-based hit detection, cached-index
reuse, stale-metadata re-fetch, path-traversal rejection,
cross-workflow auto-fetch, find_registry_cache_location.
* New end-to-end test (`TestCrossWorkflowRegistryRef`) reproduces the
original bug with a parent workflow that references a sibling via
`../other/workflow.yaml` and verifies the auto-fetch hook populates
the cache and runs the sub-workflow.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jrob5756
Jason Robert (jrob5756) merged commit 3e726a0 into mainMay 15, 2026
9 checks passed
@jrob5756
Jason Robert (jrob5756) deleted the fix/registry-cache-layout branch May 15, 2026 14:50
@jrob5756Jason Robert (jrob5756) mentioned this pull request May 21, 2026
4 tasks
Jason Robert (jrob5756) added a commit that referenced this pull request May 21, 2026
- feat(script): script agents output schemas (#206, #118)
- feat(validate): warn on undeclared agent.output refs and field-level mismatches in explicit mode (#208)
- feat(copilot): attribute verbose logs to agents in parallel/for-each runs (#207)
- fix(resume): replay original event log into dashboard on --web (#167, #205)
- fix(windows): make --web-bg startup crashes diagnosable (#116, #204)
- fix(engine,web): resolve max-iterations gate from dashboard in --web-bg (#202)
- fix(bg): detach --web-bg child from Windows job to prevent kill-on-close (#200)
- fix(bg): stop passing redundant --silent to bg child (#199, #196)
- fix(cli): suppress web-bg dashboard output in silent mode (#203, #211)
- fix(config): auto-fetch sibling sub-workflow from registry cache during validation (#197)
- fix(registry): mirror repo layout in cache so cross-workflow refs resolve (#194)
- fix(copilot): tolerate SDK metadata parsing errors when listing models (#193)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@jrob5756