Uh oh!
There was an error while loading. Please reload this page.
feat: Generic git URL support (GitLab, Bitbucket, any host) - #150
Conversation
Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
…t, etc.) - Modify is_supported_git_host() to accept any valid FQDN - Add ssh:// protocol URL normalization to DependencyReference.parse() - Fix SSH .git suffix stripping when #ref is present - Update error messages and security tests for new permissive model - Add comprehensive tests for GitLab, Bitbucket, and self-hosted git URLs Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
…ities Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
…d CLI ops
- Normalize-on-write: all dependency inputs canonicalized before storing in apm.yml
(Docker-style: github.com stripped, non-default hosts preserved as FQDN)
- Object-form entries: {git, path, ref, alias} dicts in apm.yml alongside strings
- Identity-based install dedup, uninstall matching, and --only filter
- Token scoping: PAT only embedded for GitHub hosts (is_github_hostname check)
- Credential helper: token-based env logic (has token → locked, else relaxed)
- 104 new tests (60 canonicalization, 32 auth/parser, 12 integration with real repos)
- Documentation updated (dependencies.md, cli-reference.md)Daniel Meppiel (danielmeppiel)
commented
Mar 4, 2026
Implementation Summary: Canonical Format, Object-Form Deps & Auth ScopingFull test suite: 1,402 passed, 64 skipped, 0 failures The Problem (Before)APM stored dependency strings exactly as the user typed them and only supported string entries. This caused seven cascading bugs:
The Design (Docker-Style Default Registry)
Scenarios Unlocked (with Examples)1. Object-Form DependenciesUsers can now declare structured entries in # apm.ymldependencies:
apm:
- microsoft/apm-sample-package # string (simple)
- git: https://gitlab.com/acme/standards.git # object (structured)path: instructions/securityref: v2.0
- git: git@bitbucket.org:team/rules.gitpath: prompts/review.prompt.mdalias: reviewBefore: Only string entries supported. Specifying a sub-path + ref on a non-GitHub URL required FQDN shorthand with ambiguity risk. 2. Mixed Manifests (Strings + Objects)String and object entries coexist in the same list — # apm.yml — both forms in the same listdependencies:
apm:
- microsoft/apm-sample-package
- github/awesome-copilot/skills/review-and-refactor
- git: https://gitlab.com/acme/coding-standards.gitpath: instructions/securityref: v2.0apm install # installs all three — string and object handled uniformly
apm deps list # lists all three correctly3. URL-Based Install (Normalize on Write)Paste any clone URL — APM canonicalizes it before writing: # All three commands produce the SAME entry in apm.yml:
apm install microsoft/apm-sample-package
apm install https://github.com/microsoft/apm-sample-package.git
apm install git@github.com:microsoft/apm-sample-package.git# apm.yml result (always canonical)dependencies:
apm:
- microsoft/apm-sample-package # ← not the raw HTTPS/SSH URLBefore: Raw URL stored verbatim ( 4. Multi-Host TeamsGitLab, Bitbucket, and self-hosted instances work alongside GitHub. Non-default hosts preserve their FQDN: apm install https://gitlab.com/acme/coding-standards.git
apm install git@bitbucket.org:team/security-rules.git# apm.yml resultdependencies:
apm:
- microsoft/apm-sample-package # GitHub (default) → host stripped
- gitlab.com/acme/coding-standards # GitLab → FQDN preserved
- bitbucket.org/team/security-rules # Bitbucket → FQDN preserved5. Flexible Uninstall (Any Input Form)Users no longer need to remember how they installed a package: # Package stored as "microsoft/apm-sample-package" in apm.yml.# ALL of these uninstall it:
apm uninstall microsoft/apm-sample-package
apm uninstall https://github.com/microsoft/apm-sample-package.git
apm uninstall git@github.com:microsoft/apm-sample-package.git
apm uninstall github.com/microsoft/apm-sample-packageBefore: 6. Duplicate Prevention Across Formats# First install — adds to apm.yml
apm install microsoft/apm-sample-package
# Second install with different format — detected as duplicate, skipped
apm install https://github.com/microsoft/apm-sample-package.git
# → "already declared, skipping"# apm.yml — still exactly one entrydependencies:
apm:
- microsoft/apm-sample-packageBefore: Both entries would be added, causing duplicate downloads and potential conflicts. 7. Clean ManifestsRegardless of how packages are added, apm install git@github.com:microsoft/apm-sample-package.git
apm install https://github.com/github/awesome-copilot.git
apm install https://gitlab.com/acme/coding-standards.git# apm.yml — clean canonical entries, not a mess of URLsdependencies:
apm:
- microsoft/apm-sample-package
- github/awesome-copilot
- gitlab.com/acme/coding-standards8. Secure Multi-Host Authexport GITHUB_CLI_PAT=ghp_xxx
apm install microsoft/apm-sample-package # → token embedded in clone URL ✓
apm install https://gitlab.com/acme/standards.git # → NO token, credential helper used ✓Before: GitHub PAT was embedded in ALL clone URLs including GitLab/Bitbucket — token leakage. GITHUB_HOST and PortabilityThe This is not a new risk — the same ambiguity existed before this PR. Bare This is the standard pattern across package managers:
The lockfile is the safety net — Recommendation for enterprise teams using Virtual Paths in Canonical FormPaths are preserved in the canonical form. Examples:
The path is part of the identity — Integration Test → Scenario MappingEach integration test validates a specific real-world scenario with actual network calls to live GitHub repositories:
|
| Test | Scenario | Real Repo | Verifies |
|---|---|---|---|
test_https_git_url_github | #3 (URL install) | microsoft/apm-sample-package | HTTPS URL parsed → correct host/repo_url → package cloned → apm.yml exists in clone |
test_ssh_git_url_github | #3 (URL install) | microsoft/apm-sample-package | SSH URL parsed → same host/repo_url as HTTPS → package cloned successfully |
test_object_format_git_url_with_path | #1 (Object-form) | github/awesome-copilot → skills/aspire | Dict parsed via parse_from_dict → virtual path resolved → SKILL.md found at sub-path |
test_object_format_with_ref | #1 (Object-form) | github/awesome-copilot → skills/review-and-refactor @ main | Ref preserved through parse → virtual path + ref combo works → SKILL.md downloaded |
test_mixed_string_and_object_deps | #2 (Mixed manifest) | microsoft/apm-sample-package + github/awesome-copilot | String + dict coexist → both parsed correctly → correct dep count and types |
TestNormalizeOnWriteRoundtrip — Canonical format storage and dedup
| Test | Scenario | Verifies |
|---|---|---|
test_install_https_url_stores_canonical | #3 (URL install) | HTTPS URL input → apm.yml contains microsoft/apm-sample-package, NOT the raw URL |
test_install_ssh_url_stores_canonical | #3 (URL install) | SSH URL input → apm.yml contains microsoft/apm-sample-package, NOT the raw URL |
test_no_duplicate_when_already_in_canonical_form | #6 (Dedup) | Re-installing microsoft/apm-sample-package when it already exists → no duplicate, returns empty |
test_no_duplicate_when_url_matches_existing_canonical | #6 (Dedup) | Installing HTTPS URL when shorthand already in manifest → identity match → no duplicate |
test_canonical_form_stable_on_reparse | #7 (Clean manifest) | Write canonical → read → to_canonical() → identical output (idempotent) |
test_canonical_with_host_stable | #4 (Multi-host) | gitlab.com/acme/standards stored → reparse → host preserved, to_canonical() stable |
test_canonical_stored_entry_installs_correctly | #7 (Clean manifest) | Canonical entry in apm.yml → real download → apm.yml exists in clone, name matches |
Backward Compatibility
Verdict: No Breaking Changes
This PR is fully backward-compatible. Every existing apm.yml file, CLI workflow, and API surface continues to work without modification.
| Aspect | Compatibility | Detail |
|---|---|---|
Existing apm.yml files | ✅ 100% compatible | All existing entries are owner/repo shorthand strings — already in canonical form. from_apm_yml() parses them through the same DependencyReference.parse() codepath as before. |
apm install <shorthand> | ✅ Identical behavior | owner/repo input canonicalizes to owner/repo — no change in stored form. |
apm uninstall <shorthand> | ✅ Identical behavior | Shorthand → identity → matches existing shorthand entry. Same result as before. |
apm deps list / apm deps tree | ✅ Unchanged | Both use get_unique_key() (host-blind owner/repo) to match filesystem layout. No code changes in deps.py. |
apm.lock format | ✅ Unchanged | Lockfile uses get_unique_key() which returns the same owner/repo string as before. |
get_canonical_dependency_string() | ✅ Preserved | Kept as backward-compat shim → delegates to get_unique_key(). Any code calling this method gets the same result. |
get_unique_key() | ✅ Unchanged | Returns owner/repo (host-blind). Used by 20+ callsites — all continue to work. |
DependencyReference.parse() | ✅ Extended, not changed | Existing inputs produce the same DependencyReference objects. New URL/SSH/FQDN patterns are additive. |
Object-form {git, path, ref, alias} | ✅ Additive only | New capability. Old apm.yml files don't have dict entries, so this codepath is never triggered for existing manifests. |
| Normalize-on-write | ✅ Transparent | Only affects new apm install <pkg> calls. Existing entries are already canonical (owner/repo), so re-installing doesn't mutate them. |
| Auth env variables | ✅ Unchanged | GITHUB_CLI_PAT, GITHUB_APM_PAT, GITHUB_TOKEN, GITHUB_HOST — all honored as before. Token scoping is additive (restricts token to GitHub hosts only — a security improvement, not a breaking change). |
Why No Migration Is Needed
The canonical form for GitHub packages is owner/repo — which is exactly what existing apm.yml files already contain. The normalize-on-write logic is idempotent: canonicalize("owner/repo") returns "owner/repo". There is no scenario where an existing manifest gets rewritten or invalidated.
Test Coverage Summary
| Test File | Tests | Scope |
|---|---|---|
tests/unit/test_canonicalization.py | 60 | to_canonical, get_identity, normalize-on-write, uninstall matching, filter matching |
tests/unit/test_auth_scoping.py | 32 | Token scoping, env logic, parse_from_dict, mixed deps wiring |
tests/integration/test_generic_git_url_install.py | 12 | Real repo install roundtrips (5), canonical storage roundtrips (7) |
| Full suite | 1,402 passed | Zero regressions |
New test code: 1,191 lines across 3 files (104 test cases)
Files Changed
| File | Lines | Change |
|---|---|---|
src/apm_cli/models/apm_package.py | +161/-3 | to_canonical(), get_identity(), canonicalize(), parse_from_dict(), dict dispatch in from_apm_yml() |
src/apm_cli/cli.py | +111/-38 | Normalize-on-write, identity-based uninstall, identity-based filter |
src/apm_cli/deps/github_downloader.py | +50/-12 | Token scoping, credential helper relaxation |
docs/dependencies.md | +64/-2 | Canonical format section, object format docs, uninstall any-form example |
docs/cli-reference.md | +13/-3 | Updated install/uninstall argument descriptions and examples |
tests/test_github_downloader.py | +1 | Regression fix for enterprise test |
Total: 330 insertions, 70 deletions (production code) + 1,191 lines new tests
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Pull request overview
This PR expands APM dependency parsing and installation to support generic git hosts (any valid FQDN) beyond GitHub/Azure DevOps, including normalization/canonicalization improvements and updated documentation/tests.
Changes:
- Allow any valid FQDN as a supported git host and update related error messaging.
- Add canonicalization + identity concepts to normalize dependency storage and improve dedup/uninstall matching across URL forms.
- Add extensive unit/integration test coverage and update docs to describe the new supported dependency formats.
Reviewed changes
Copilot reviewed 14 out of 15 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
src/apm_cli/utils/github_host.py | Accept any valid FQDN as a supported host; refresh host-related error message text. |
src/apm_cli/models/apm_package.py | Add canonicalization/identity helpers, ssh:// normalization, object-style dependency parsing, and SSH .git-suffix fix. |
src/apm_cli/deps/github_downloader.py | Scope GitHub tokens to GitHub hosts; relax git env when no applicable token is available. |
src/apm_cli/cli.py | Normalize-on-write for apm install, identity-based uninstall + --only filtering. |
tests/unit/test_github_host.py | Update expectations for generic host support. |
tests/unit/test_generic_git_urls.py | New unit tests for generic host parsing, ssh:// normalization, clone URL building, and virtual-path rules. |
tests/unit/test_canonicalization.py | New unit tests for canonicalization/identity behavior and CLI normalize-on-write helpers. |
tests/unit/test_auth_scoping.py | New unit tests ensuring tokens/env behavior are scoped correctly by host type. |
tests/test_github_downloader.py | Ensure enterprise-host token env is set for the tested scenario. |
tests/test_apm_package_models.py | Adjust security tests to reflect generic-host acceptance and updated error strings. |
tests/integration/test_generic_git_url_install.py | New integration tests for real-repo installs and normalize-on-write roundtrips. |
docs/getting-started.md | Document git URL formats for GitLab/Bitbucket/any host. |
docs/dependencies.md | Document generic git hosts, object-style deps, canonical storage, and auth behavior for non-GitHub hosts. |
docs/cli-reference.md | Update CLI reference to reflect new accepted dependency formats and canonicalization behavior. |
Comments suppressed due to low confidence (1)
src/apm_cli/cli.py:1123
- uninstall() identity matching currently breaks for object-style dependencies because dict entries are converted to str(dep_entry) (e.g., "{'git': 'https://...'}"), which DependencyReference.parse() cannot parse. This makes it impossible to uninstall deps declared as objects. Treat dict entries by parsing their 'git' field (or use DependencyReference.parse_from_dict(dep_entry)) when comparing identities, and ensure downstream removal/path calculation logic also handles dict entries safely (TypeError vs ValueError).
# Match by identity: parse the user input and each apm.yml entry,
# compare using get_identity() which normalizes host differences.
matched_dep = None
try:
pkg_ref = DependencyReference.parse(package)
pkg_identity = pkg_ref.get_identity()
except Exception:
pkg_identity = package
for dep_entry in current_deps:
dep_str = dep_entry if isinstance(dep_entry, str) else str(dep_entry)
try:
dep_ref = DependencyReference.parse(dep_str)
if dep_ref.get_identity() == pkg_identity:
matched_dep = dep_entry # preserve original entry for removal
break
except Exception:
# Fallback: exact string match
if dep_str == package:
matched_dep = dep_entry
break
pass
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
- Fix is_github variable readability (operator precedence → explicit if/else) - Fix test name mismatch (test_custom_ghe_host → test_default_github_host_stripped) - Fix dict identity parsing (use parse_from_dict() for dict-style deps) - Fix validation env for generic hosts (relax locked-down env for non-GitHub/ADO) - Add nested GitLab group support (N-segment repo paths for generic hosts) - Add 35 new tests covering nested groups, dict identity, env scoping, and host classification - Update docs for nested group syntax and FQDN shorthand examples
- Test shorthand ambiguity: proves 5-segment FQDN misparses repo boundary - Test dict format resolution: proves explicit git+path is unambiguous - Test dict format with collections, install paths, canonical forms, clone URLs - Test all fields (ref, alias) on dict-based nested-group deps - Improve docs with concrete DON'T/DO example for the ambiguity
Daniel Meppiel (danielmeppiel)
commented
Mar 4, 2026
Nested GitLab Group Support: Design & Ambiguity ResolutionThe ProblemGitLab (and Gitea, etc.) supports nested groups — a repo can live at But virtual packages (e.g., The Solution: Host-Aware 3-Way DispatchThe parser classifies hosts into three categories at parse time:
For generic hosts, a secondary check looks for virtual indicators: ifis_generic_host:
has_virtual_ext=any(seg.endswith(ext) forextinVIRTUAL_FILE_EXTENSIONS)
has_collection='collections'inpath_segmentsifhas_virtual_extorhas_collection:
min_base_segments=2# Simple owner/repo + virtual path after itelse:
min_base_segments=len(path_segments) # ALL segments = repo pathThe Ambiguity Case (and why dict format exists)With shorthand,
The dict format resolves this unambiguously: # DON'T — ambiguous: APM can't tell where the repo path ends# gitlab.com/group/subgroup/repo/file.prompt.md# → parsed as repo=group/subgroup, virtual=repo/file.prompt.md (wrong!)# DO — explicit and unambiguous
- git: gitlab.com/group/subgroup/repopath: file.prompt.mdWhat Works Without Dict FormatThe common cases just work with shorthand:
Test CoverageThe
All 1372 tests pass including integration tests (85 passed, 62 skipped for env-dependent ones). |
Daniel Meppiel (danielmeppiel)
commented
Mar 5, 2026
Copilot the README.md section "Install From Anywhere" should reflect this capability specially for GitLab and BitBucket users. Be pragmatic, changes should be small and surgical to this file. I give you permission to change the README.md file |
…examples Co-authored-by: danielmeppiel <51440732+danielmeppiel@users.noreply.github.com>
Updated in ab95472. Three surgical changes to README.md:
|
PR #150 — Review Summary Overall: Approve. Core feature is well-implemented, security posture is strong, all 1495 tests pass (200 new). What's good:
Non-blocking follow-ups:
|
Uh oh!
There was an error while loading. Please reload this page.
- Fix is_github variable readability (operator precedence → explicit if/else) - Fix test name mismatch (test_custom_ghe_host → test_default_github_host_stripped) - Fix dict identity parsing (use parse_from_dict() for dict-style deps) - Fix validation env for generic hosts (relax locked-down env for non-GitHub/ADO) - Add nested GitLab group support (N-segment repo paths for generic hosts) - Add 35 new tests covering nested groups, dict identity, env scoping, and host classification - Update docs for nested group syntax and FQDN shorthand examples
…pport-again feat: Generic git URL support (GitLab, Bitbucket, any host)
Description
APM only supported GitHub.com, GitHub Enterprise, and Azure DevOps as dependency hosts. This PR opens
is_supported_git_host()to accept any valid FQDN, enabling standard git protocol URLs from any host.Core change:
is_valid_fqdn()fallback inis_supported_git_host()— 3 lines that unlock every git-accessible host.ssh://protocol normalization —ssh://git@host/owner/repo.gitnormalized togit@host:owner/repo.gitbefore parsing.gitsuffix was not stripped when#refwas present (stripping now happens after ref/alias extraction)Type of change
Testing
63 new tests in
test_generic_git_urls.pycovering GitLab/Bitbucket/self-hosted × HTTPS/SSH/FQDN,ssh://normalization, clone URL building, install paths, virtual paths on generic hosts, and security invariants. 1051 total tests pass. CodeQL: 0 alerts.Original prompt
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.