Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions docs-site/docs/plugins/imported-docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,8 @@ Each pull source is a repository to pull documentation from:
| `includeSourceMarker` | bool | `false` | If `true`, adds `import_source` and `import_url` metadata to imported pages. Useful for displaying "view source" links. |
| `exclude` | array | `[]` | Glob patterns for files to exclude (e.g., `["draft/**", "INTERNAL-*.md"]`). Supports `*` (segment) and `**` (any dirs). |
| `frontMatterDefaults` | object | `{}` | Front-matter key-value pairs to apply as fallback for imported pages. Extracted values take precedence. |
| `repoUrl` | string | _(derived)_ | Browsable URL of the source repo for edit/view links, e.g. `"https://github.com/org/handbook"`. Derived from `repository` when omitted. |
| `editUri` | string | _(derived)_ | Path appended to `repoUrl` to reach an editable file, e.g. `"edit/main/docs"`. Derived from the checked-out branch and `sourcePath` when omitted. |

### S3 Source Configuration

Expand Down Expand Up @@ -125,6 +127,43 @@ Pull documentation directly from S3 buckets (no git clone overhead):
| `includeSourceMarker` | bool | `false` | If `true`, adds `import_source` with S3 URL and `import_url` metadata. |
| `exclude` | array | `[]` | Glob patterns for files to exclude. |
| `frontMatterDefaults` | object | `{}` | Front-matter key-value pairs to apply as fallback. |
| `repoUrl` | string | _(none)_ | Browsable URL of the repo backing this bucket, for edit/view links. Without it (and `editUri`) imported pages show no source buttons. |
| `editUri` | string | _(none)_ | Path appended to `repoUrl` to reach an editable file, e.g. `"edit/main/docs"`. |

## Edit and view source links

An imported page lives in someone else's repository, but its path in *this* site is wherever
`destinationPath` put it. The site-wide `repoUrl`/`editUri` would therefore aim the "edit this page"
button at this repo, at a path that only exists upstream. Imported pages resolve their own links
instead:

1. **`repoUrl` + `editUri` on the source**, if you set them — always wins.
2. **Derived from the clone** for pull sources: the repository URL gives the host (an
`git@host:org/repo.git` remote is rewritten to its `https://` form), and the branch actually
checked out plus `sourcePath` give the rest. A source that never pinned a `reference` still gets
correct links this way.
3. **No buttons at all**, if neither applies — a repository pinned to a tag or commit (detached, so
there is no branch to build a URL around), a remote that is a local path, or an S3 source with
nothing configured. A missing button is better than one that 404s.

S3 sources have nothing to derive from, so they need both options to show links.

```json
{
"repository": "git@github.com:org/handbook.git",
"sourcePath": "docs",
"destinationPath": "imported/handbook",
"repoUrl": "https://github.com/org/handbook",
"editUri": "edit/main/docs"
}
```

!!! note "Page dates for imported content"
[git-revision-date](git-revision-date.md) reads *this* repository's history, which knows nothing
about imported files. It no longer falls back to their file timestamps either, since those are
just the moment the import cloned them — which would show every imported page as updated today,
on every build. Imported pages therefore carry no git-derived dates; set them through
`frontMatterDefaults` if you need them.

## Use Cases

Expand Down
18 changes: 18 additions & 0 deletions src/Netdocs.Abstractions/Page.cs
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,17 @@ public sealed class Page
/// <summary>Arbitrary per-page state shared between plugins.</summary>
public Dictionary<string, object?> Meta { get; } = [];

/// <summary>
/// Where this page's "edit"/"view source" links should point, when the page does not come from
/// this site's own repository. Null means the site-wide <c>repoUrl</c>/<c>editUri</c> apply, as
/// they do for ordinary content. A non-null value is authoritative — and either link inside it
/// may itself be null, which means "this page has no source to link to, show no button".
/// <para>Set by importers: an imported page's <see cref="RelativePath"/> is where it landed in
/// this site, not where it lives upstream, so the site-wide pattern would build a link into the
/// wrong repository at a path that does not exist there.</para>
/// </summary>
public SourceLinks? SourceLinks { get; set; }

/// <summary>True when produced by an <see cref="IContentGenerator"/> rather than a source file.</summary>
public bool IsGenerated { get; init; }

Expand All @@ -74,3 +85,10 @@ public sealed class TocEntry
public required string Title { get; init; }
public IReadOnlyList<TocEntry> Children { get; set; } = [];
}

/// <summary>
/// Absolute "edit"/"view source" URLs for a page whose origin is not this site's repository.
/// A null member means that link is unavailable and its button should not render — better than
/// pointing somewhere plausible but wrong.
/// </summary>
public sealed record SourceLinks(string? Edit, string? View);
25 changes: 25 additions & 0 deletions src/Netdocs.Abstractions/SiteConfig.cs
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,20 @@ public sealed class ImportedDocsPullSource
/// e.g., ["*.draft.md", "private/**"]
/// </summary>
public IReadOnlyList<string>? Exclude { get; init; }

/// <summary>
/// Web URL of the source repository for this page's "edit"/"view source" links, e.g.
/// "https://github.com/org/repo". Derived from <see cref="Repository"/> when omitted.
/// </summary>
public string? RepoUrl { get; init; }

/// <summary>
/// Path segment appended to <see cref="RepoUrl"/> to reach an editable file, e.g.
/// "edit/main/docs". Derived from the cloned branch and <see cref="SourcePath"/> when omitted.
/// <para>When neither this nor a derivation is available the page renders no edit/view button,
/// rather than one pointing at a file that does not exist.</para>
/// </summary>
public string? EditUri { get; init; }
}

/// <summary>Configuration for pulling docs from an S3 bucket.</summary>
Expand Down Expand Up @@ -384,6 +398,17 @@ public sealed class ImportedDocsS3Source
/// </summary>
public bool IncludeSourceMarker { get; init; }

/// <summary>
/// Web URL of the repository backing this bucket's content, for "edit"/"view source" links.
/// There is nothing to derive it from, so without both this and <see cref="EditUri"/> the
/// imported pages render no source buttons.
/// </summary>
public string? RepoUrl { get; init; }

/// <summary>Path segment appended to <see cref="RepoUrl"/> to reach an editable file,
/// e.g. "edit/main/docs".</summary>
public string? EditUri { get; init; }

/// <summary>
/// Optional glob patterns to exclude from import.
/// e.g., ["*.draft.md", "private/**"]
Expand Down
4 changes: 4 additions & 0 deletions src/Netdocs.Core/Configuration/JsonConfigLoader.cs
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@ public static SiteConfig Load(string appSettingsPath)
IncludeSourceMarker = m.Get("includeSourceMarker").AsBool(false),
Exclude = StringList(m.Get("exclude")),
FrontMatterDefaults = m.Get("frontMatterDefaults").AsMap(),
RepoUrl = m.Get("repoUrl").AsString(),
EditUri = m.Get("editUri").AsString(),
};

private static ImportedDocsS3Source ParseS3Source(IReadOnlyDictionary<string, object?> m) => new()
Expand All @@ -79,6 +81,8 @@ public static SiteConfig Load(string appSettingsPath)
IncludeSourceMarker = m.Get("includeSourceMarker").AsBool(false),
Exclude = StringList(m.Get("exclude")),
FrontMatterDefaults = m.Get("frontMatterDefaults").AsMap(),
RepoUrl = m.Get("repoUrl").AsString(),
EditUri = m.Get("editUri").AsString(),
};

private static string Required(IReadOnlyDictionary<string, object?> m, string key, string section)
Expand Down
11 changes: 10 additions & 1 deletion src/Netdocs.Core/PageRenderer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,16 @@ public static string Render(TemplateEngine engine, SiteContext site, Page page,
// Per-page "edit"/"view source" links. Only meaningful for real source files
// (not generated pages) when the repo URL + edit_uri are configured.
string? editUrl = null, viewUrl = null;
if (!page.IsGenerated
if (page.SourceLinks is { } imported)
{
// The page came from somewhere other than this repository and its origin already
// decided where (if anywhere) these links point. Either may be null, meaning the
// upstream location is unknown and no button should render — the site-wide pattern
// below would otherwise link into this repo at a path that does not exist in it.
editUrl = imported.Edit;
viewUrl = imported.View;
}
else if (!page.IsGenerated
&& !string.IsNullOrEmpty(site.Config.RepoUrl)
&& !string.IsNullOrEmpty(site.Config.EditUri))
{
Expand Down
11 changes: 11 additions & 0 deletions src/Netdocs.Plugins/GitRevisionDatePlugin.cs
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,10 @@ public Task OnBuildStartAsync(SiteContext site, CancellationToken ct)
{
if (page.IsGenerated || !File.Exists(page.SourcePath)) continue;
var rel = Path.GetRelativePath(workDir, page.SourcePath).Replace('\\', '/');
// A page imported from elsewhere sits outside this repository, so it has no history
// here. Falling through to the filesystem would stamp it with the moment the import
// cloned it -- "last updated: today", every build. Better to leave it unset.
if (Outside(rel)) continue;
if (updated.TryGetValue(rel, out var u)) { page.Updated = u; matched++; }
else page.Updated ??= File.GetLastWriteTimeUtc(page.SourcePath);
if (_enableCreationDate && created.TryGetValue(rel, out var cr)) page.Created ??= cr;
Expand Down Expand Up @@ -87,11 +91,18 @@ private static (Dictionary<string, DateTimeOffset> Created, Dictionary<string, D
return (created, updated);
}

/// <summary>True when a repo-relative path escapes the repository (imported content staged in
/// a temp clone, for example), so this repository has nothing to say about its history.</summary>
private static bool Outside(string relativePath) =>
relativePath.StartsWith("../", StringComparison.Ordinal) || Path.IsPathRooted(relativePath);

private static void ApplyFilesystem(SiteContext site)
{
foreach (var page in site.Pages)
{
if (page.IsGenerated || !File.Exists(page.SourcePath)) continue;
if (page.SourceLinks is not null) continue; // imported: not this repo's history, and
// its mtime is just when the import ran
page.Updated ??= File.GetLastWriteTimeUtc(page.SourcePath);
page.Created ??= File.GetCreationTimeUtc(page.SourcePath);
}
Expand Down
34 changes: 31 additions & 3 deletions src/Netdocs.Plugins/ImportedDocsPlugin.cs
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,10 @@ private async Task<int> PullAndImportAsync(
return 0;
}

// Resolve where this source's files can be edited upstream. The checked-out branch is
// read from the clone, so a source that never pinned a `reference` still gets real links.
var sourceLinks = SourceLinkBuilder.ForPullSource(source, HeadBranch(repo), _logger);

var count = 0;
var excludePatterns = source.Exclude?.ToList() ?? [];
var mdFiles = Directory.EnumerateFiles(docsPath, "*.md", SearchOption.AllDirectories);
Expand All @@ -192,7 +196,7 @@ private async Task<int> PullAndImportAsync(
continue;
}

var page = await LoadPageFromFileAsync(file, docsPath, source, site);
var page = await LoadPageFromFileAsync(file, docsPath, source, site, sourceLinks);
if (page is not null)
{
site.Pages.Add(page);
Expand Down Expand Up @@ -230,11 +234,28 @@ private bool GlobMatch(string path, string pattern)
return Regex.IsMatch(path, regexPattern);
}

/// <summary>The clone's checked-out branch, or null if it cannot be determined (detached
/// HEAD from a pinned tag or commit, say) — in which case there is no branch to build an
/// edit URL around.</summary>
private static string? HeadBranch(Repository repo)
{
try
{
var name = repo.Head?.FriendlyName;
return string.IsNullOrWhiteSpace(name) || name == "(no branch)" ? null : name;
}
catch (LibGit2SharpException)
{
return null;
}
}

private async Task<Page?> LoadPageFromFileAsync(
string filePath,
string baseDir,
ImportedDocsPullSource? source,
SiteContext site)
SiteContext site,
SourceLinkBuilder? links = null)
{
try
{
Expand All @@ -254,7 +275,11 @@ private bool GlobMatch(string path, string pattern)
Url = url,
OutputPath = Path.Combine(site.Config.AbsoluteSiteDir, ContentDiscovery.OutputFileFor(url)),
RawMarkdown = content,
IsGenerated = false
IsGenerated = false,
// Where the file lives upstream, not where it landed here. Null when the origin
// is unknown, which renders no button rather than a link into the wrong repo.
SourceLinks = links?.For(Path.GetRelativePath(baseDir, filePath).Replace('\\', '/'))
?? new SourceLinks(null, null),
};

ApplyFrontMatter(page, source?.FrontMatterDefaults);
Expand Down Expand Up @@ -443,6 +468,9 @@ private IAmazonS3 CreateS3Client(ImportedDocsS3Source source)
Url = url,
OutputPath = Path.Combine(site.Config.AbsoluteSiteDir, ContentDiscovery.OutputFileFor(url)),
RawMarkdown = content,
// A bucket carries no repository information, so links exist only if configured.
SourceLinks = SourceLinkBuilder.ForS3Source(source)?.For(relPath)
?? new SourceLinks(null, null),
};

ApplyFrontMatter(page, source.FrontMatterDefaults);
Expand Down
101 changes: 101 additions & 0 deletions src/Netdocs.Plugins/SourceLinkBuilder.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
using Microsoft.Extensions.Logging;
using Netdocs.Abstractions;

namespace Netdocs.Plugins;

/// <summary>
/// Works out where an imported page's "edit"/"view source" buttons should point.
///
/// <para>An imported page's <see cref="Page.RelativePath"/> is where it landed in <em>this</em>
/// site, so the site-wide <c>repoUrl</c>/<c>editUri</c> pattern would build a link into the wrong
/// repository, at a path that does not exist in it. This resolves the upstream location instead —
/// explicitly configured, or derived from the repository the file was cloned from — and yields
/// null when neither is possible, so the page renders no button at all.</para>
/// </summary>
internal sealed class SourceLinkBuilder(string repoUrl, string editUri)
{
/// <summary>Links for one file, given its path relative to the source's docs directory.</summary>
public SourceLinks For(string pathInSource)
{
var rel = pathInSource.Replace('\\', '/').TrimStart('/');
var edit = $"{repoUrl}/{editUri}/{rel}";
// Same derivation the site-wide links use: one edit_uri drives both actions.
return new SourceLinks(edit, edit.Replace("/edit/", "/blob/"));
}

/// <summary>
/// Resolves links for a git pull source. An explicit <c>repoUrl</c>/<c>editUri</c> wins;
/// otherwise both are derived from the clone — the repository URL for the host, and the
/// checked-out branch plus the source's docs path for the edit prefix. Returns null when the
/// branch is unknown (a pinned tag or commit leaves HEAD detached) and nothing was configured.
/// </summary>
public static SourceLinkBuilder? ForPullSource(ImportedDocsPullSource source, string? branch, ILogger logger)
{
var repoUrl = source.RepoUrl ?? WebUrlFor(source.Repository);
if (repoUrl is null)
{
logger.LogDebug("imported-docs: no web URL for {Repository}; source links disabled", source.Repository);
return null;
}

var editUri = source.EditUri;
if (editUri is null)
{
if (branch is null)
{
logger.LogInformation(
"imported-docs: {Repository} is not on a branch and sets no 'editUri', so its pages get no edit/view links.",
source.Repository);
return null;
}
var docsPath = (source.SourcePath ?? "docs").Replace('\\', '/').Trim('/');
editUri = docsPath.Length > 0 ? $"edit/{branch}/{docsPath}" : $"edit/{branch}";
}

return new SourceLinkBuilder(repoUrl.TrimEnd('/'), editUri.Trim('/'));
}

/// <summary>
/// Resolves links for an S3 source. A bucket carries no repository information, so both
/// <c>repoUrl</c> and <c>editUri</c> must be configured or there are no links to give.
/// </summary>
public static SourceLinkBuilder? ForS3Source(ImportedDocsS3Source source) =>
string.IsNullOrWhiteSpace(source.RepoUrl) || string.IsNullOrWhiteSpace(source.EditUri)
? null
: new SourceLinkBuilder(source.RepoUrl.TrimEnd('/'), source.EditUri.Trim('/'));

/// <summary>
/// Browsable URL for a clone URL: strips a trailing <c>.git</c>, and rewrites the scp-style
/// SSH form (<c>git@host:org/repo</c>) that browsers cannot follow. Returns null for anything
/// else — a local path, say — where guessing a host would be worse than showing no button.
/// </summary>
internal static string? WebUrlFor(string repository)
{
var repo = repository.Trim();
if (repo.Length == 0) return null;

if (repo.StartsWith("git@", StringComparison.OrdinalIgnoreCase))
{
var at = repo.IndexOf('@');
var colon = repo.IndexOf(':', at);
if (colon < 0) return null;
var host = repo[(at + 1)..colon];
var path = repo[(colon + 1)..].TrimStart('/');
repo = $"https://{host}/{path}";
}
else if (repo.StartsWith("ssh://", StringComparison.OrdinalIgnoreCase))
{
repo = "https://" + repo["ssh://".Length..];
var at = repo.IndexOf('@');
if (at > "https://".Length) repo = "https://" + repo[(at + 1)..];
}
else if (!repo.StartsWith("http://", StringComparison.OrdinalIgnoreCase)
&& !repo.StartsWith("https://", StringComparison.OrdinalIgnoreCase))
{
return null;
}

if (repo.EndsWith(".git", StringComparison.OrdinalIgnoreCase)) repo = repo[..^4];
return repo.TrimEnd('/');
}
}
Loading