Add certificate loading APIs with content-based format detection

[yosuke-wolfssl]

Problem

wolfSSH has a single entry point for loading key files, wolfSSH_ReadKey_file(), which sniffs the format from the file's content. Certificates had no equivalent: wolfSSH_CTX_UseCert_buffer() and wolfSSH_CTX_AddRootCert_buffer() are buffer-only and require the caller to already know PEM from DER. As a result five call sites across apps/ and examples/ hand-roll file reading plus a blind try-PEM-then-retry-ASN1 dance, and apps/wolfsshd/wolfsshd.c carries a TODO asking for exactly this helper.

Fix (src/ssh.c)

New public API; the form is detected from the content, so there is no format argument.

intwolfSSH_ReadCert_buffer(constbyte*in, word32inSz, byte**out,
word32*outSz, constbyte**outType, word32*outTypeSz,
byte*flavor, void*heap);
intwolfSSH_ReadCert_file(constchar*name, /* same out-params */ ...);
intwolfSSH_CTX_UseCert_file(WOLFSSH_CTX*ctx, constchar*name);
intwolfSSH_CTX_AddRootCert_file(WOLFSSH_CTX*ctx, constchar*name);

Input form	Result
PEM CERTIFICATE (header may follow an openssl text dump)	DER, flavor = WOLFSSH_CERT_FLAVOR_X509
DER X.509	DER, flavor = WOLFSSH_CERT_FLAVOR_X509
*-cert-v01@openssh.com line	blob, flavor = WOLFSSH_CERT_FLAVOR_OSSH

outType is the wire algorithm name derived from the certificate rather than a compile-time flag. An RFC 6187 x509v3-* line is a public key carrying a chain, not a certificate, so it is declined here — wolfSSH_ReadKey_buffer() with WOLFSSH_FORMAT_SSH already reads that form.

Certificate identification is now consistent (src/internal.c). The old IdentifyCert() names the key inside a certificate, so it is renamed IdentifyCertKey(), and a new IdentifyCert() returns the x509v3-* wire algorithm, rejecting a key type that has no such name with WS_INVALID_ALGO_ID. SetHostCertificate() now takes certId, which lets wolfSSH_ProcessBuffer() apply the same rule. Previously wolfSSH_CTX_UseCert_buffer() accepted such a certificate, stored it under a plain key id, and advertised an unusable host-key algorithm.

This is the library half; converting the five existing call sites is a follow-up PR.

Tests (tests/api.c)

test_wolfSSH_ReadCert_buffer, _file, test_wolfSSH_CTX_UseCert_file, _AddRootCert_file, and an OpenSSH round-trip. Negative cases cover a truncated PEM, a bare DER header, a DER and a PEM private key, an SSH public key line, an x509v3-* line, and a certificate with no x509v3 name — for which keys/server-cert-ed25519.{pem,der} is added, generated by renewcerts.sh.

Verification

• make check passes with --enable-certs, --enable-ossh-certs, both, and neither.
• Clean under gcc-13 -Werror across the CI configurations, including small-stack and Zephyr defines.
• ASan + UBSan clean.
• Certificate/private-key pairing in the CTX is unchanged for both load orders; in a build without an x509v3-* name for the key type, the read and CTX paths now agree on rejecting the certificate.

[Copilot]

Pull request overview

Adds certificate-loading APIs that mirror wolfSSH_ReadKey_file() by detecting certificate format from content, while also tightening certificate algorithm identification and fixing an SSH public key parsing edge case.

Changes:

• Introduces wolfSSH_ReadCert_{buffer,file}() and wolfSSH_CTX_{UseCert,AddRootCert}_file() (content-sniffed PEM/DER and OpenSSH cert-line support where enabled).
• Refactors internal certificate identification to return the wirex509v3-* algorithm (and reject unmappable certs).
• Fixes DoSshPubKey() handling for SSH public key lines that do not end with a trailing newline; adds API test coverage for new cert APIs and the newline edge case.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
wolfssh/ssh.h	Declares new public certificate read/file APIs and WS_CertFlavors.
wolfssh/internal.h	Exposes IdentifyCert() for internal callers under WOLFSSH_CERTS.
src/ssh.c	Implements new cert read APIs, adds cert form sniffing, refactors file reading helper, fixes DoSshPubKey() termination.
src/internal.c	Splits “identify key inside cert” vs “identify cert wire algo”; adjusts host certificate slot handling logic.
tests/api.c	Adds coverage for new cert APIs and validates no-trailing-newline SSH public key parsing.
keys/server-key-ed25519-cert.pem	Adds Ed25519 private key fixture used to generate the unmappable X.509 cert.
keys/server-cert-ed25519.pem	Adds Ed25519 X.509 cert fixture used to exercise unmappable x509v3-* rejection.
keys/renewcerts.sh	Extends renewal script to generate the new Ed25519 cert/key fixtures.
keys/include.am	Distributes new key/cert fixtures via Automake EXTRA_DIST.

Suppressed comments (1)

src/internal.c:2364

• SetHostCertificate() always uses destIdx after the scan loop, but destIdx is guaranteed to equal ctx->privateKeyCount at that point. That means an existing certificate slot (certIdx) is never reused/replaced; instead a new entry is appended each time, leading to duplicate cert slots, leaked old certs, and eventual WS_CTX_KEY_COUNT_E once the key table fills.

Use certIdx when it was found (replace-in-place), and only use privateKeyCount as the insertion index when no existing cert slot exists.

 }
if (destIdx >= WOLFSSH_MAX_PVT_KEYS) {
ret = WS_CTX_KEY_COUNT_E;
}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #1140

Scan targets checked:wolfssh-bugs, wolfssh-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #1140

Scan targets checked:wolfssh-bugs, wolfssh-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #1140

Scan targets checked:wolfssh-bugs, wolfssh-src

Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #1140

Scan targets checked:wolfssh-bugs, wolfssh-src

No new issues found in the changed files. ✅

[ejohnstown]

Reviewed at c7200ca7. All non-blocking -- the flavor issue from the last round is genuinely fixed, and everything below is either a follow-on in the sibling function or a nit.

Verified first.make check passes with --enable-all and with --enable-all --enable-ossh-certs (10/10 each). api.test passes in an OpenSSH-certs-only build, a build with no certificate support at all, and a build with -DWOLFSSH_NO_ECDSA_SHA2_NISTP256. No new warnings in any of them. Clean under ASan and UBSan, and leaks --atExit reports zero leaks -- so the PEM-reject path that releases its intermediate DER does hold up.

I also confirmed the fix rather than taking the tests' word for it: every decode-failure path out of wolfSSH_ReadCert_buffer() returns flavor == WOLFSSH_CERT_FLAVOR_UNKNOWN, including when the caller passes a non-zero flavor in.

The fix stopped at the buffer function

wolfssh/ssh.h:110 now promises:

Past the argument checks every out parameter is written, flavor as UNKNOWN on failure.

wolfSSH_ReadCert_buffer() honors that. wolfSSH_ReadCert_file() (src/ssh.c:2651) does not: when ReadFileIntoBuffer() fails it returns straight out without writing any out parameter. A missing file, an empty file and a directory are all past the argument checks.

Driving it with sentinel values -- out set to a live pointer, outSz to 0xDEADBEEF, flavor to 0xFF:

 ret flavor out outSz
missing file -1019 255 STALE 3735928559
empty file -1019 255 STALE 3735928559
directory -1019 255 STALE 3735928559
private key DER -140 0 NULL 0 <- buffer path, correct
ssh pubkey line -1016 0 NULL 0 <- correct
good cert 0 1 new 798

A caller who reads the header and frees out on the strength of it gets a stale pointer. test_wolfSSH_ReadCert_file() does exercise the missing-file case, but checks only the return code, which is why it is green.

Fix is a few lines -- clear the five out parameters before calling ReadFileIntoBuffer(), the same way the buffer function does at the top.

Nits

DYNTYPE_CERT is not reachable from a public header.ssh.h:109 tells the caller to free out with it, but it is defined in wolfssh/internal.h. examples/echoserver/echoserver.c:2228 has a comment about exactly this problem ("not using WMALLOC because internal.h is not included for DYNTYPE_*"). Worth naming something the caller can actually see, or saying "free with the heap's free" and leaving the type out.

Raw wolfSSL codes can escape the new reader. A truncated DER comes back as -140 (ASN_PARSE_E) rather than a WS_ code -- IdentifyCert() returns wc_ParseCert()'s value verbatim. This is pre-existing behavior, not something the PR introduces, and the tests accommodate it with AssertIntLT(..., 0). Since the header is documenting a new public entry point, it is worth a sentence saying the return may be a wolfSSL code, or mapping it.

The Ed25519 fixture does not reach the branch it was added for.keys/server-cert-ed25519.{pem,der} are rejected by IdentifyCertKey() with WS_UNIMPLEMENTED_E (-1017) before IdentifyCert()'s new WS_INVALID_ALGO_ID branch is consulted -- the test comment says as much. The branch itself is fine and is genuinely covered: I built with -DWOLFSSH_NO_ECDSA_SHA2_NISTP256 and the #else arm asserting WS_INVALID_ALGO_ID passes. So this is only a note that the fixture and the renewcerts.sh/include.am changes are buying a weaker regression test than they look like they are -- they do lock in that an Ed25519 certificate is not accepted under a plain ssh-ed25519 name, which is worth having.

.gitignore.scp_rekey_*.txt looks unrelated to certificates -- fine either way, just flagging it in case it rode along by accident.

[yosuke-wolfssl]

Hello @ejohnstown ,
Thanks for the detailed pass — driving the failure paths with sentinels rather than trusting the tests is what made item 1 unambiguous. All four are addressed; I've noted where I deviated from your suggestion.

1. wolfSSH_ReadCert_file() contract — fixed

Now clears all five out parameters before ReadFileIntoBuffer(), matching the buffer function. The test checked only the return code, so it now drives sentinels through the missing-file and directory cases; reverting the fix makes it abort on AssertNull(out).

A later round found a sibling: the OpenSSH decode-failure path left outType set while out was NULL. Rather than patch that decoder, wolfSSH_ReadCert_buffer() clears all five out params in its post-switch failure block, so the guarantee holds at the boundary for any decoder. Header promise is now "on failure every out param is cleared".

2. DYNTYPE_CERT — fixed

Took your second option: the header says the caller frees out via the heap allocator and names no type. There's no public consumer of DYNTYPE_CERT in the tree, so naming it in ssh.h pointed at a symbol public-header-only code can't compile against.

3. Raw wolfSSL codes — mapped, not documented

Wider than the truncated DER: 8 of 11 malformed inputs returned -140/-132, and wolfSSH_CTX_UseCert_buffer() / AddRootCert_buffer() did the same, while the sibling wolfSSH_ReadKey_buffer() never leaks.

Worth flagging how it landed, since my first attempt was wrong. I classified by range (ret > WS_ERROR || ret < WS_LAST_E); wolfSSL's error-crypt.h reserves span 2 as -1000 .. -1999, which contains the whole WS_ band (ASCON_AUTH_E is -1001, WC_LAST_E already -1018), so the test couldn't distinguish. The final form maps at the point of production — each wc_ParseCert() / wc_GetPubKeyDerFromCert() result inside IdentifyCertKey() — and the heuristic is deleted rather than tightened. Malformed input returns WS_PARSE_E; WS_MEMORY_E, WS_UNIMPLEMENTED_E, WS_CRYPTO_FAILED and WS_INVALID_ALGO_ID pass through unchanged. Two of the four map sites are test-covered; the two wc_GetPubKeyDerFromCert() ones are defensive and unreachable with current fixtures.

4. Ed25519 fixture — kept, test made honest

Root cause: wc_GetPubKeyDerFromCert() returns Ed25519 as a raw 32-byte key where RSA and ECC come back as SPKI, so the decode fails before IdentifyCert()'s x509v3 check.

Kept the fixture — teaching wolfSSH to read that raw form would otherwise let an Ed25519 cert load under a plain ssh-ed25519 name, the exact bug this PR prevents. But the assertions now accept either WS_UNIMPLEMENTED_E or WS_INVALID_ALGO_ID, since which one lands depends on wolfSSL and CI resolves the two latest releases at run time rather than pinning. Not loosened to < 0: a mutant returning WS_PARSE_E still fails.

5. .gitignore — you were right

scp_rekey_*.txt rode along; those files come from scp_rekey_test(), which landed on master in 1efd6474 without the entry. Removed here — it belongs in a one-line PR against master.

Not in this PR

Converting the five existing call sites (as noted in the description). wolfSSH_CERTMAN_LoadRootCA_buffer() still returns raw wolfSSL codes — src/certman.c isn't otherwise in this PR, so the CA path uses an explicit whitelist at the wolfSSH_ProcessBuffer() call site; no live bug, every caller compares only against WS_SUCCESS. Separately, SetHostCertificate() discards the certIdx its scan loop finds and appends a new slot each time — pre-existing, flagging rather than fixing.

Verification

make check passes with --enable-certs, --enable-ossh-certs, both, and neither, plus -DWOLFSSH_NO_ECDSA_SHA2_NISTP256 and -DWOLFSSH_NO_ED25519. Clean under gcc-13 -Werror across the CI configurations including small-stack and Zephyr defines. Each fix has a negative control confirming the test fails without it.