rustls: dual-provider tree (ring + aws-lc-rs) has no process-level CryptoProvider default — panics on the VSS TLS path

[hash-money]

Context: @tnull asked us to split this out of lightningdevkit/orange-sdk#90 (comment: "Can you expand on that and maybe open a separate issue for it? LDK Node should do that for you, so we should investigate what's going on exactly."). Filing here per that request — and flagging honestly below where the trigger actually lives, in case you conclude it belongs in bitreq/corepc instead.

Symptom

When a binary embedding ldk-node compiles both rustls crypto providers — aws-lc-rs (rustls' default) and ring — rustls has no unambiguous process-level default, so any code that builds a ClientConfig via the process-default-dependent ClientConfig::builder() panics:

thread 'main' panicked at rustls-0.23.37/src/crypto/mod.rs:249:14:
Could not automatically determine the process-level CryptoProvider from Rustls crate features.
Call CryptoProvider::install_default() before this point to select a provider manually, or make sure exactly one of the 'aws-lc-rs' and 'ring' features is enabled.
See the documentation of the CryptoProvider type for more information.

Reproduced standalone to isolate it to rustls itself (v0.23.37, our pin):

rustls = { version = "=0.23.37", features = ["ring"] } # default features also bring in aws-lc-rs

fn main() { let _ = rustls::ClientConfig::builder(); } // panics as above

Where the two providers come from (our tree)

• ring — pulled in by electrum-client (our chain source, under ldk-node).
• aws-lc-rs — rustls' default, on the VSS / HTTP path.

ldk-node is where these two meet in a single process, which is why the collision surfaces at wallet init rather than in either dependency alone.

Where it triggers today

We hit this while integrating rust-bitcoin/corepc#661 — the bitreq connection-persistence fix you and @TheBlueMatt asked us to benchmark on #90. The patched bitreq builds its rustls ClientConfig from the process-level default, so with both providers present it panics at the first VSS request.

On the released stack (without corepc#661) we do not hit it — nothing on that path calls the process-default builder — so today it's latent, and gets exposed by the very patch that fixes the VSS latency. That's the part worth investigating: the perf fix and this panic arrive together.

Our workaround (the shim referenced on #90)

At wallet init, before any TLS client is constructed:

// tree unions `ring` (electrum-client) + `aws-lc-rs` (rustls default) — ambiguous,
// so select one explicitly. Idempotent; the Err on a second call is expected.
let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();

It works, but install_default() makes a process-global choice from library/app init — the decision rustls says the top-level binary should own. Pushing that onto every downstream app embedding ldk-node is the awkward part.

Fix direction (deferring to you on the right layer)

Rather than depend on the process-level default, the code that builds these ClientConfigs (ldk-node's own TLS clients, or bitreq's) could construct them with an explicit provider via ClientConfig::builder_with_provider(...), so a multi-provider tree neither panics nor forces downstream to install a process-global default.

If you conclude the right place is bitreq, we're glad to carry this straight to corepc#661 — we just surfaced it here per your request, since ldk-node is where the two providers collide.

Repro / standing offer

The standalone snippet above pins the panic to rustls. Our full-stack capture (A12 device + desktop wallet-cli against staging VSS) is scripted — point us at a branch and we'll confirm the panic is gone, before/after, same as we've done for #93 / #94 / corepc#661.

[big14way]

@hash-money dug into this on main. The root cause isn't quite where the report puts it, and there's a second bug.

aws-lc-rs comes from chain-electrum, not the VSS/HTTP path. cargo tree -e no-dev with chain-electrum off: rustls v0.23.43 feats=ring,std,tls12, and no aws-lc-rs in the tree at all. With it on: feats=aws-lc-rs,aws_lc_rs,default,log,logging,prefer-post-quantum,ring,std,tls12. bitreq (the VSS/HTTP path) asks only for ["ring","std","tls12"].

electrum-client 0.25 has an optional dep rustls and a same-named feature rustls = ["webpki-roots","dep:rustls","rustls/default"]. The explicit feature shadows the implicit one, so rustls-ring's "rustls/ring" also enables the rustls feature → rustls/default → aws-lc-rs. The resolved tree shows electrum-client feats=…,rustls,rustls-ring,use-rustls-ring,… even though bdk_electrum and lightning-transaction-sync both pin default-features = false (note default is absent from that list). So 4b45d7c ("Switch to use rustls-ring everywhere") never actually produced the ring-only tree it advertises — which matters for the step 2 @tnull sketched in #600.

The safeguard runs after the first TLS request on the VSS path. optionally_install_rustls_cryptoprovider() is called in build_with_store_internal (builder.rs:1494), but all four build_with_vss_store* methods construct the store first (builder.rs:803, 841, 869, 895). VssStore::new does a blocking determine_and_write_schema_version on a scoped thread (io/vss_store.rs:128-143 → :810), and vss-client-ng → bitreq builds its ClientConfig from the process default. No corepc#661 needed — this is reachable on the current pin against an https:// VSS URL. The scoped-thread join() catches the unwind and turns it into BuildError::KVStoreSetupFailed, so it surfaces as a confusing setup failure rather than the rustls message; under panic = "abort" profiles it aborts.

Happy to take: hoist the install ahead of VSS store construction, declare rustls's ring feature explicitly in our own manifest (builder.rs:2508 calls rustls::crypto::ring::default_provider(), but Cargo.toml:108 declares rustls with default-features = false and no features, so that only compiles because something else in the tree turns ring on), and carry the manifest fix upstream to electrum-client — renaming the dep to rustls_dep = { package = "rustls" } removes the shadowing and yields rustls feats=ring,std,tls12.

Do you want provider choice behind tls-ring/tls-aws-lc-rs features, along the lines of what @TheBlueMatt described on #600, or keep a single pinned provider?

[hash-money]

@big14way thanks — we re-derived all of this independently and it holds. Corrections to our own report first, then the one thing you may not have seen, then your question.

Confirmed (cargo tree on main 6480bff):

# default features
rustls v0.23.43 feats=aws-lc-rs,aws_lc_rs,default,log,logging,prefer-post-quantum,ring,std,tls12
electrum-client v0.25.0 feats=byteorder,libc,proxy,rustls,rustls-ring,use-rustls-ring,webpki-roots,winapi

# --no-default-features --features chain-esplora,chain-bitcoind,storage-sqlite,storage-filesystem,storage-vss,unified-payments
rustls v0.23.43 feats=ring,std,tls12 # no aws-lc-rs in the tree

Minimal repro of the shadowing, one dependency only: electrum-client = { version = "=0.25.0", default-features = false, features = ["use-rustls-ring"] } resolves rustls with aws-lc-rs,…,default,…,ring; the same crate with rustls-ring = [..., "rustls?/ring", "rustls?/logging", "rustls?/std", "rustls?/tls12"] resolves rustls feats=log,logging,ring,std,tls12.

Ordering: read as you describe — build_with_vss_store (builder.rs:797–809) runs VssStoreBuilder::build_with_sigs_auth → VssStore::new → the scoped-thread determine_and_write_schema_version (io/vss_store.rs:128–143) before build_with_store_internal reaches optionally_install_rustls_cryptoprovider() (:1494); the join() error becomes KVStoreSetupFailed (:805). One nuance: the rustls panic text still reaches stderr through the default panic hook — it's only the returned BuildError that loses it. We have not executed this path on main; see the offer below.

Two corrections to the opening report:

1. The aws-lc-rs attribution was wrong. It is not "rustls' default on the VSS/HTTP path" — in our tree, exactly as in yours, the only source is electrum-client's rustls feature leaking rustls/default. Substituting the rustls?/… manifest (and no default features on our own rustls dep) leaves our Android target at rustls feats=log,logging,ring,std,tls12 — aws-lc-rs gone entirely.
2. "The released stack doesn't hit it" was a lockfile artifact, not a property of the code. Our Cargo.lock holds bitreq 0.3.4, which is on rustls 0.21 (no CryptoProvider concept). bitreq ≥ 0.3.5 requires rustls ^0.23.38, and 0.3.7 — what a fresh resolve of main's bitreq = "0.3" gets — builds its config with ClientConfig::builder() (connection/rustls_stream.rs:36–50). So corepc#661 was incidental: it was simply the first rustls-0.23 bitreq we ran. No patch needed to reach it on main; your reading is right.

Already in flight upstream: the electrum-client half exists as bitcoindevkit/rust-electrum-client#220 (open since 2026-07-02, same rustls?/ring diff, tACK'd, awaiting maintainer review), and electrum-client#183 already prefers ring internally when both features are on — so no second electrum-client PR is needed, and the ldk-node-side exposure reduces to the VSS ordering plus the explicit ring feature in ldk-node's own manifest.

On tls-ring/tls-aws-lc-rs features vs. a single pinned provider: that's @tnull's and @TheBlueMatt's call (they already leaned toward downstream-selectable features on #600), and we don't need the toggle either way. What we need is narrower: (a) the install — or an explicit-provider ClientConfig — runs before any TLS client is constructed, VssStore::new included; (b) a provider already installed by the host process keeps winning, as the current is_none() check does. Two data points, not asks: we consume ldk-node through orange-sdk, so an ldk-node feature only reaches us if orange-sdk re-exports it — the app-level install_default() is what we rely on regardless; and with #220 landed, ring is what everything else in our tree (spark-sdk, cdk, reqwest 0.12, and bitreq ≥0.3.5 itself) already selects, so a single pinned ring would be the zero-config outcome for us and would drop aws-lc-sys from our mobile build.

Offer stands: our harness (Galaxy A12 + wallet-cli against our staging VSS over https://) is scripted; point us at a branch with the hoist and we'll post before/after on the current tree. No urgency from our side — we install at app init, so we route around the ordering today.