feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground) - #28

Merged
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls
Apr 23, 2026
Merged

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground)#28
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls

Conversation

@moonming

Copy link
Copy Markdown
Member

Summary

Adds two closely-related DP-side capabilities required for AISIX to act as an aisix.cloud tenant:

  1. etcd mTLS client — the aisix.cloud DP Manager serves etcd v3 over mTLS, so the DP needs to read a CA + client cert + client key from disk and wire them into the existing etcd-clientConnectOptions.
  2. Managed-mode switch — when enabled, the admin API listener, admin UI, and Playground endpoint are not bound. All configuration flows from etcd; resource mutations happen through the cloud control plane.

This is the first of ~three DP-side PRs. Registration (fetching the mTLS bundle from cp-api at boot), heartbeat, and local snapshot fallback land in follow-ups.

Config shape

etcd:
endpoints: ["https://etcd.aisix.cloud:2379"]tls:
ca_cert_file: "/etc/aisix/mtls/ca.crt"client_cert_file: "/etc/aisix/mtls/client.crt"client_key_file: "/etc/aisix/mtls/client.key"# domain_name: "etcd.aisix.cloud" # optional; defaults to endpoints[0] hostproxy:
addr: "0.0.0.0:3000"managed:
enabled: true

Standalone users are unaffected: new fields are all optional and defaulted, validate() keeps the existing invariants when managed.enabled = false.

What the managed switch actually flips

ComponentStandaloneManaged
Proxy listener✅ bound✅ bound
etcd provider + watch supervisor
Admin etcd client (write path)❌ skipped
Admin router (/admin/v1/*, UI, Playground)✅ bound❌ never constructed
Admin TCP listener✅ bound❌ never bound
/metrics (currently mounted on admin)✅ served❌ not served

The last row is worth calling out: Phase 1 mounts Prometheus on the admin listener, so managed mode loses it. Adding a dedicated observability listener is a follow-up; for aisix.cloud tenants, Prometheus scraping is expected to happen on the cloud control plane side anyway.

New helper

build_etcd_connect_options(&EtcdConfig) -> anyhow::Result<Option<ConnectOptions>>:

  • Returns None for plain HTTP etcd (keeps the hot path cheap — no pointless ConnectOptions::new() allocation when nothing needs wiring).
  • Reads the three PEM files and constructs TlsOptions::new().domain_name(...).ca_certificate(...).identity(...).
  • Missing files surface with the config key name (etcd.tls.ca_cert_file = "/...") in the error so operators don't have to diff config against filesystem state.

default_domain_from_endpoint() extracts the SNI from URL-ish strings: http://, https://, bare host:port, and IPv6 literals [::1]:2379 — table-tested.

Tests

cargo test --workspace --all-features green (407 tests). New cases:

  • aisix-core::config::tests::managed_mode_lets_admin_fields_be_omitted — minimum aisix.cloud tenant YAML loads cleanly.
  • aisix-core::config::tests::standalone_still_requires_admin_keys_even_with_managed_false — original invariant preserved.
  • aisix-core::config::tests::parses_etcd_tls_block — round-trip for all four TLS fields.
  • aisix-server::tests::default_domain_strips_scheme_port_and_brackets — SNI extractor table.
  • aisix-server::tests::build_connect_options_none_when_plain_http — hot path stays zero-cost.
  • aisix-server::tests::build_connect_options_surfaces_missing_cert_files — operator-friendly error.

cargo fmt + cargo clippy --workspace --all-targets -- -D warnings clean.

Explicitly out of scope (follow-up PRs on this branch's successor stack)

  • Registration flow: the DP currently expects the mTLS bundle already on disk. The next PR adds POST /dp/register client that exchanges a one-time Deployment Token for the bundle at boot and persists it atomically.
  • Heartbeat: periodic POST /api/ai_dataplane/heartbeat so cp-api knows the DP is alive (and so the Gateway page in aisix.cloud shows green dots).
  • Local config snapshot: continue serving proxy traffic when the etcd watch disconnects mid-flight (see aisix.cloud PRD prd-09 §9.7.2).
  • Structured request logs + hashes: the observability schema aisix.cloud telemetry consumes.

Relationship

Paired with the aisix.cloud side in api7/AISIX-Cloud#8: that PR's GatewayHandlers.create calls IssueDataplaneCertificate on api7ee CP and returns the bundle to the user, who then drops it on their DP machine. The DP then boots with this PR's new config block pointing at those files.

…und)
Required for AISIX data planes to talk to the aisix.cloud control
plane: the CP's DP Manager serves etcd v3 over mTLS (see the
aisix.cloud PRD prd-09 §9.3.3). Phase 1 of the DP-side changes —
follow-up PRs wire registration + heartbeat + local snapshot.
## What's new
### `aisix-core::config`
- `EtcdConfig.tls: Option<EtcdTlsConfig>` — new optional mTLS bundle.
Three PEM file paths (CA cert, client cert, client key) plus an
optional `domain_name` for SNI. Defaults derive the domain from the
first endpoint's hostname.
- `Config.managed: ManagedConfig { enabled: bool }` — new top-level
switch. Defaults to standalone so existing configs keep working.
- `AdminConfig` now implements `Default` so managed-mode configs
can omit the `admin:` block entirely.
- `validate()` relaxes the `admin.addr` + `admin.admin_keys`
invariants when `managed.enabled = true`, and keeps them as-is
otherwise — no silent regression for standalone setups.
### `aisix-server::main`
- New `build_etcd_connect_options(&EtcdConfig) -> Option<ConnectOptions>`
helper. Returns `None` for plain HTTP (keep the test path cheap),
wires `with_user` + `with_tls` when present, surfaces missing
cert-file errors with the config key name in the message.
- `default_domain_from_endpoint()` extracts the SNI from URL-like
endpoint strings (`http://host:port`, `https://host:port`, bare
`host:port`, and IPv6 literals with brackets).
- The `EtcdConfigProvider::connect` + the separate admin `Client`
now share the same options (user + mTLS).
- **Admin listener is conditional.** In managed mode the admin
surface is never built:
* `admin_client` stays `None` (no second etcd connection)
* `admin_state` / `admin_router` are not constructed
* The admin TCP listener is not bound
* The Playground endpoint (mounted inside admin) vanishes
The proxy listener keeps running with the same request path.
- `run()` awaits the admin task via an `Option<JoinHandle>` so a
managed-mode start-up no longer joins on a nonexistent future.
### `config.example.yaml`
- Commented-out `etcd.tls` block with the three PEM paths.
- Commented-out `managed.enabled: true` section with a short
explanation of what flips in that mode.
## Tests
### `aisix-core`
- `managed_mode_lets_admin_fields_be_omitted`: minimum aisix.cloud
tenant YAML loads without an `admin:` block.
- `standalone_still_requires_admin_keys_even_with_managed_false`:
original invariant preserved for non-managed configs.
- `parses_etcd_tls_block`: round-trip for all four TLS fields.
### `aisix-server`
- `default_domain_strips_scheme_port_and_brackets`: table for the
SNI extractor including IPv6 brackets and bare-host cases.
- `build_connect_options_none_when_plain_http`: plain HTTP etcd
doesn't synthesise options (hot path).
- `build_connect_options_surfaces_missing_cert_files`: operator
sees *which* file is missing without grepping filesystem state.
`cargo fmt` / `cargo clippy --workspace --all-targets -- -D warnings`
clean. `cargo test --workspace --all-features` green (407 tests).
## Explicitly out of scope (follow-up PRs)
- DP registration flow (`POST /dp/register` against cp-api) that
*fetches* the mTLS bundle and persists it. This PR expects the
bundle already on disk — integration test path.
- Heartbeat (`POST /api/ai_dataplane/heartbeat` every 15s).
- Local config snapshot so the DP serves from cache when the etcd
connection dies mid-flight.
- Structured request logs + per-request hashes (prd-09 §9.6.2).
CopilotAI review requested due to automatic review settings April 23, 2026 07:58

CopilotAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds dataplane support for aisix.cloud tenants by introducing etcd mTLS client wiring and a managed-mode switch that disables the standalone admin surface.

Changes:

  • Add etcd.tls config (CA/cert/key + optional domain_name) and build etcd-clientConnectOptions with mTLS.
  • Add managed.enabled config and skip admin etcd client + admin router/listener when managed mode is on.
  • Update example config and add tests for domain derivation, connect options behavior, and managed/standalone config validation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

FileDescription
crates/aisix-server/src/main.rsBuild etcd ConnectOptions (mTLS/auth) and gate admin listener/router creation behind managed mode.
crates/aisix-core/src/lib.rsRe-export new config types (EtcdTlsConfig, ManagedConfig).
crates/aisix-core/src/config.rsExtend config schema with etcd.tls + managed, default admin for managed configs, and update validation rules.
config.example.yamlDocument new etcd.tls block and managed.enabled toggle.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +236 to +241
if let (Some(user), Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()) {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_etcd_connect_options() only applies basic auth if bothetcd.user and etcd.password_env are set. If a user sets only one of these fields (common config mistake), it will silently skip auth and likely fail with an opaque etcd permission error. Prefer returning a config error (or add a Config::validate() check) when exactly one of the two is set.

Suggested change
iflet(Some(user),Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()){
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
match(etcd.user.as_ref(), etcd.password_env.as_ref()){
(Some(user),Some(env_key)) => {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
}
(None,None) => {}
(Some(_),None) => {
returnErr(anyhow::anyhow!(
"etcd.user is set but etcd.password_env is missing; set both fields to enable etcd basic auth"
));
}
(None,Some(_)) => {
returnErr(anyhow::anyhow!(
"etcd.password_env is set but etcd.user is missing; set both fields to enable etcd basic auth"
));
}

Copilot uses AI. Check for mistakes.
fn build_etcd_connect_options(etcd: &EtcdConfig) -> anyhow::Result<Option<ConnectOptions>> {
let mut needs_options = false;
let mut options = ConnectOptions::new();

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EtcdConfig exposes dial_timeout_ms and request_timeout_ms, but build_etcd_connect_options() never uses them and there are no other call sites wiring these into etcd-client. This makes the timeout knobs in config.example.yaml ineffective. Consider setting the appropriate timeouts on ConnectOptions (and marking needs_options = true accordingly) so the config fields actually take effect.

Suggested change
ifletSome(dial_timeout_ms) = etcd.dial_timeout_ms{
options = options.with_connect_timeout(std::time::Duration::from_millis(dial_timeout_ms));
needs_options = true;
}
ifletSome(request_timeout_ms) = etcd.request_timeout_ms{
options = options.with_timeout(std::time::Duration::from_millis(request_timeout_ms));
needs_options = true;
}

Copilot uses AI. Check for mistakes.
Comment on lines +226 to +227
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc comment says cert/key I/O failures “bubble up as a … BootstrapError”, but this helper returns anyhow::Result and the produced errors are anyhow::Error strings. Consider adjusting the wording to avoid implying a specific error type here.

Suggested change
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.
/// failure bubbles up as a nicelycontextualised error at the same
/// point as other etcd connection errors.

Copilot uses AI. Check for mistakes.
Comment on lines +389 to +393
if self.admin.addr.parse::<std::net::SocketAddr>().is_err() {
return Err(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AdminConfig::default_addr() sets 127.0.0.1:0, and Config::validate() only checks that admin.addr parses (in standalone mode). This allows a standalone config that omits admin.addr (but sets admin_keys) to bind the admin listener on an ephemeral port, which contradicts the comment and is operationally surprising. Consider either (a) defaulting to the previous explicit port, or (b) rejecting port 0 / the default value in validate() when managed.enabled is false with a clear error message.

Suggested change
ifself.admin.addr.parse::<std::net::SocketAddr>().is_err(){
returnErr(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));
let admin_addr = self
.admin
.addr
.parse::<std::net::SocketAddr>()
.map_err(|_| {
BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
))
})?;
if admin_addr.port() == 0{
returnErr(BootstrapError::Config(
"admin.addr must use an explicit non-zero port \
(required when managed.enabled is false)"
.into(),
));

Copilot uses AI. Check for mistakes.
moonming added a commit that referenced this pull request Apr 23, 2026
Free-tier Actions storage is 500 MB, shared across the whole repo.
Each \`aisix-bin\` artifact is ~72 MB and we publish one per CI run,
so storage saturates after <10 main-branch pushes and blocks
\`actions/upload-artifact\` on every subsequent PR (the failure that
paused #28 twice today).
Retention tightened per artifact by expected re-read horizon:
- aisix-bin 1 day (consumed by the same-day e2e job only)
- ui-dist 7 days (consumed by same-day e2e; light enough
to keep a week for manual inspection)
- coverage-* 7 days (manual download for debugging flaky
coverage gates; LCOV is tiny)
Nothing downstream relies on week+ old binaries — \`build-bin\` is
re-runnable from source and the \`needs:\` chain on \`build-aisix
(instrumented)\` → \`e2e\` already re-produces the artifact when an
earlier run has expired.
No behaviour change on passing runs; only limits how long stale
runs squat on quota.
@moonming
moonming merged commit 56d2d54 into mainApr 23, 2026
13 of 17 checks passed
@moonming
moonming deleted the feat/managed-mode-mtls branch April 23, 2026 08:57
jarvis9443 added a commit that referenced this pull request Jun 2, 2026
Output guardrails only inspected message.content, so client-visible
output that lives elsewhere bypassed content/DLP checks:
- tool_calls / Anthropic tool_use (normalized into message.extra) are
now folded into a single guardrail-inspected text view via
ChatResponse::guardrail_output_text(), used by the keyword, text-
moderation, Bedrock, and Prompt Shield output checks (#3/#18/#21).
Reasoning/thinking content is intentionally left out of scope.
- Non-streaming cache hits now run the resolved output guardrail chain
before returning the stored body, instead of replaying it unchecked
(#28). Streaming output guardrails already run end-of-stream.
Part of #448 (findings #3, #18, #21, #28)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@moonming
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground) - #28

Merged
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls
Apr 23, 2026
Merged

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground)#28
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls

Conversation

@moonming

Copy link
Copy Markdown
Member

Summary

Adds two closely-related DP-side capabilities required for AISIX to act as an aisix.cloud tenant:

  1. etcd mTLS client — the aisix.cloud DP Manager serves etcd v3 over mTLS, so the DP needs to read a CA + client cert + client key from disk and wire them into the existing etcd-clientConnectOptions.
  2. Managed-mode switch — when enabled, the admin API listener, admin UI, and Playground endpoint are not bound. All configuration flows from etcd; resource mutations happen through the cloud control plane.

This is the first of ~three DP-side PRs. Registration (fetching the mTLS bundle from cp-api at boot), heartbeat, and local snapshot fallback land in follow-ups.

Config shape

etcd:
endpoints: ["https://etcd.aisix.cloud:2379"]tls:
ca_cert_file: "/etc/aisix/mtls/ca.crt"client_cert_file: "/etc/aisix/mtls/client.crt"client_key_file: "/etc/aisix/mtls/client.key"# domain_name: "etcd.aisix.cloud" # optional; defaults to endpoints[0] hostproxy:
addr: "0.0.0.0:3000"managed:
enabled: true

Standalone users are unaffected: new fields are all optional and defaulted, validate() keeps the existing invariants when managed.enabled = false.

What the managed switch actually flips

ComponentStandaloneManaged
Proxy listener✅ bound✅ bound
etcd provider + watch supervisor
Admin etcd client (write path)❌ skipped
Admin router (/admin/v1/*, UI, Playground)✅ bound❌ never constructed
Admin TCP listener✅ bound❌ never bound
/metrics (currently mounted on admin)✅ served❌ not served

The last row is worth calling out: Phase 1 mounts Prometheus on the admin listener, so managed mode loses it. Adding a dedicated observability listener is a follow-up; for aisix.cloud tenants, Prometheus scraping is expected to happen on the cloud control plane side anyway.

New helper

build_etcd_connect_options(&EtcdConfig) -> anyhow::Result<Option<ConnectOptions>>:

  • Returns None for plain HTTP etcd (keeps the hot path cheap — no pointless ConnectOptions::new() allocation when nothing needs wiring).
  • Reads the three PEM files and constructs TlsOptions::new().domain_name(...).ca_certificate(...).identity(...).
  • Missing files surface with the config key name (etcd.tls.ca_cert_file = "/...") in the error so operators don't have to diff config against filesystem state.

default_domain_from_endpoint() extracts the SNI from URL-ish strings: http://, https://, bare host:port, and IPv6 literals [::1]:2379 — table-tested.

Tests

cargo test --workspace --all-features green (407 tests). New cases:

  • aisix-core::config::tests::managed_mode_lets_admin_fields_be_omitted — minimum aisix.cloud tenant YAML loads cleanly.
  • aisix-core::config::tests::standalone_still_requires_admin_keys_even_with_managed_false — original invariant preserved.
  • aisix-core::config::tests::parses_etcd_tls_block — round-trip for all four TLS fields.
  • aisix-server::tests::default_domain_strips_scheme_port_and_brackets — SNI extractor table.
  • aisix-server::tests::build_connect_options_none_when_plain_http — hot path stays zero-cost.
  • aisix-server::tests::build_connect_options_surfaces_missing_cert_files — operator-friendly error.

cargo fmt + cargo clippy --workspace --all-targets -- -D warnings clean.

Explicitly out of scope (follow-up PRs on this branch's successor stack)

  • Registration flow: the DP currently expects the mTLS bundle already on disk. The next PR adds POST /dp/register client that exchanges a one-time Deployment Token for the bundle at boot and persists it atomically.
  • Heartbeat: periodic POST /api/ai_dataplane/heartbeat so cp-api knows the DP is alive (and so the Gateway page in aisix.cloud shows green dots).
  • Local config snapshot: continue serving proxy traffic when the etcd watch disconnects mid-flight (see aisix.cloud PRD prd-09 §9.7.2).
  • Structured request logs + hashes: the observability schema aisix.cloud telemetry consumes.

Relationship

Paired with the aisix.cloud side in api7/AISIX-Cloud#8: that PR's GatewayHandlers.create calls IssueDataplaneCertificate on api7ee CP and returns the bundle to the user, who then drops it on their DP machine. The DP then boots with this PR's new config block pointing at those files.

…und)
Required for AISIX data planes to talk to the aisix.cloud control
plane: the CP's DP Manager serves etcd v3 over mTLS (see the
aisix.cloud PRD prd-09 §9.3.3). Phase 1 of the DP-side changes —
follow-up PRs wire registration + heartbeat + local snapshot.
## What's new
### `aisix-core::config`
- `EtcdConfig.tls: Option<EtcdTlsConfig>` — new optional mTLS bundle.
Three PEM file paths (CA cert, client cert, client key) plus an
optional `domain_name` for SNI. Defaults derive the domain from the
first endpoint's hostname.
- `Config.managed: ManagedConfig { enabled: bool }` — new top-level
switch. Defaults to standalone so existing configs keep working.
- `AdminConfig` now implements `Default` so managed-mode configs
can omit the `admin:` block entirely.
- `validate()` relaxes the `admin.addr` + `admin.admin_keys`
invariants when `managed.enabled = true`, and keeps them as-is
otherwise — no silent regression for standalone setups.
### `aisix-server::main`
- New `build_etcd_connect_options(&EtcdConfig) -> Option<ConnectOptions>`
helper. Returns `None` for plain HTTP (keep the test path cheap),
wires `with_user` + `with_tls` when present, surfaces missing
cert-file errors with the config key name in the message.
- `default_domain_from_endpoint()` extracts the SNI from URL-like
endpoint strings (`http://host:port`, `https://host:port`, bare
`host:port`, and IPv6 literals with brackets).
- The `EtcdConfigProvider::connect` + the separate admin `Client`
now share the same options (user + mTLS).
- **Admin listener is conditional.** In managed mode the admin
surface is never built:
* `admin_client` stays `None` (no second etcd connection)
* `admin_state` / `admin_router` are not constructed
* The admin TCP listener is not bound
* The Playground endpoint (mounted inside admin) vanishes
The proxy listener keeps running with the same request path.
- `run()` awaits the admin task via an `Option<JoinHandle>` so a
managed-mode start-up no longer joins on a nonexistent future.
### `config.example.yaml`
- Commented-out `etcd.tls` block with the three PEM paths.
- Commented-out `managed.enabled: true` section with a short
explanation of what flips in that mode.
## Tests
### `aisix-core`
- `managed_mode_lets_admin_fields_be_omitted`: minimum aisix.cloud
tenant YAML loads without an `admin:` block.
- `standalone_still_requires_admin_keys_even_with_managed_false`:
original invariant preserved for non-managed configs.
- `parses_etcd_tls_block`: round-trip for all four TLS fields.
### `aisix-server`
- `default_domain_strips_scheme_port_and_brackets`: table for the
SNI extractor including IPv6 brackets and bare-host cases.
- `build_connect_options_none_when_plain_http`: plain HTTP etcd
doesn't synthesise options (hot path).
- `build_connect_options_surfaces_missing_cert_files`: operator
sees *which* file is missing without grepping filesystem state.
`cargo fmt` / `cargo clippy --workspace --all-targets -- -D warnings`
clean. `cargo test --workspace --all-features` green (407 tests).
## Explicitly out of scope (follow-up PRs)
- DP registration flow (`POST /dp/register` against cp-api) that
*fetches* the mTLS bundle and persists it. This PR expects the
bundle already on disk — integration test path.
- Heartbeat (`POST /api/ai_dataplane/heartbeat` every 15s).
- Local config snapshot so the DP serves from cache when the etcd
connection dies mid-flight.
- Structured request logs + per-request hashes (prd-09 §9.6.2).
CopilotAI review requested due to automatic review settings April 23, 2026 07:58

CopilotAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds dataplane support for aisix.cloud tenants by introducing etcd mTLS client wiring and a managed-mode switch that disables the standalone admin surface.

Changes:

  • Add etcd.tls config (CA/cert/key + optional domain_name) and build etcd-clientConnectOptions with mTLS.
  • Add managed.enabled config and skip admin etcd client + admin router/listener when managed mode is on.
  • Update example config and add tests for domain derivation, connect options behavior, and managed/standalone config validation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

FileDescription
crates/aisix-server/src/main.rsBuild etcd ConnectOptions (mTLS/auth) and gate admin listener/router creation behind managed mode.
crates/aisix-core/src/lib.rsRe-export new config types (EtcdTlsConfig, ManagedConfig).
crates/aisix-core/src/config.rsExtend config schema with etcd.tls + managed, default admin for managed configs, and update validation rules.
config.example.yamlDocument new etcd.tls block and managed.enabled toggle.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +236 to +241
if let (Some(user), Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()) {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_etcd_connect_options() only applies basic auth if bothetcd.user and etcd.password_env are set. If a user sets only one of these fields (common config mistake), it will silently skip auth and likely fail with an opaque etcd permission error. Prefer returning a config error (or add a Config::validate() check) when exactly one of the two is set.

Suggested change
iflet(Some(user),Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()){
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
match(etcd.user.as_ref(), etcd.password_env.as_ref()){
(Some(user),Some(env_key)) => {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
}
(None,None) => {}
(Some(_),None) => {
returnErr(anyhow::anyhow!(
"etcd.user is set but etcd.password_env is missing; set both fields to enable etcd basic auth"
));
}
(None,Some(_)) => {
returnErr(anyhow::anyhow!(
"etcd.password_env is set but etcd.user is missing; set both fields to enable etcd basic auth"
));
}

Copilot uses AI. Check for mistakes.
fn build_etcd_connect_options(etcd: &EtcdConfig) -> anyhow::Result<Option<ConnectOptions>> {
let mut needs_options = false;
let mut options = ConnectOptions::new();

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EtcdConfig exposes dial_timeout_ms and request_timeout_ms, but build_etcd_connect_options() never uses them and there are no other call sites wiring these into etcd-client. This makes the timeout knobs in config.example.yaml ineffective. Consider setting the appropriate timeouts on ConnectOptions (and marking needs_options = true accordingly) so the config fields actually take effect.

Suggested change
ifletSome(dial_timeout_ms) = etcd.dial_timeout_ms{
options = options.with_connect_timeout(std::time::Duration::from_millis(dial_timeout_ms));
needs_options = true;
}
ifletSome(request_timeout_ms) = etcd.request_timeout_ms{
options = options.with_timeout(std::time::Duration::from_millis(request_timeout_ms));
needs_options = true;
}

Copilot uses AI. Check for mistakes.
Comment on lines +226 to +227
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc comment says cert/key I/O failures “bubble up as a … BootstrapError”, but this helper returns anyhow::Result and the produced errors are anyhow::Error strings. Consider adjusting the wording to avoid implying a specific error type here.

Suggested change
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.
/// failure bubbles up as a nicelycontextualised error at the same
/// point as other etcd connection errors.

Copilot uses AI. Check for mistakes.
Comment on lines +389 to +393
if self.admin.addr.parse::<std::net::SocketAddr>().is_err() {
return Err(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AdminConfig::default_addr() sets 127.0.0.1:0, and Config::validate() only checks that admin.addr parses (in standalone mode). This allows a standalone config that omits admin.addr (but sets admin_keys) to bind the admin listener on an ephemeral port, which contradicts the comment and is operationally surprising. Consider either (a) defaulting to the previous explicit port, or (b) rejecting port 0 / the default value in validate() when managed.enabled is false with a clear error message.

Suggested change
ifself.admin.addr.parse::<std::net::SocketAddr>().is_err(){
returnErr(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));
let admin_addr = self
.admin
.addr
.parse::<std::net::SocketAddr>()
.map_err(|_| {
BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
))
})?;
if admin_addr.port() == 0{
returnErr(BootstrapError::Config(
"admin.addr must use an explicit non-zero port \
(required when managed.enabled is false)"
.into(),
));

Copilot uses AI. Check for mistakes.
moonming added a commit that referenced this pull request Apr 23, 2026
Free-tier Actions storage is 500 MB, shared across the whole repo.
Each \`aisix-bin\` artifact is ~72 MB and we publish one per CI run,
so storage saturates after <10 main-branch pushes and blocks
\`actions/upload-artifact\` on every subsequent PR (the failure that
paused #28 twice today).
Retention tightened per artifact by expected re-read horizon:
- aisix-bin 1 day (consumed by the same-day e2e job only)
- ui-dist 7 days (consumed by same-day e2e; light enough
to keep a week for manual inspection)
- coverage-* 7 days (manual download for debugging flaky
coverage gates; LCOV is tiny)
Nothing downstream relies on week+ old binaries — \`build-bin\` is
re-runnable from source and the \`needs:\` chain on \`build-aisix
(instrumented)\` → \`e2e\` already re-produces the artifact when an
earlier run has expired.
No behaviour change on passing runs; only limits how long stale
runs squat on quota.
@moonming
moonming merged commit 56d2d54 into mainApr 23, 2026
13 of 17 checks passed
@moonming
moonming deleted the feat/managed-mode-mtls branch April 23, 2026 08:57
jarvis9443 added a commit that referenced this pull request Jun 2, 2026
Output guardrails only inspected message.content, so client-visible
output that lives elsewhere bypassed content/DLP checks:
- tool_calls / Anthropic tool_use (normalized into message.extra) are
now folded into a single guardrail-inspected text view via
ChatResponse::guardrail_output_text(), used by the keyword, text-
moderation, Bedrock, and Prompt Shield output checks (#3/#18/#21).
Reasoning/thinking content is intentionally left out of scope.
- Non-streaming cache hits now run the resolved output guardrail chain
before returning the stored body, instead of replaying it unchecked
(#28). Streaming output guardrails already run end-of-stream.
Part of #448 (findings #3, #18, #21, #28)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@moonming
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground) - #28

Merged
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls
Apr 23, 2026
Merged

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground)#28
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls

Conversation

@moonming

Copy link
Copy Markdown
Member

Summary

Adds two closely-related DP-side capabilities required for AISIX to act as an aisix.cloud tenant:

  1. etcd mTLS client — the aisix.cloud DP Manager serves etcd v3 over mTLS, so the DP needs to read a CA + client cert + client key from disk and wire them into the existing etcd-clientConnectOptions.
  2. Managed-mode switch — when enabled, the admin API listener, admin UI, and Playground endpoint are not bound. All configuration flows from etcd; resource mutations happen through the cloud control plane.

This is the first of ~three DP-side PRs. Registration (fetching the mTLS bundle from cp-api at boot), heartbeat, and local snapshot fallback land in follow-ups.

Config shape

etcd:
endpoints: ["https://etcd.aisix.cloud:2379"]tls:
ca_cert_file: "/etc/aisix/mtls/ca.crt"client_cert_file: "/etc/aisix/mtls/client.crt"client_key_file: "/etc/aisix/mtls/client.key"# domain_name: "etcd.aisix.cloud" # optional; defaults to endpoints[0] hostproxy:
addr: "0.0.0.0:3000"managed:
enabled: true

Standalone users are unaffected: new fields are all optional and defaulted, validate() keeps the existing invariants when managed.enabled = false.

What the managed switch actually flips

ComponentStandaloneManaged
Proxy listener✅ bound✅ bound
etcd provider + watch supervisor
Admin etcd client (write path)❌ skipped
Admin router (/admin/v1/*, UI, Playground)✅ bound❌ never constructed
Admin TCP listener✅ bound❌ never bound
/metrics (currently mounted on admin)✅ served❌ not served

The last row is worth calling out: Phase 1 mounts Prometheus on the admin listener, so managed mode loses it. Adding a dedicated observability listener is a follow-up; for aisix.cloud tenants, Prometheus scraping is expected to happen on the cloud control plane side anyway.

New helper

build_etcd_connect_options(&EtcdConfig) -> anyhow::Result<Option<ConnectOptions>>:

  • Returns None for plain HTTP etcd (keeps the hot path cheap — no pointless ConnectOptions::new() allocation when nothing needs wiring).
  • Reads the three PEM files and constructs TlsOptions::new().domain_name(...).ca_certificate(...).identity(...).
  • Missing files surface with the config key name (etcd.tls.ca_cert_file = "/...") in the error so operators don't have to diff config against filesystem state.

default_domain_from_endpoint() extracts the SNI from URL-ish strings: http://, https://, bare host:port, and IPv6 literals [::1]:2379 — table-tested.

Tests

cargo test --workspace --all-features green (407 tests). New cases:

  • aisix-core::config::tests::managed_mode_lets_admin_fields_be_omitted — minimum aisix.cloud tenant YAML loads cleanly.
  • aisix-core::config::tests::standalone_still_requires_admin_keys_even_with_managed_false — original invariant preserved.
  • aisix-core::config::tests::parses_etcd_tls_block — round-trip for all four TLS fields.
  • aisix-server::tests::default_domain_strips_scheme_port_and_brackets — SNI extractor table.
  • aisix-server::tests::build_connect_options_none_when_plain_http — hot path stays zero-cost.
  • aisix-server::tests::build_connect_options_surfaces_missing_cert_files — operator-friendly error.

cargo fmt + cargo clippy --workspace --all-targets -- -D warnings clean.

Explicitly out of scope (follow-up PRs on this branch's successor stack)

  • Registration flow: the DP currently expects the mTLS bundle already on disk. The next PR adds POST /dp/register client that exchanges a one-time Deployment Token for the bundle at boot and persists it atomically.
  • Heartbeat: periodic POST /api/ai_dataplane/heartbeat so cp-api knows the DP is alive (and so the Gateway page in aisix.cloud shows green dots).
  • Local config snapshot: continue serving proxy traffic when the etcd watch disconnects mid-flight (see aisix.cloud PRD prd-09 §9.7.2).
  • Structured request logs + hashes: the observability schema aisix.cloud telemetry consumes.

Relationship

Paired with the aisix.cloud side in api7/AISIX-Cloud#8: that PR's GatewayHandlers.create calls IssueDataplaneCertificate on api7ee CP and returns the bundle to the user, who then drops it on their DP machine. The DP then boots with this PR's new config block pointing at those files.

…und)
Required for AISIX data planes to talk to the aisix.cloud control
plane: the CP's DP Manager serves etcd v3 over mTLS (see the
aisix.cloud PRD prd-09 §9.3.3). Phase 1 of the DP-side changes —
follow-up PRs wire registration + heartbeat + local snapshot.
## What's new
### `aisix-core::config`
- `EtcdConfig.tls: Option<EtcdTlsConfig>` — new optional mTLS bundle.
Three PEM file paths (CA cert, client cert, client key) plus an
optional `domain_name` for SNI. Defaults derive the domain from the
first endpoint's hostname.
- `Config.managed: ManagedConfig { enabled: bool }` — new top-level
switch. Defaults to standalone so existing configs keep working.
- `AdminConfig` now implements `Default` so managed-mode configs
can omit the `admin:` block entirely.
- `validate()` relaxes the `admin.addr` + `admin.admin_keys`
invariants when `managed.enabled = true`, and keeps them as-is
otherwise — no silent regression for standalone setups.
### `aisix-server::main`
- New `build_etcd_connect_options(&EtcdConfig) -> Option<ConnectOptions>`
helper. Returns `None` for plain HTTP (keep the test path cheap),
wires `with_user` + `with_tls` when present, surfaces missing
cert-file errors with the config key name in the message.
- `default_domain_from_endpoint()` extracts the SNI from URL-like
endpoint strings (`http://host:port`, `https://host:port`, bare
`host:port`, and IPv6 literals with brackets).
- The `EtcdConfigProvider::connect` + the separate admin `Client`
now share the same options (user + mTLS).
- **Admin listener is conditional.** In managed mode the admin
surface is never built:
* `admin_client` stays `None` (no second etcd connection)
* `admin_state` / `admin_router` are not constructed
* The admin TCP listener is not bound
* The Playground endpoint (mounted inside admin) vanishes
The proxy listener keeps running with the same request path.
- `run()` awaits the admin task via an `Option<JoinHandle>` so a
managed-mode start-up no longer joins on a nonexistent future.
### `config.example.yaml`
- Commented-out `etcd.tls` block with the three PEM paths.
- Commented-out `managed.enabled: true` section with a short
explanation of what flips in that mode.
## Tests
### `aisix-core`
- `managed_mode_lets_admin_fields_be_omitted`: minimum aisix.cloud
tenant YAML loads without an `admin:` block.
- `standalone_still_requires_admin_keys_even_with_managed_false`:
original invariant preserved for non-managed configs.
- `parses_etcd_tls_block`: round-trip for all four TLS fields.
### `aisix-server`
- `default_domain_strips_scheme_port_and_brackets`: table for the
SNI extractor including IPv6 brackets and bare-host cases.
- `build_connect_options_none_when_plain_http`: plain HTTP etcd
doesn't synthesise options (hot path).
- `build_connect_options_surfaces_missing_cert_files`: operator
sees *which* file is missing without grepping filesystem state.
`cargo fmt` / `cargo clippy --workspace --all-targets -- -D warnings`
clean. `cargo test --workspace --all-features` green (407 tests).
## Explicitly out of scope (follow-up PRs)
- DP registration flow (`POST /dp/register` against cp-api) that
*fetches* the mTLS bundle and persists it. This PR expects the
bundle already on disk — integration test path.
- Heartbeat (`POST /api/ai_dataplane/heartbeat` every 15s).
- Local config snapshot so the DP serves from cache when the etcd
connection dies mid-flight.
- Structured request logs + per-request hashes (prd-09 §9.6.2).
CopilotAI review requested due to automatic review settings April 23, 2026 07:58

CopilotAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds dataplane support for aisix.cloud tenants by introducing etcd mTLS client wiring and a managed-mode switch that disables the standalone admin surface.

Changes:

  • Add etcd.tls config (CA/cert/key + optional domain_name) and build etcd-clientConnectOptions with mTLS.
  • Add managed.enabled config and skip admin etcd client + admin router/listener when managed mode is on.
  • Update example config and add tests for domain derivation, connect options behavior, and managed/standalone config validation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

FileDescription
crates/aisix-server/src/main.rsBuild etcd ConnectOptions (mTLS/auth) and gate admin listener/router creation behind managed mode.
crates/aisix-core/src/lib.rsRe-export new config types (EtcdTlsConfig, ManagedConfig).
crates/aisix-core/src/config.rsExtend config schema with etcd.tls + managed, default admin for managed configs, and update validation rules.
config.example.yamlDocument new etcd.tls block and managed.enabled toggle.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +236 to +241
if let (Some(user), Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()) {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_etcd_connect_options() only applies basic auth if bothetcd.user and etcd.password_env are set. If a user sets only one of these fields (common config mistake), it will silently skip auth and likely fail with an opaque etcd permission error. Prefer returning a config error (or add a Config::validate() check) when exactly one of the two is set.

Suggested change
iflet(Some(user),Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()){
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
match(etcd.user.as_ref(), etcd.password_env.as_ref()){
(Some(user),Some(env_key)) => {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
}
(None,None) => {}
(Some(_),None) => {
returnErr(anyhow::anyhow!(
"etcd.user is set but etcd.password_env is missing; set both fields to enable etcd basic auth"
));
}
(None,Some(_)) => {
returnErr(anyhow::anyhow!(
"etcd.password_env is set but etcd.user is missing; set both fields to enable etcd basic auth"
));
}

Copilot uses AI. Check for mistakes.
fn build_etcd_connect_options(etcd: &EtcdConfig) -> anyhow::Result<Option<ConnectOptions>> {
let mut needs_options = false;
let mut options = ConnectOptions::new();

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EtcdConfig exposes dial_timeout_ms and request_timeout_ms, but build_etcd_connect_options() never uses them and there are no other call sites wiring these into etcd-client. This makes the timeout knobs in config.example.yaml ineffective. Consider setting the appropriate timeouts on ConnectOptions (and marking needs_options = true accordingly) so the config fields actually take effect.

Suggested change
ifletSome(dial_timeout_ms) = etcd.dial_timeout_ms{
options = options.with_connect_timeout(std::time::Duration::from_millis(dial_timeout_ms));
needs_options = true;
}
ifletSome(request_timeout_ms) = etcd.request_timeout_ms{
options = options.with_timeout(std::time::Duration::from_millis(request_timeout_ms));
needs_options = true;
}

Copilot uses AI. Check for mistakes.
Comment on lines +226 to +227
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc comment says cert/key I/O failures “bubble up as a … BootstrapError”, but this helper returns anyhow::Result and the produced errors are anyhow::Error strings. Consider adjusting the wording to avoid implying a specific error type here.

Suggested change
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.
/// failure bubbles up as a nicelycontextualised error at the same
/// point as other etcd connection errors.

Copilot uses AI. Check for mistakes.
Comment on lines +389 to +393
if self.admin.addr.parse::<std::net::SocketAddr>().is_err() {
return Err(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AdminConfig::default_addr() sets 127.0.0.1:0, and Config::validate() only checks that admin.addr parses (in standalone mode). This allows a standalone config that omits admin.addr (but sets admin_keys) to bind the admin listener on an ephemeral port, which contradicts the comment and is operationally surprising. Consider either (a) defaulting to the previous explicit port, or (b) rejecting port 0 / the default value in validate() when managed.enabled is false with a clear error message.

Suggested change
ifself.admin.addr.parse::<std::net::SocketAddr>().is_err(){
returnErr(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));
let admin_addr = self
.admin
.addr
.parse::<std::net::SocketAddr>()
.map_err(|_| {
BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
))
})?;
if admin_addr.port() == 0{
returnErr(BootstrapError::Config(
"admin.addr must use an explicit non-zero port \
(required when managed.enabled is false)"
.into(),
));

Copilot uses AI. Check for mistakes.
moonming added a commit that referenced this pull request Apr 23, 2026
Free-tier Actions storage is 500 MB, shared across the whole repo.
Each \`aisix-bin\` artifact is ~72 MB and we publish one per CI run,
so storage saturates after <10 main-branch pushes and blocks
\`actions/upload-artifact\` on every subsequent PR (the failure that
paused #28 twice today).
Retention tightened per artifact by expected re-read horizon:
- aisix-bin 1 day (consumed by the same-day e2e job only)
- ui-dist 7 days (consumed by same-day e2e; light enough
to keep a week for manual inspection)
- coverage-* 7 days (manual download for debugging flaky
coverage gates; LCOV is tiny)
Nothing downstream relies on week+ old binaries — \`build-bin\` is
re-runnable from source and the \`needs:\` chain on \`build-aisix
(instrumented)\` → \`e2e\` already re-produces the artifact when an
earlier run has expired.
No behaviour change on passing runs; only limits how long stale
runs squat on quota.
@moonming
moonming merged commit 56d2d54 into mainApr 23, 2026
13 of 17 checks passed
@moonming
moonming deleted the feat/managed-mode-mtls branch April 23, 2026 08:57
jarvis9443 added a commit that referenced this pull request Jun 2, 2026
Output guardrails only inspected message.content, so client-visible
output that lives elsewhere bypassed content/DLP checks:
- tool_calls / Anthropic tool_use (normalized into message.extra) are
now folded into a single guardrail-inspected text view via
ChatResponse::guardrail_output_text(), used by the keyword, text-
moderation, Bedrock, and Prompt Shield output checks (#3/#18/#21).
Reasoning/thinking content is intentionally left out of scope.
- Non-streaming cache hits now run the resolved output guardrail chain
before returning the stored body, instead of replaying it unchecked
(#28). Streaming output guardrails already run end-of-stream.
Part of #448 (findings #3, #18, #21, #28)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@moonming
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground) - #28

Merged
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls
Apr 23, 2026
Merged

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground)#28
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls

Conversation

@moonming

Copy link
Copy Markdown
Member

Summary

Adds two closely-related DP-side capabilities required for AISIX to act as an aisix.cloud tenant:

  1. etcd mTLS client — the aisix.cloud DP Manager serves etcd v3 over mTLS, so the DP needs to read a CA + client cert + client key from disk and wire them into the existing etcd-clientConnectOptions.
  2. Managed-mode switch — when enabled, the admin API listener, admin UI, and Playground endpoint are not bound. All configuration flows from etcd; resource mutations happen through the cloud control plane.

This is the first of ~three DP-side PRs. Registration (fetching the mTLS bundle from cp-api at boot), heartbeat, and local snapshot fallback land in follow-ups.

Config shape

etcd:
endpoints: ["https://etcd.aisix.cloud:2379"]tls:
ca_cert_file: "/etc/aisix/mtls/ca.crt"client_cert_file: "/etc/aisix/mtls/client.crt"client_key_file: "/etc/aisix/mtls/client.key"# domain_name: "etcd.aisix.cloud" # optional; defaults to endpoints[0] hostproxy:
addr: "0.0.0.0:3000"managed:
enabled: true

Standalone users are unaffected: new fields are all optional and defaulted, validate() keeps the existing invariants when managed.enabled = false.

What the managed switch actually flips

ComponentStandaloneManaged
Proxy listener✅ bound✅ bound
etcd provider + watch supervisor
Admin etcd client (write path)❌ skipped
Admin router (/admin/v1/*, UI, Playground)✅ bound❌ never constructed
Admin TCP listener✅ bound❌ never bound
/metrics (currently mounted on admin)✅ served❌ not served

The last row is worth calling out: Phase 1 mounts Prometheus on the admin listener, so managed mode loses it. Adding a dedicated observability listener is a follow-up; for aisix.cloud tenants, Prometheus scraping is expected to happen on the cloud control plane side anyway.

New helper

build_etcd_connect_options(&EtcdConfig) -> anyhow::Result<Option<ConnectOptions>>:

  • Returns None for plain HTTP etcd (keeps the hot path cheap — no pointless ConnectOptions::new() allocation when nothing needs wiring).
  • Reads the three PEM files and constructs TlsOptions::new().domain_name(...).ca_certificate(...).identity(...).
  • Missing files surface with the config key name (etcd.tls.ca_cert_file = "/...") in the error so operators don't have to diff config against filesystem state.

default_domain_from_endpoint() extracts the SNI from URL-ish strings: http://, https://, bare host:port, and IPv6 literals [::1]:2379 — table-tested.

Tests

cargo test --workspace --all-features green (407 tests). New cases:

  • aisix-core::config::tests::managed_mode_lets_admin_fields_be_omitted — minimum aisix.cloud tenant YAML loads cleanly.
  • aisix-core::config::tests::standalone_still_requires_admin_keys_even_with_managed_false — original invariant preserved.
  • aisix-core::config::tests::parses_etcd_tls_block — round-trip for all four TLS fields.
  • aisix-server::tests::default_domain_strips_scheme_port_and_brackets — SNI extractor table.
  • aisix-server::tests::build_connect_options_none_when_plain_http — hot path stays zero-cost.
  • aisix-server::tests::build_connect_options_surfaces_missing_cert_files — operator-friendly error.

cargo fmt + cargo clippy --workspace --all-targets -- -D warnings clean.

Explicitly out of scope (follow-up PRs on this branch's successor stack)

  • Registration flow: the DP currently expects the mTLS bundle already on disk. The next PR adds POST /dp/register client that exchanges a one-time Deployment Token for the bundle at boot and persists it atomically.
  • Heartbeat: periodic POST /api/ai_dataplane/heartbeat so cp-api knows the DP is alive (and so the Gateway page in aisix.cloud shows green dots).
  • Local config snapshot: continue serving proxy traffic when the etcd watch disconnects mid-flight (see aisix.cloud PRD prd-09 §9.7.2).
  • Structured request logs + hashes: the observability schema aisix.cloud telemetry consumes.

Relationship

Paired with the aisix.cloud side in api7/AISIX-Cloud#8: that PR's GatewayHandlers.create calls IssueDataplaneCertificate on api7ee CP and returns the bundle to the user, who then drops it on their DP machine. The DP then boots with this PR's new config block pointing at those files.

…und)
Required for AISIX data planes to talk to the aisix.cloud control
plane: the CP's DP Manager serves etcd v3 over mTLS (see the
aisix.cloud PRD prd-09 §9.3.3). Phase 1 of the DP-side changes —
follow-up PRs wire registration + heartbeat + local snapshot.
## What's new
### `aisix-core::config`
- `EtcdConfig.tls: Option<EtcdTlsConfig>` — new optional mTLS bundle.
Three PEM file paths (CA cert, client cert, client key) plus an
optional `domain_name` for SNI. Defaults derive the domain from the
first endpoint's hostname.
- `Config.managed: ManagedConfig { enabled: bool }` — new top-level
switch. Defaults to standalone so existing configs keep working.
- `AdminConfig` now implements `Default` so managed-mode configs
can omit the `admin:` block entirely.
- `validate()` relaxes the `admin.addr` + `admin.admin_keys`
invariants when `managed.enabled = true`, and keeps them as-is
otherwise — no silent regression for standalone setups.
### `aisix-server::main`
- New `build_etcd_connect_options(&EtcdConfig) -> Option<ConnectOptions>`
helper. Returns `None` for plain HTTP (keep the test path cheap),
wires `with_user` + `with_tls` when present, surfaces missing
cert-file errors with the config key name in the message.
- `default_domain_from_endpoint()` extracts the SNI from URL-like
endpoint strings (`http://host:port`, `https://host:port`, bare
`host:port`, and IPv6 literals with brackets).
- The `EtcdConfigProvider::connect` + the separate admin `Client`
now share the same options (user + mTLS).
- **Admin listener is conditional.** In managed mode the admin
surface is never built:
* `admin_client` stays `None` (no second etcd connection)
* `admin_state` / `admin_router` are not constructed
* The admin TCP listener is not bound
* The Playground endpoint (mounted inside admin) vanishes
The proxy listener keeps running with the same request path.
- `run()` awaits the admin task via an `Option<JoinHandle>` so a
managed-mode start-up no longer joins on a nonexistent future.
### `config.example.yaml`
- Commented-out `etcd.tls` block with the three PEM paths.
- Commented-out `managed.enabled: true` section with a short
explanation of what flips in that mode.
## Tests
### `aisix-core`
- `managed_mode_lets_admin_fields_be_omitted`: minimum aisix.cloud
tenant YAML loads without an `admin:` block.
- `standalone_still_requires_admin_keys_even_with_managed_false`:
original invariant preserved for non-managed configs.
- `parses_etcd_tls_block`: round-trip for all four TLS fields.
### `aisix-server`
- `default_domain_strips_scheme_port_and_brackets`: table for the
SNI extractor including IPv6 brackets and bare-host cases.
- `build_connect_options_none_when_plain_http`: plain HTTP etcd
doesn't synthesise options (hot path).
- `build_connect_options_surfaces_missing_cert_files`: operator
sees *which* file is missing without grepping filesystem state.
`cargo fmt` / `cargo clippy --workspace --all-targets -- -D warnings`
clean. `cargo test --workspace --all-features` green (407 tests).
## Explicitly out of scope (follow-up PRs)
- DP registration flow (`POST /dp/register` against cp-api) that
*fetches* the mTLS bundle and persists it. This PR expects the
bundle already on disk — integration test path.
- Heartbeat (`POST /api/ai_dataplane/heartbeat` every 15s).
- Local config snapshot so the DP serves from cache when the etcd
connection dies mid-flight.
- Structured request logs + per-request hashes (prd-09 §9.6.2).
CopilotAI review requested due to automatic review settings April 23, 2026 07:58

CopilotAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds dataplane support for aisix.cloud tenants by introducing etcd mTLS client wiring and a managed-mode switch that disables the standalone admin surface.

Changes:

  • Add etcd.tls config (CA/cert/key + optional domain_name) and build etcd-clientConnectOptions with mTLS.
  • Add managed.enabled config and skip admin etcd client + admin router/listener when managed mode is on.
  • Update example config and add tests for domain derivation, connect options behavior, and managed/standalone config validation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

FileDescription
crates/aisix-server/src/main.rsBuild etcd ConnectOptions (mTLS/auth) and gate admin listener/router creation behind managed mode.
crates/aisix-core/src/lib.rsRe-export new config types (EtcdTlsConfig, ManagedConfig).
crates/aisix-core/src/config.rsExtend config schema with etcd.tls + managed, default admin for managed configs, and update validation rules.
config.example.yamlDocument new etcd.tls block and managed.enabled toggle.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +236 to +241
if let (Some(user), Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()) {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_etcd_connect_options() only applies basic auth if bothetcd.user and etcd.password_env are set. If a user sets only one of these fields (common config mistake), it will silently skip auth and likely fail with an opaque etcd permission error. Prefer returning a config error (or add a Config::validate() check) when exactly one of the two is set.

Suggested change
iflet(Some(user),Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()){
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
match(etcd.user.as_ref(), etcd.password_env.as_ref()){
(Some(user),Some(env_key)) => {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
}
(None,None) => {}
(Some(_),None) => {
returnErr(anyhow::anyhow!(
"etcd.user is set but etcd.password_env is missing; set both fields to enable etcd basic auth"
));
}
(None,Some(_)) => {
returnErr(anyhow::anyhow!(
"etcd.password_env is set but etcd.user is missing; set both fields to enable etcd basic auth"
));
}

Copilot uses AI. Check for mistakes.
fn build_etcd_connect_options(etcd: &EtcdConfig) -> anyhow::Result<Option<ConnectOptions>> {
let mut needs_options = false;
let mut options = ConnectOptions::new();

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EtcdConfig exposes dial_timeout_ms and request_timeout_ms, but build_etcd_connect_options() never uses them and there are no other call sites wiring these into etcd-client. This makes the timeout knobs in config.example.yaml ineffective. Consider setting the appropriate timeouts on ConnectOptions (and marking needs_options = true accordingly) so the config fields actually take effect.

Suggested change
ifletSome(dial_timeout_ms) = etcd.dial_timeout_ms{
options = options.with_connect_timeout(std::time::Duration::from_millis(dial_timeout_ms));
needs_options = true;
}
ifletSome(request_timeout_ms) = etcd.request_timeout_ms{
options = options.with_timeout(std::time::Duration::from_millis(request_timeout_ms));
needs_options = true;
}

Copilot uses AI. Check for mistakes.
Comment on lines +226 to +227
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc comment says cert/key I/O failures “bubble up as a … BootstrapError”, but this helper returns anyhow::Result and the produced errors are anyhow::Error strings. Consider adjusting the wording to avoid implying a specific error type here.

Suggested change
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.
/// failure bubbles up as a nicelycontextualised error at the same
/// point as other etcd connection errors.

Copilot uses AI. Check for mistakes.
Comment on lines +389 to +393
if self.admin.addr.parse::<std::net::SocketAddr>().is_err() {
return Err(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AdminConfig::default_addr() sets 127.0.0.1:0, and Config::validate() only checks that admin.addr parses (in standalone mode). This allows a standalone config that omits admin.addr (but sets admin_keys) to bind the admin listener on an ephemeral port, which contradicts the comment and is operationally surprising. Consider either (a) defaulting to the previous explicit port, or (b) rejecting port 0 / the default value in validate() when managed.enabled is false with a clear error message.

Suggested change
ifself.admin.addr.parse::<std::net::SocketAddr>().is_err(){
returnErr(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));
let admin_addr = self
.admin
.addr
.parse::<std::net::SocketAddr>()
.map_err(|_| {
BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
))
})?;
if admin_addr.port() == 0{
returnErr(BootstrapError::Config(
"admin.addr must use an explicit non-zero port \
(required when managed.enabled is false)"
.into(),
));

Copilot uses AI. Check for mistakes.
moonming added a commit that referenced this pull request Apr 23, 2026
Free-tier Actions storage is 500 MB, shared across the whole repo.
Each \`aisix-bin\` artifact is ~72 MB and we publish one per CI run,
so storage saturates after <10 main-branch pushes and blocks
\`actions/upload-artifact\` on every subsequent PR (the failure that
paused #28 twice today).
Retention tightened per artifact by expected re-read horizon:
- aisix-bin 1 day (consumed by the same-day e2e job only)
- ui-dist 7 days (consumed by same-day e2e; light enough
to keep a week for manual inspection)
- coverage-* 7 days (manual download for debugging flaky
coverage gates; LCOV is tiny)
Nothing downstream relies on week+ old binaries — \`build-bin\` is
re-runnable from source and the \`needs:\` chain on \`build-aisix
(instrumented)\` → \`e2e\` already re-produces the artifact when an
earlier run has expired.
No behaviour change on passing runs; only limits how long stale
runs squat on quota.
@moonming
moonming merged commit 56d2d54 into mainApr 23, 2026
13 of 17 checks passed
@moonming
moonming deleted the feat/managed-mode-mtls branch April 23, 2026 08:57
jarvis9443 added a commit that referenced this pull request Jun 2, 2026
Output guardrails only inspected message.content, so client-visible
output that lives elsewhere bypassed content/DLP checks:
- tool_calls / Anthropic tool_use (normalized into message.extra) are
now folded into a single guardrail-inspected text view via
ChatResponse::guardrail_output_text(), used by the keyword, text-
moderation, Bedrock, and Prompt Shield output checks (#3/#18/#21).
Reasoning/thinking content is intentionally left out of scope.
- Non-streaming cache hits now run the resolved output guardrail chain
before returning the stored body, instead of replaying it unchecked
(#28). Streaming output guardrails already run end-of-stream.
Part of #448 (findings #3, #18, #21, #28)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@moonming
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground) - #28

Merged
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls
Apr 23, 2026
Merged

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground)#28
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls

Conversation

@moonming

Copy link
Copy Markdown
Member

Summary

Adds two closely-related DP-side capabilities required for AISIX to act as an aisix.cloud tenant:

  1. etcd mTLS client — the aisix.cloud DP Manager serves etcd v3 over mTLS, so the DP needs to read a CA + client cert + client key from disk and wire them into the existing etcd-clientConnectOptions.
  2. Managed-mode switch — when enabled, the admin API listener, admin UI, and Playground endpoint are not bound. All configuration flows from etcd; resource mutations happen through the cloud control plane.

This is the first of ~three DP-side PRs. Registration (fetching the mTLS bundle from cp-api at boot), heartbeat, and local snapshot fallback land in follow-ups.

Config shape

etcd:
endpoints: ["https://etcd.aisix.cloud:2379"]tls:
ca_cert_file: "/etc/aisix/mtls/ca.crt"client_cert_file: "/etc/aisix/mtls/client.crt"client_key_file: "/etc/aisix/mtls/client.key"# domain_name: "etcd.aisix.cloud" # optional; defaults to endpoints[0] hostproxy:
addr: "0.0.0.0:3000"managed:
enabled: true

Standalone users are unaffected: new fields are all optional and defaulted, validate() keeps the existing invariants when managed.enabled = false.

What the managed switch actually flips

ComponentStandaloneManaged
Proxy listener✅ bound✅ bound
etcd provider + watch supervisor
Admin etcd client (write path)❌ skipped
Admin router (/admin/v1/*, UI, Playground)✅ bound❌ never constructed
Admin TCP listener✅ bound❌ never bound
/metrics (currently mounted on admin)✅ served❌ not served

The last row is worth calling out: Phase 1 mounts Prometheus on the admin listener, so managed mode loses it. Adding a dedicated observability listener is a follow-up; for aisix.cloud tenants, Prometheus scraping is expected to happen on the cloud control plane side anyway.

New helper

build_etcd_connect_options(&EtcdConfig) -> anyhow::Result<Option<ConnectOptions>>:

  • Returns None for plain HTTP etcd (keeps the hot path cheap — no pointless ConnectOptions::new() allocation when nothing needs wiring).
  • Reads the three PEM files and constructs TlsOptions::new().domain_name(...).ca_certificate(...).identity(...).
  • Missing files surface with the config key name (etcd.tls.ca_cert_file = "/...") in the error so operators don't have to diff config against filesystem state.

default_domain_from_endpoint() extracts the SNI from URL-ish strings: http://, https://, bare host:port, and IPv6 literals [::1]:2379 — table-tested.

Tests

cargo test --workspace --all-features green (407 tests). New cases:

  • aisix-core::config::tests::managed_mode_lets_admin_fields_be_omitted — minimum aisix.cloud tenant YAML loads cleanly.
  • aisix-core::config::tests::standalone_still_requires_admin_keys_even_with_managed_false — original invariant preserved.
  • aisix-core::config::tests::parses_etcd_tls_block — round-trip for all four TLS fields.
  • aisix-server::tests::default_domain_strips_scheme_port_and_brackets — SNI extractor table.
  • aisix-server::tests::build_connect_options_none_when_plain_http — hot path stays zero-cost.
  • aisix-server::tests::build_connect_options_surfaces_missing_cert_files — operator-friendly error.

cargo fmt + cargo clippy --workspace --all-targets -- -D warnings clean.

Explicitly out of scope (follow-up PRs on this branch's successor stack)

  • Registration flow: the DP currently expects the mTLS bundle already on disk. The next PR adds POST /dp/register client that exchanges a one-time Deployment Token for the bundle at boot and persists it atomically.
  • Heartbeat: periodic POST /api/ai_dataplane/heartbeat so cp-api knows the DP is alive (and so the Gateway page in aisix.cloud shows green dots).
  • Local config snapshot: continue serving proxy traffic when the etcd watch disconnects mid-flight (see aisix.cloud PRD prd-09 §9.7.2).
  • Structured request logs + hashes: the observability schema aisix.cloud telemetry consumes.

Relationship

Paired with the aisix.cloud side in api7/AISIX-Cloud#8: that PR's GatewayHandlers.create calls IssueDataplaneCertificate on api7ee CP and returns the bundle to the user, who then drops it on their DP machine. The DP then boots with this PR's new config block pointing at those files.

…und)
Required for AISIX data planes to talk to the aisix.cloud control
plane: the CP's DP Manager serves etcd v3 over mTLS (see the
aisix.cloud PRD prd-09 §9.3.3). Phase 1 of the DP-side changes —
follow-up PRs wire registration + heartbeat + local snapshot.
## What's new
### `aisix-core::config`
- `EtcdConfig.tls: Option<EtcdTlsConfig>` — new optional mTLS bundle.
Three PEM file paths (CA cert, client cert, client key) plus an
optional `domain_name` for SNI. Defaults derive the domain from the
first endpoint's hostname.
- `Config.managed: ManagedConfig { enabled: bool }` — new top-level
switch. Defaults to standalone so existing configs keep working.
- `AdminConfig` now implements `Default` so managed-mode configs
can omit the `admin:` block entirely.
- `validate()` relaxes the `admin.addr` + `admin.admin_keys`
invariants when `managed.enabled = true`, and keeps them as-is
otherwise — no silent regression for standalone setups.
### `aisix-server::main`
- New `build_etcd_connect_options(&EtcdConfig) -> Option<ConnectOptions>`
helper. Returns `None` for plain HTTP (keep the test path cheap),
wires `with_user` + `with_tls` when present, surfaces missing
cert-file errors with the config key name in the message.
- `default_domain_from_endpoint()` extracts the SNI from URL-like
endpoint strings (`http://host:port`, `https://host:port`, bare
`host:port`, and IPv6 literals with brackets).
- The `EtcdConfigProvider::connect` + the separate admin `Client`
now share the same options (user + mTLS).
- **Admin listener is conditional.** In managed mode the admin
surface is never built:
* `admin_client` stays `None` (no second etcd connection)
* `admin_state` / `admin_router` are not constructed
* The admin TCP listener is not bound
* The Playground endpoint (mounted inside admin) vanishes
The proxy listener keeps running with the same request path.
- `run()` awaits the admin task via an `Option<JoinHandle>` so a
managed-mode start-up no longer joins on a nonexistent future.
### `config.example.yaml`
- Commented-out `etcd.tls` block with the three PEM paths.
- Commented-out `managed.enabled: true` section with a short
explanation of what flips in that mode.
## Tests
### `aisix-core`
- `managed_mode_lets_admin_fields_be_omitted`: minimum aisix.cloud
tenant YAML loads without an `admin:` block.
- `standalone_still_requires_admin_keys_even_with_managed_false`:
original invariant preserved for non-managed configs.
- `parses_etcd_tls_block`: round-trip for all four TLS fields.
### `aisix-server`
- `default_domain_strips_scheme_port_and_brackets`: table for the
SNI extractor including IPv6 brackets and bare-host cases.
- `build_connect_options_none_when_plain_http`: plain HTTP etcd
doesn't synthesise options (hot path).
- `build_connect_options_surfaces_missing_cert_files`: operator
sees *which* file is missing without grepping filesystem state.
`cargo fmt` / `cargo clippy --workspace --all-targets -- -D warnings`
clean. `cargo test --workspace --all-features` green (407 tests).
## Explicitly out of scope (follow-up PRs)
- DP registration flow (`POST /dp/register` against cp-api) that
*fetches* the mTLS bundle and persists it. This PR expects the
bundle already on disk — integration test path.
- Heartbeat (`POST /api/ai_dataplane/heartbeat` every 15s).
- Local config snapshot so the DP serves from cache when the etcd
connection dies mid-flight.
- Structured request logs + per-request hashes (prd-09 §9.6.2).
CopilotAI review requested due to automatic review settings April 23, 2026 07:58

CopilotAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds dataplane support for aisix.cloud tenants by introducing etcd mTLS client wiring and a managed-mode switch that disables the standalone admin surface.

Changes:

  • Add etcd.tls config (CA/cert/key + optional domain_name) and build etcd-clientConnectOptions with mTLS.
  • Add managed.enabled config and skip admin etcd client + admin router/listener when managed mode is on.
  • Update example config and add tests for domain derivation, connect options behavior, and managed/standalone config validation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

FileDescription
crates/aisix-server/src/main.rsBuild etcd ConnectOptions (mTLS/auth) and gate admin listener/router creation behind managed mode.
crates/aisix-core/src/lib.rsRe-export new config types (EtcdTlsConfig, ManagedConfig).
crates/aisix-core/src/config.rsExtend config schema with etcd.tls + managed, default admin for managed configs, and update validation rules.
config.example.yamlDocument new etcd.tls block and managed.enabled toggle.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +236 to +241
if let (Some(user), Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()) {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_etcd_connect_options() only applies basic auth if bothetcd.user and etcd.password_env are set. If a user sets only one of these fields (common config mistake), it will silently skip auth and likely fail with an opaque etcd permission error. Prefer returning a config error (or add a Config::validate() check) when exactly one of the two is set.

Suggested change
iflet(Some(user),Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()){
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
match(etcd.user.as_ref(), etcd.password_env.as_ref()){
(Some(user),Some(env_key)) => {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
}
(None,None) => {}
(Some(_),None) => {
returnErr(anyhow::anyhow!(
"etcd.user is set but etcd.password_env is missing; set both fields to enable etcd basic auth"
));
}
(None,Some(_)) => {
returnErr(anyhow::anyhow!(
"etcd.password_env is set but etcd.user is missing; set both fields to enable etcd basic auth"
));
}

Copilot uses AI. Check for mistakes.
fn build_etcd_connect_options(etcd: &EtcdConfig) -> anyhow::Result<Option<ConnectOptions>> {
let mut needs_options = false;
let mut options = ConnectOptions::new();

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EtcdConfig exposes dial_timeout_ms and request_timeout_ms, but build_etcd_connect_options() never uses them and there are no other call sites wiring these into etcd-client. This makes the timeout knobs in config.example.yaml ineffective. Consider setting the appropriate timeouts on ConnectOptions (and marking needs_options = true accordingly) so the config fields actually take effect.

Suggested change
ifletSome(dial_timeout_ms) = etcd.dial_timeout_ms{
options = options.with_connect_timeout(std::time::Duration::from_millis(dial_timeout_ms));
needs_options = true;
}
ifletSome(request_timeout_ms) = etcd.request_timeout_ms{
options = options.with_timeout(std::time::Duration::from_millis(request_timeout_ms));
needs_options = true;
}

Copilot uses AI. Check for mistakes.
Comment on lines +226 to +227
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc comment says cert/key I/O failures “bubble up as a … BootstrapError”, but this helper returns anyhow::Result and the produced errors are anyhow::Error strings. Consider adjusting the wording to avoid implying a specific error type here.

Suggested change
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.
/// failure bubbles up as a nicelycontextualised error at the same
/// point as other etcd connection errors.

Copilot uses AI. Check for mistakes.
Comment on lines +389 to +393
if self.admin.addr.parse::<std::net::SocketAddr>().is_err() {
return Err(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AdminConfig::default_addr() sets 127.0.0.1:0, and Config::validate() only checks that admin.addr parses (in standalone mode). This allows a standalone config that omits admin.addr (but sets admin_keys) to bind the admin listener on an ephemeral port, which contradicts the comment and is operationally surprising. Consider either (a) defaulting to the previous explicit port, or (b) rejecting port 0 / the default value in validate() when managed.enabled is false with a clear error message.

Suggested change
ifself.admin.addr.parse::<std::net::SocketAddr>().is_err(){
returnErr(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));
let admin_addr = self
.admin
.addr
.parse::<std::net::SocketAddr>()
.map_err(|_| {
BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
))
})?;
if admin_addr.port() == 0{
returnErr(BootstrapError::Config(
"admin.addr must use an explicit non-zero port \
(required when managed.enabled is false)"
.into(),
));

Copilot uses AI. Check for mistakes.
moonming added a commit that referenced this pull request Apr 23, 2026
Free-tier Actions storage is 500 MB, shared across the whole repo.
Each \`aisix-bin\` artifact is ~72 MB and we publish one per CI run,
so storage saturates after <10 main-branch pushes and blocks
\`actions/upload-artifact\` on every subsequent PR (the failure that
paused #28 twice today).
Retention tightened per artifact by expected re-read horizon:
- aisix-bin 1 day (consumed by the same-day e2e job only)
- ui-dist 7 days (consumed by same-day e2e; light enough
to keep a week for manual inspection)
- coverage-* 7 days (manual download for debugging flaky
coverage gates; LCOV is tiny)
Nothing downstream relies on week+ old binaries — \`build-bin\` is
re-runnable from source and the \`needs:\` chain on \`build-aisix
(instrumented)\` → \`e2e\` already re-produces the artifact when an
earlier run has expired.
No behaviour change on passing runs; only limits how long stale
runs squat on quota.
@moonming
moonming merged commit 56d2d54 into mainApr 23, 2026
13 of 17 checks passed
@moonming
moonming deleted the feat/managed-mode-mtls branch April 23, 2026 08:57
jarvis9443 added a commit that referenced this pull request Jun 2, 2026
Output guardrails only inspected message.content, so client-visible
output that lives elsewhere bypassed content/DLP checks:
- tool_calls / Anthropic tool_use (normalized into message.extra) are
now folded into a single guardrail-inspected text view via
ChatResponse::guardrail_output_text(), used by the keyword, text-
moderation, Bedrock, and Prompt Shield output checks (#3/#18/#21).
Reasoning/thinking content is intentionally left out of scope.
- Non-streaming cache hits now run the resolved output guardrail chain
before returning the stored body, instead of replaying it unchecked
(#28). Streaming output guardrails already run end-of-stream.
Part of #448 (findings #3, #18, #21, #28)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@moonming
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground) - #28

Merged
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls
Apr 23, 2026
Merged

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground)#28
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls

Conversation

@moonming

Copy link
Copy Markdown
Member

Summary

Adds two closely-related DP-side capabilities required for AISIX to act as an aisix.cloud tenant:

  1. etcd mTLS client — the aisix.cloud DP Manager serves etcd v3 over mTLS, so the DP needs to read a CA + client cert + client key from disk and wire them into the existing etcd-clientConnectOptions.
  2. Managed-mode switch — when enabled, the admin API listener, admin UI, and Playground endpoint are not bound. All configuration flows from etcd; resource mutations happen through the cloud control plane.

This is the first of ~three DP-side PRs. Registration (fetching the mTLS bundle from cp-api at boot), heartbeat, and local snapshot fallback land in follow-ups.

Config shape

etcd:
endpoints: ["https://etcd.aisix.cloud:2379"]tls:
ca_cert_file: "/etc/aisix/mtls/ca.crt"client_cert_file: "/etc/aisix/mtls/client.crt"client_key_file: "/etc/aisix/mtls/client.key"# domain_name: "etcd.aisix.cloud" # optional; defaults to endpoints[0] hostproxy:
addr: "0.0.0.0:3000"managed:
enabled: true

Standalone users are unaffected: new fields are all optional and defaulted, validate() keeps the existing invariants when managed.enabled = false.

What the managed switch actually flips

ComponentStandaloneManaged
Proxy listener✅ bound✅ bound
etcd provider + watch supervisor
Admin etcd client (write path)❌ skipped
Admin router (/admin/v1/*, UI, Playground)✅ bound❌ never constructed
Admin TCP listener✅ bound❌ never bound
/metrics (currently mounted on admin)✅ served❌ not served

The last row is worth calling out: Phase 1 mounts Prometheus on the admin listener, so managed mode loses it. Adding a dedicated observability listener is a follow-up; for aisix.cloud tenants, Prometheus scraping is expected to happen on the cloud control plane side anyway.

New helper

build_etcd_connect_options(&EtcdConfig) -> anyhow::Result<Option<ConnectOptions>>:

  • Returns None for plain HTTP etcd (keeps the hot path cheap — no pointless ConnectOptions::new() allocation when nothing needs wiring).
  • Reads the three PEM files and constructs TlsOptions::new().domain_name(...).ca_certificate(...).identity(...).
  • Missing files surface with the config key name (etcd.tls.ca_cert_file = "/...") in the error so operators don't have to diff config against filesystem state.

default_domain_from_endpoint() extracts the SNI from URL-ish strings: http://, https://, bare host:port, and IPv6 literals [::1]:2379 — table-tested.

Tests

cargo test --workspace --all-features green (407 tests). New cases:

  • aisix-core::config::tests::managed_mode_lets_admin_fields_be_omitted — minimum aisix.cloud tenant YAML loads cleanly.
  • aisix-core::config::tests::standalone_still_requires_admin_keys_even_with_managed_false — original invariant preserved.
  • aisix-core::config::tests::parses_etcd_tls_block — round-trip for all four TLS fields.
  • aisix-server::tests::default_domain_strips_scheme_port_and_brackets — SNI extractor table.
  • aisix-server::tests::build_connect_options_none_when_plain_http — hot path stays zero-cost.
  • aisix-server::tests::build_connect_options_surfaces_missing_cert_files — operator-friendly error.

cargo fmt + cargo clippy --workspace --all-targets -- -D warnings clean.

Explicitly out of scope (follow-up PRs on this branch's successor stack)

  • Registration flow: the DP currently expects the mTLS bundle already on disk. The next PR adds POST /dp/register client that exchanges a one-time Deployment Token for the bundle at boot and persists it atomically.
  • Heartbeat: periodic POST /api/ai_dataplane/heartbeat so cp-api knows the DP is alive (and so the Gateway page in aisix.cloud shows green dots).
  • Local config snapshot: continue serving proxy traffic when the etcd watch disconnects mid-flight (see aisix.cloud PRD prd-09 §9.7.2).
  • Structured request logs + hashes: the observability schema aisix.cloud telemetry consumes.

Relationship

Paired with the aisix.cloud side in api7/AISIX-Cloud#8: that PR's GatewayHandlers.create calls IssueDataplaneCertificate on api7ee CP and returns the bundle to the user, who then drops it on their DP machine. The DP then boots with this PR's new config block pointing at those files.

…und)
Required for AISIX data planes to talk to the aisix.cloud control
plane: the CP's DP Manager serves etcd v3 over mTLS (see the
aisix.cloud PRD prd-09 §9.3.3). Phase 1 of the DP-side changes —
follow-up PRs wire registration + heartbeat + local snapshot.
## What's new
### `aisix-core::config`
- `EtcdConfig.tls: Option<EtcdTlsConfig>` — new optional mTLS bundle.
Three PEM file paths (CA cert, client cert, client key) plus an
optional `domain_name` for SNI. Defaults derive the domain from the
first endpoint's hostname.
- `Config.managed: ManagedConfig { enabled: bool }` — new top-level
switch. Defaults to standalone so existing configs keep working.
- `AdminConfig` now implements `Default` so managed-mode configs
can omit the `admin:` block entirely.
- `validate()` relaxes the `admin.addr` + `admin.admin_keys`
invariants when `managed.enabled = true`, and keeps them as-is
otherwise — no silent regression for standalone setups.
### `aisix-server::main`
- New `build_etcd_connect_options(&EtcdConfig) -> Option<ConnectOptions>`
helper. Returns `None` for plain HTTP (keep the test path cheap),
wires `with_user` + `with_tls` when present, surfaces missing
cert-file errors with the config key name in the message.
- `default_domain_from_endpoint()` extracts the SNI from URL-like
endpoint strings (`http://host:port`, `https://host:port`, bare
`host:port`, and IPv6 literals with brackets).
- The `EtcdConfigProvider::connect` + the separate admin `Client`
now share the same options (user + mTLS).
- **Admin listener is conditional.** In managed mode the admin
surface is never built:
* `admin_client` stays `None` (no second etcd connection)
* `admin_state` / `admin_router` are not constructed
* The admin TCP listener is not bound
* The Playground endpoint (mounted inside admin) vanishes
The proxy listener keeps running with the same request path.
- `run()` awaits the admin task via an `Option<JoinHandle>` so a
managed-mode start-up no longer joins on a nonexistent future.
### `config.example.yaml`
- Commented-out `etcd.tls` block with the three PEM paths.
- Commented-out `managed.enabled: true` section with a short
explanation of what flips in that mode.
## Tests
### `aisix-core`
- `managed_mode_lets_admin_fields_be_omitted`: minimum aisix.cloud
tenant YAML loads without an `admin:` block.
- `standalone_still_requires_admin_keys_even_with_managed_false`:
original invariant preserved for non-managed configs.
- `parses_etcd_tls_block`: round-trip for all four TLS fields.
### `aisix-server`
- `default_domain_strips_scheme_port_and_brackets`: table for the
SNI extractor including IPv6 brackets and bare-host cases.
- `build_connect_options_none_when_plain_http`: plain HTTP etcd
doesn't synthesise options (hot path).
- `build_connect_options_surfaces_missing_cert_files`: operator
sees *which* file is missing without grepping filesystem state.
`cargo fmt` / `cargo clippy --workspace --all-targets -- -D warnings`
clean. `cargo test --workspace --all-features` green (407 tests).
## Explicitly out of scope (follow-up PRs)
- DP registration flow (`POST /dp/register` against cp-api) that
*fetches* the mTLS bundle and persists it. This PR expects the
bundle already on disk — integration test path.
- Heartbeat (`POST /api/ai_dataplane/heartbeat` every 15s).
- Local config snapshot so the DP serves from cache when the etcd
connection dies mid-flight.
- Structured request logs + per-request hashes (prd-09 §9.6.2).
CopilotAI review requested due to automatic review settings April 23, 2026 07:58

CopilotAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds dataplane support for aisix.cloud tenants by introducing etcd mTLS client wiring and a managed-mode switch that disables the standalone admin surface.

Changes:

  • Add etcd.tls config (CA/cert/key + optional domain_name) and build etcd-clientConnectOptions with mTLS.
  • Add managed.enabled config and skip admin etcd client + admin router/listener when managed mode is on.
  • Update example config and add tests for domain derivation, connect options behavior, and managed/standalone config validation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

FileDescription
crates/aisix-server/src/main.rsBuild etcd ConnectOptions (mTLS/auth) and gate admin listener/router creation behind managed mode.
crates/aisix-core/src/lib.rsRe-export new config types (EtcdTlsConfig, ManagedConfig).
crates/aisix-core/src/config.rsExtend config schema with etcd.tls + managed, default admin for managed configs, and update validation rules.
config.example.yamlDocument new etcd.tls block and managed.enabled toggle.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +236 to +241
if let (Some(user), Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()) {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_etcd_connect_options() only applies basic auth if bothetcd.user and etcd.password_env are set. If a user sets only one of these fields (common config mistake), it will silently skip auth and likely fail with an opaque etcd permission error. Prefer returning a config error (or add a Config::validate() check) when exactly one of the two is set.

Suggested change
iflet(Some(user),Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()){
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
match(etcd.user.as_ref(), etcd.password_env.as_ref()){
(Some(user),Some(env_key)) => {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
}
(None,None) => {}
(Some(_),None) => {
returnErr(anyhow::anyhow!(
"etcd.user is set but etcd.password_env is missing; set both fields to enable etcd basic auth"
));
}
(None,Some(_)) => {
returnErr(anyhow::anyhow!(
"etcd.password_env is set but etcd.user is missing; set both fields to enable etcd basic auth"
));
}

Copilot uses AI. Check for mistakes.
fn build_etcd_connect_options(etcd: &EtcdConfig) -> anyhow::Result<Option<ConnectOptions>> {
let mut needs_options = false;
let mut options = ConnectOptions::new();

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EtcdConfig exposes dial_timeout_ms and request_timeout_ms, but build_etcd_connect_options() never uses them and there are no other call sites wiring these into etcd-client. This makes the timeout knobs in config.example.yaml ineffective. Consider setting the appropriate timeouts on ConnectOptions (and marking needs_options = true accordingly) so the config fields actually take effect.

Suggested change
ifletSome(dial_timeout_ms) = etcd.dial_timeout_ms{
options = options.with_connect_timeout(std::time::Duration::from_millis(dial_timeout_ms));
needs_options = true;
}
ifletSome(request_timeout_ms) = etcd.request_timeout_ms{
options = options.with_timeout(std::time::Duration::from_millis(request_timeout_ms));
needs_options = true;
}

Copilot uses AI. Check for mistakes.
Comment on lines +226 to +227
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc comment says cert/key I/O failures “bubble up as a … BootstrapError”, but this helper returns anyhow::Result and the produced errors are anyhow::Error strings. Consider adjusting the wording to avoid implying a specific error type here.

Suggested change
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.
/// failure bubbles up as a nicelycontextualised error at the same
/// point as other etcd connection errors.

Copilot uses AI. Check for mistakes.
Comment on lines +389 to +393
if self.admin.addr.parse::<std::net::SocketAddr>().is_err() {
return Err(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AdminConfig::default_addr() sets 127.0.0.1:0, and Config::validate() only checks that admin.addr parses (in standalone mode). This allows a standalone config that omits admin.addr (but sets admin_keys) to bind the admin listener on an ephemeral port, which contradicts the comment and is operationally surprising. Consider either (a) defaulting to the previous explicit port, or (b) rejecting port 0 / the default value in validate() when managed.enabled is false with a clear error message.

Suggested change
ifself.admin.addr.parse::<std::net::SocketAddr>().is_err(){
returnErr(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));
let admin_addr = self
.admin
.addr
.parse::<std::net::SocketAddr>()
.map_err(|_| {
BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
))
})?;
if admin_addr.port() == 0{
returnErr(BootstrapError::Config(
"admin.addr must use an explicit non-zero port \
(required when managed.enabled is false)"
.into(),
));

Copilot uses AI. Check for mistakes.
moonming added a commit that referenced this pull request Apr 23, 2026
Free-tier Actions storage is 500 MB, shared across the whole repo.
Each \`aisix-bin\` artifact is ~72 MB and we publish one per CI run,
so storage saturates after <10 main-branch pushes and blocks
\`actions/upload-artifact\` on every subsequent PR (the failure that
paused #28 twice today).
Retention tightened per artifact by expected re-read horizon:
- aisix-bin 1 day (consumed by the same-day e2e job only)
- ui-dist 7 days (consumed by same-day e2e; light enough
to keep a week for manual inspection)
- coverage-* 7 days (manual download for debugging flaky
coverage gates; LCOV is tiny)
Nothing downstream relies on week+ old binaries — \`build-bin\` is
re-runnable from source and the \`needs:\` chain on \`build-aisix
(instrumented)\` → \`e2e\` already re-produces the artifact when an
earlier run has expired.
No behaviour change on passing runs; only limits how long stale
runs squat on quota.
@moonming
moonming merged commit 56d2d54 into mainApr 23, 2026
13 of 17 checks passed
@moonming
moonming deleted the feat/managed-mode-mtls branch April 23, 2026 08:57
jarvis9443 added a commit that referenced this pull request Jun 2, 2026
Output guardrails only inspected message.content, so client-visible
output that lives elsewhere bypassed content/DLP checks:
- tool_calls / Anthropic tool_use (normalized into message.extra) are
now folded into a single guardrail-inspected text view via
ChatResponse::guardrail_output_text(), used by the keyword, text-
moderation, Bedrock, and Prompt Shield output checks (#3/#18/#21).
Reasoning/thinking content is intentionally left out of scope.
- Non-streaming cache hits now run the resolved output guardrail chain
before returning the stored body, instead of replaying it unchecked
(#28). Streaming output guardrails already run end-of-stream.
Part of #448 (findings #3, #18, #21, #28)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@moonming
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground) - #28

Merged
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls
Apr 23, 2026
Merged

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground)#28
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls

Conversation

@moonming

Copy link
Copy Markdown
Member

Summary

Adds two closely-related DP-side capabilities required for AISIX to act as an aisix.cloud tenant:

  1. etcd mTLS client — the aisix.cloud DP Manager serves etcd v3 over mTLS, so the DP needs to read a CA + client cert + client key from disk and wire them into the existing etcd-clientConnectOptions.
  2. Managed-mode switch — when enabled, the admin API listener, admin UI, and Playground endpoint are not bound. All configuration flows from etcd; resource mutations happen through the cloud control plane.

This is the first of ~three DP-side PRs. Registration (fetching the mTLS bundle from cp-api at boot), heartbeat, and local snapshot fallback land in follow-ups.

Config shape

etcd:
endpoints: ["https://etcd.aisix.cloud:2379"]tls:
ca_cert_file: "/etc/aisix/mtls/ca.crt"client_cert_file: "/etc/aisix/mtls/client.crt"client_key_file: "/etc/aisix/mtls/client.key"# domain_name: "etcd.aisix.cloud" # optional; defaults to endpoints[0] hostproxy:
addr: "0.0.0.0:3000"managed:
enabled: true

Standalone users are unaffected: new fields are all optional and defaulted, validate() keeps the existing invariants when managed.enabled = false.

What the managed switch actually flips

ComponentStandaloneManaged
Proxy listener✅ bound✅ bound
etcd provider + watch supervisor
Admin etcd client (write path)❌ skipped
Admin router (/admin/v1/*, UI, Playground)✅ bound❌ never constructed
Admin TCP listener✅ bound❌ never bound
/metrics (currently mounted on admin)✅ served❌ not served

The last row is worth calling out: Phase 1 mounts Prometheus on the admin listener, so managed mode loses it. Adding a dedicated observability listener is a follow-up; for aisix.cloud tenants, Prometheus scraping is expected to happen on the cloud control plane side anyway.

New helper

build_etcd_connect_options(&EtcdConfig) -> anyhow::Result<Option<ConnectOptions>>:

  • Returns None for plain HTTP etcd (keeps the hot path cheap — no pointless ConnectOptions::new() allocation when nothing needs wiring).
  • Reads the three PEM files and constructs TlsOptions::new().domain_name(...).ca_certificate(...).identity(...).
  • Missing files surface with the config key name (etcd.tls.ca_cert_file = "/...") in the error so operators don't have to diff config against filesystem state.

default_domain_from_endpoint() extracts the SNI from URL-ish strings: http://, https://, bare host:port, and IPv6 literals [::1]:2379 — table-tested.

Tests

cargo test --workspace --all-features green (407 tests). New cases:

  • aisix-core::config::tests::managed_mode_lets_admin_fields_be_omitted — minimum aisix.cloud tenant YAML loads cleanly.
  • aisix-core::config::tests::standalone_still_requires_admin_keys_even_with_managed_false — original invariant preserved.
  • aisix-core::config::tests::parses_etcd_tls_block — round-trip for all four TLS fields.
  • aisix-server::tests::default_domain_strips_scheme_port_and_brackets — SNI extractor table.
  • aisix-server::tests::build_connect_options_none_when_plain_http — hot path stays zero-cost.
  • aisix-server::tests::build_connect_options_surfaces_missing_cert_files — operator-friendly error.

cargo fmt + cargo clippy --workspace --all-targets -- -D warnings clean.

Explicitly out of scope (follow-up PRs on this branch's successor stack)

  • Registration flow: the DP currently expects the mTLS bundle already on disk. The next PR adds POST /dp/register client that exchanges a one-time Deployment Token for the bundle at boot and persists it atomically.
  • Heartbeat: periodic POST /api/ai_dataplane/heartbeat so cp-api knows the DP is alive (and so the Gateway page in aisix.cloud shows green dots).
  • Local config snapshot: continue serving proxy traffic when the etcd watch disconnects mid-flight (see aisix.cloud PRD prd-09 §9.7.2).
  • Structured request logs + hashes: the observability schema aisix.cloud telemetry consumes.

Relationship

Paired with the aisix.cloud side in api7/AISIX-Cloud#8: that PR's GatewayHandlers.create calls IssueDataplaneCertificate on api7ee CP and returns the bundle to the user, who then drops it on their DP machine. The DP then boots with this PR's new config block pointing at those files.

…und)
Required for AISIX data planes to talk to the aisix.cloud control
plane: the CP's DP Manager serves etcd v3 over mTLS (see the
aisix.cloud PRD prd-09 §9.3.3). Phase 1 of the DP-side changes —
follow-up PRs wire registration + heartbeat + local snapshot.
## What's new
### `aisix-core::config`
- `EtcdConfig.tls: Option<EtcdTlsConfig>` — new optional mTLS bundle.
Three PEM file paths (CA cert, client cert, client key) plus an
optional `domain_name` for SNI. Defaults derive the domain from the
first endpoint's hostname.
- `Config.managed: ManagedConfig { enabled: bool }` — new top-level
switch. Defaults to standalone so existing configs keep working.
- `AdminConfig` now implements `Default` so managed-mode configs
can omit the `admin:` block entirely.
- `validate()` relaxes the `admin.addr` + `admin.admin_keys`
invariants when `managed.enabled = true`, and keeps them as-is
otherwise — no silent regression for standalone setups.
### `aisix-server::main`
- New `build_etcd_connect_options(&EtcdConfig) -> Option<ConnectOptions>`
helper. Returns `None` for plain HTTP (keep the test path cheap),
wires `with_user` + `with_tls` when present, surfaces missing
cert-file errors with the config key name in the message.
- `default_domain_from_endpoint()` extracts the SNI from URL-like
endpoint strings (`http://host:port`, `https://host:port`, bare
`host:port`, and IPv6 literals with brackets).
- The `EtcdConfigProvider::connect` + the separate admin `Client`
now share the same options (user + mTLS).
- **Admin listener is conditional.** In managed mode the admin
surface is never built:
* `admin_client` stays `None` (no second etcd connection)
* `admin_state` / `admin_router` are not constructed
* The admin TCP listener is not bound
* The Playground endpoint (mounted inside admin) vanishes
The proxy listener keeps running with the same request path.
- `run()` awaits the admin task via an `Option<JoinHandle>` so a
managed-mode start-up no longer joins on a nonexistent future.
### `config.example.yaml`
- Commented-out `etcd.tls` block with the three PEM paths.
- Commented-out `managed.enabled: true` section with a short
explanation of what flips in that mode.
## Tests
### `aisix-core`
- `managed_mode_lets_admin_fields_be_omitted`: minimum aisix.cloud
tenant YAML loads without an `admin:` block.
- `standalone_still_requires_admin_keys_even_with_managed_false`:
original invariant preserved for non-managed configs.
- `parses_etcd_tls_block`: round-trip for all four TLS fields.
### `aisix-server`
- `default_domain_strips_scheme_port_and_brackets`: table for the
SNI extractor including IPv6 brackets and bare-host cases.
- `build_connect_options_none_when_plain_http`: plain HTTP etcd
doesn't synthesise options (hot path).
- `build_connect_options_surfaces_missing_cert_files`: operator
sees *which* file is missing without grepping filesystem state.
`cargo fmt` / `cargo clippy --workspace --all-targets -- -D warnings`
clean. `cargo test --workspace --all-features` green (407 tests).
## Explicitly out of scope (follow-up PRs)
- DP registration flow (`POST /dp/register` against cp-api) that
*fetches* the mTLS bundle and persists it. This PR expects the
bundle already on disk — integration test path.
- Heartbeat (`POST /api/ai_dataplane/heartbeat` every 15s).
- Local config snapshot so the DP serves from cache when the etcd
connection dies mid-flight.
- Structured request logs + per-request hashes (prd-09 §9.6.2).
CopilotAI review requested due to automatic review settings April 23, 2026 07:58

CopilotAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds dataplane support for aisix.cloud tenants by introducing etcd mTLS client wiring and a managed-mode switch that disables the standalone admin surface.

Changes:

  • Add etcd.tls config (CA/cert/key + optional domain_name) and build etcd-clientConnectOptions with mTLS.
  • Add managed.enabled config and skip admin etcd client + admin router/listener when managed mode is on.
  • Update example config and add tests for domain derivation, connect options behavior, and managed/standalone config validation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

FileDescription
crates/aisix-server/src/main.rsBuild etcd ConnectOptions (mTLS/auth) and gate admin listener/router creation behind managed mode.
crates/aisix-core/src/lib.rsRe-export new config types (EtcdTlsConfig, ManagedConfig).
crates/aisix-core/src/config.rsExtend config schema with etcd.tls + managed, default admin for managed configs, and update validation rules.
config.example.yamlDocument new etcd.tls block and managed.enabled toggle.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +236 to +241
if let (Some(user), Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()) {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_etcd_connect_options() only applies basic auth if bothetcd.user and etcd.password_env are set. If a user sets only one of these fields (common config mistake), it will silently skip auth and likely fail with an opaque etcd permission error. Prefer returning a config error (or add a Config::validate() check) when exactly one of the two is set.

Suggested change
iflet(Some(user),Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()){
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
match(etcd.user.as_ref(), etcd.password_env.as_ref()){
(Some(user),Some(env_key)) => {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
}
(None,None) => {}
(Some(_),None) => {
returnErr(anyhow::anyhow!(
"etcd.user is set but etcd.password_env is missing; set both fields to enable etcd basic auth"
));
}
(None,Some(_)) => {
returnErr(anyhow::anyhow!(
"etcd.password_env is set but etcd.user is missing; set both fields to enable etcd basic auth"
));
}

Copilot uses AI. Check for mistakes.
fn build_etcd_connect_options(etcd: &EtcdConfig) -> anyhow::Result<Option<ConnectOptions>> {
let mut needs_options = false;
let mut options = ConnectOptions::new();

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EtcdConfig exposes dial_timeout_ms and request_timeout_ms, but build_etcd_connect_options() never uses them and there are no other call sites wiring these into etcd-client. This makes the timeout knobs in config.example.yaml ineffective. Consider setting the appropriate timeouts on ConnectOptions (and marking needs_options = true accordingly) so the config fields actually take effect.

Suggested change
ifletSome(dial_timeout_ms) = etcd.dial_timeout_ms{
options = options.with_connect_timeout(std::time::Duration::from_millis(dial_timeout_ms));
needs_options = true;
}
ifletSome(request_timeout_ms) = etcd.request_timeout_ms{
options = options.with_timeout(std::time::Duration::from_millis(request_timeout_ms));
needs_options = true;
}

Copilot uses AI. Check for mistakes.
Comment on lines +226 to +227
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc comment says cert/key I/O failures “bubble up as a … BootstrapError”, but this helper returns anyhow::Result and the produced errors are anyhow::Error strings. Consider adjusting the wording to avoid implying a specific error type here.

Suggested change
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.
/// failure bubbles up as a nicelycontextualised error at the same
/// point as other etcd connection errors.

Copilot uses AI. Check for mistakes.
Comment on lines +389 to +393
if self.admin.addr.parse::<std::net::SocketAddr>().is_err() {
return Err(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AdminConfig::default_addr() sets 127.0.0.1:0, and Config::validate() only checks that admin.addr parses (in standalone mode). This allows a standalone config that omits admin.addr (but sets admin_keys) to bind the admin listener on an ephemeral port, which contradicts the comment and is operationally surprising. Consider either (a) defaulting to the previous explicit port, or (b) rejecting port 0 / the default value in validate() when managed.enabled is false with a clear error message.

Suggested change
ifself.admin.addr.parse::<std::net::SocketAddr>().is_err(){
returnErr(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));
let admin_addr = self
.admin
.addr
.parse::<std::net::SocketAddr>()
.map_err(|_| {
BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
))
})?;
if admin_addr.port() == 0{
returnErr(BootstrapError::Config(
"admin.addr must use an explicit non-zero port \
(required when managed.enabled is false)"
.into(),
));

Copilot uses AI. Check for mistakes.
moonming added a commit that referenced this pull request Apr 23, 2026
Free-tier Actions storage is 500 MB, shared across the whole repo.
Each \`aisix-bin\` artifact is ~72 MB and we publish one per CI run,
so storage saturates after <10 main-branch pushes and blocks
\`actions/upload-artifact\` on every subsequent PR (the failure that
paused #28 twice today).
Retention tightened per artifact by expected re-read horizon:
- aisix-bin 1 day (consumed by the same-day e2e job only)
- ui-dist 7 days (consumed by same-day e2e; light enough
to keep a week for manual inspection)
- coverage-* 7 days (manual download for debugging flaky
coverage gates; LCOV is tiny)
Nothing downstream relies on week+ old binaries — \`build-bin\` is
re-runnable from source and the \`needs:\` chain on \`build-aisix
(instrumented)\` → \`e2e\` already re-produces the artifact when an
earlier run has expired.
No behaviour change on passing runs; only limits how long stale
runs squat on quota.
@moonming
moonming merged commit 56d2d54 into mainApr 23, 2026
13 of 17 checks passed
@moonming
moonming deleted the feat/managed-mode-mtls branch April 23, 2026 08:57
jarvis9443 added a commit that referenced this pull request Jun 2, 2026
Output guardrails only inspected message.content, so client-visible
output that lives elsewhere bypassed content/DLP checks:
- tool_calls / Anthropic tool_use (normalized into message.extra) are
now folded into a single guardrail-inspected text view via
ChatResponse::guardrail_output_text(), used by the keyword, text-
moderation, Bedrock, and Prompt Shield output checks (#3/#18/#21).
Reasoning/thinking content is intentionally left out of scope.
- Non-streaming cache hits now run the resolved output guardrail chain
before returning the stored body, instead of replaying it unchecked
(#28). Streaming output guardrails already run end-of-stream.
Part of #448 (findings #3, #18, #21, #28)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@moonming
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground) - #28

Merged
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls
Apr 23, 2026
Merged

feat(managed): etcd mTLS + managed-mode switch (skip admin/UI/Playground)#28
moonming merged 1 commit into
mainfrom
feat/managed-mode-mtls

Conversation

@moonming

Copy link
Copy Markdown
Member

Summary

Adds two closely-related DP-side capabilities required for AISIX to act as an aisix.cloud tenant:

  1. etcd mTLS client — the aisix.cloud DP Manager serves etcd v3 over mTLS, so the DP needs to read a CA + client cert + client key from disk and wire them into the existing etcd-clientConnectOptions.
  2. Managed-mode switch — when enabled, the admin API listener, admin UI, and Playground endpoint are not bound. All configuration flows from etcd; resource mutations happen through the cloud control plane.

This is the first of ~three DP-side PRs. Registration (fetching the mTLS bundle from cp-api at boot), heartbeat, and local snapshot fallback land in follow-ups.

Config shape

etcd:
endpoints: ["https://etcd.aisix.cloud:2379"]tls:
ca_cert_file: "/etc/aisix/mtls/ca.crt"client_cert_file: "/etc/aisix/mtls/client.crt"client_key_file: "/etc/aisix/mtls/client.key"# domain_name: "etcd.aisix.cloud" # optional; defaults to endpoints[0] hostproxy:
addr: "0.0.0.0:3000"managed:
enabled: true

Standalone users are unaffected: new fields are all optional and defaulted, validate() keeps the existing invariants when managed.enabled = false.

What the managed switch actually flips

ComponentStandaloneManaged
Proxy listener✅ bound✅ bound
etcd provider + watch supervisor
Admin etcd client (write path)❌ skipped
Admin router (/admin/v1/*, UI, Playground)✅ bound❌ never constructed
Admin TCP listener✅ bound❌ never bound
/metrics (currently mounted on admin)✅ served❌ not served

The last row is worth calling out: Phase 1 mounts Prometheus on the admin listener, so managed mode loses it. Adding a dedicated observability listener is a follow-up; for aisix.cloud tenants, Prometheus scraping is expected to happen on the cloud control plane side anyway.

New helper

build_etcd_connect_options(&EtcdConfig) -> anyhow::Result<Option<ConnectOptions>>:

  • Returns None for plain HTTP etcd (keeps the hot path cheap — no pointless ConnectOptions::new() allocation when nothing needs wiring).
  • Reads the three PEM files and constructs TlsOptions::new().domain_name(...).ca_certificate(...).identity(...).
  • Missing files surface with the config key name (etcd.tls.ca_cert_file = "/...") in the error so operators don't have to diff config against filesystem state.

default_domain_from_endpoint() extracts the SNI from URL-ish strings: http://, https://, bare host:port, and IPv6 literals [::1]:2379 — table-tested.

Tests

cargo test --workspace --all-features green (407 tests). New cases:

  • aisix-core::config::tests::managed_mode_lets_admin_fields_be_omitted — minimum aisix.cloud tenant YAML loads cleanly.
  • aisix-core::config::tests::standalone_still_requires_admin_keys_even_with_managed_false — original invariant preserved.
  • aisix-core::config::tests::parses_etcd_tls_block — round-trip for all four TLS fields.
  • aisix-server::tests::default_domain_strips_scheme_port_and_brackets — SNI extractor table.
  • aisix-server::tests::build_connect_options_none_when_plain_http — hot path stays zero-cost.
  • aisix-server::tests::build_connect_options_surfaces_missing_cert_files — operator-friendly error.

cargo fmt + cargo clippy --workspace --all-targets -- -D warnings clean.

Explicitly out of scope (follow-up PRs on this branch's successor stack)

  • Registration flow: the DP currently expects the mTLS bundle already on disk. The next PR adds POST /dp/register client that exchanges a one-time Deployment Token for the bundle at boot and persists it atomically.
  • Heartbeat: periodic POST /api/ai_dataplane/heartbeat so cp-api knows the DP is alive (and so the Gateway page in aisix.cloud shows green dots).
  • Local config snapshot: continue serving proxy traffic when the etcd watch disconnects mid-flight (see aisix.cloud PRD prd-09 §9.7.2).
  • Structured request logs + hashes: the observability schema aisix.cloud telemetry consumes.

Relationship

Paired with the aisix.cloud side in api7/AISIX-Cloud#8: that PR's GatewayHandlers.create calls IssueDataplaneCertificate on api7ee CP and returns the bundle to the user, who then drops it on their DP machine. The DP then boots with this PR's new config block pointing at those files.

…und)
Required for AISIX data planes to talk to the aisix.cloud control
plane: the CP's DP Manager serves etcd v3 over mTLS (see the
aisix.cloud PRD prd-09 §9.3.3). Phase 1 of the DP-side changes —
follow-up PRs wire registration + heartbeat + local snapshot.
## What's new
### `aisix-core::config`
- `EtcdConfig.tls: Option<EtcdTlsConfig>` — new optional mTLS bundle.
Three PEM file paths (CA cert, client cert, client key) plus an
optional `domain_name` for SNI. Defaults derive the domain from the
first endpoint's hostname.
- `Config.managed: ManagedConfig { enabled: bool }` — new top-level
switch. Defaults to standalone so existing configs keep working.
- `AdminConfig` now implements `Default` so managed-mode configs
can omit the `admin:` block entirely.
- `validate()` relaxes the `admin.addr` + `admin.admin_keys`
invariants when `managed.enabled = true`, and keeps them as-is
otherwise — no silent regression for standalone setups.
### `aisix-server::main`
- New `build_etcd_connect_options(&EtcdConfig) -> Option<ConnectOptions>`
helper. Returns `None` for plain HTTP (keep the test path cheap),
wires `with_user` + `with_tls` when present, surfaces missing
cert-file errors with the config key name in the message.
- `default_domain_from_endpoint()` extracts the SNI from URL-like
endpoint strings (`http://host:port`, `https://host:port`, bare
`host:port`, and IPv6 literals with brackets).
- The `EtcdConfigProvider::connect` + the separate admin `Client`
now share the same options (user + mTLS).
- **Admin listener is conditional.** In managed mode the admin
surface is never built:
* `admin_client` stays `None` (no second etcd connection)
* `admin_state` / `admin_router` are not constructed
* The admin TCP listener is not bound
* The Playground endpoint (mounted inside admin) vanishes
The proxy listener keeps running with the same request path.
- `run()` awaits the admin task via an `Option<JoinHandle>` so a
managed-mode start-up no longer joins on a nonexistent future.
### `config.example.yaml`
- Commented-out `etcd.tls` block with the three PEM paths.
- Commented-out `managed.enabled: true` section with a short
explanation of what flips in that mode.
## Tests
### `aisix-core`
- `managed_mode_lets_admin_fields_be_omitted`: minimum aisix.cloud
tenant YAML loads without an `admin:` block.
- `standalone_still_requires_admin_keys_even_with_managed_false`:
original invariant preserved for non-managed configs.
- `parses_etcd_tls_block`: round-trip for all four TLS fields.
### `aisix-server`
- `default_domain_strips_scheme_port_and_brackets`: table for the
SNI extractor including IPv6 brackets and bare-host cases.
- `build_connect_options_none_when_plain_http`: plain HTTP etcd
doesn't synthesise options (hot path).
- `build_connect_options_surfaces_missing_cert_files`: operator
sees *which* file is missing without grepping filesystem state.
`cargo fmt` / `cargo clippy --workspace --all-targets -- -D warnings`
clean. `cargo test --workspace --all-features` green (407 tests).
## Explicitly out of scope (follow-up PRs)
- DP registration flow (`POST /dp/register` against cp-api) that
*fetches* the mTLS bundle and persists it. This PR expects the
bundle already on disk — integration test path.
- Heartbeat (`POST /api/ai_dataplane/heartbeat` every 15s).
- Local config snapshot so the DP serves from cache when the etcd
connection dies mid-flight.
- Structured request logs + per-request hashes (prd-09 §9.6.2).
CopilotAI review requested due to automatic review settings April 23, 2026 07:58

CopilotAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds dataplane support for aisix.cloud tenants by introducing etcd mTLS client wiring and a managed-mode switch that disables the standalone admin surface.

Changes:

  • Add etcd.tls config (CA/cert/key + optional domain_name) and build etcd-clientConnectOptions with mTLS.
  • Add managed.enabled config and skip admin etcd client + admin router/listener when managed mode is on.
  • Update example config and add tests for domain derivation, connect options behavior, and managed/standalone config validation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

FileDescription
crates/aisix-server/src/main.rsBuild etcd ConnectOptions (mTLS/auth) and gate admin listener/router creation behind managed mode.
crates/aisix-core/src/lib.rsRe-export new config types (EtcdTlsConfig, ManagedConfig).
crates/aisix-core/src/config.rsExtend config schema with etcd.tls + managed, default admin for managed configs, and update validation rules.
config.example.yamlDocument new etcd.tls block and managed.enabled toggle.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +236 to +241
if let (Some(user), Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()) {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_etcd_connect_options() only applies basic auth if bothetcd.user and etcd.password_env are set. If a user sets only one of these fields (common config mistake), it will silently skip auth and likely fail with an opaque etcd permission error. Prefer returning a config error (or add a Config::validate() check) when exactly one of the two is set.

Suggested change
iflet(Some(user),Some(env_key)) = (etcd.user.as_ref(), etcd.password_env.as_ref()){
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
match(etcd.user.as_ref(), etcd.password_env.as_ref()){
(Some(user),Some(env_key)) => {
let pw = std::env::var(env_key).map_err(|_| {
anyhow::anyhow!("etcd.password_env = {env_key:?} is set but the env var is missing")
})?;
options = options.with_user(user.clone(), pw);
needs_options = true;
}
(None,None) => {}
(Some(_),None) => {
returnErr(anyhow::anyhow!(
"etcd.user is set but etcd.password_env is missing; set both fields to enable etcd basic auth"
));
}
(None,Some(_)) => {
returnErr(anyhow::anyhow!(
"etcd.password_env is set but etcd.user is missing; set both fields to enable etcd basic auth"
));
}

Copilot uses AI. Check for mistakes.
fn build_etcd_connect_options(etcd: &EtcdConfig) -> anyhow::Result<Option<ConnectOptions>> {
let mut needs_options = false;
let mut options = ConnectOptions::new();

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EtcdConfig exposes dial_timeout_ms and request_timeout_ms, but build_etcd_connect_options() never uses them and there are no other call sites wiring these into etcd-client. This makes the timeout knobs in config.example.yaml ineffective. Consider setting the appropriate timeouts on ConnectOptions (and marking needs_options = true accordingly) so the config fields actually take effect.

Suggested change
ifletSome(dial_timeout_ms) = etcd.dial_timeout_ms{
options = options.with_connect_timeout(std::time::Duration::from_millis(dial_timeout_ms));
needs_options = true;
}
ifletSome(request_timeout_ms) = etcd.request_timeout_ms{
options = options.with_timeout(std::time::Duration::from_millis(request_timeout_ms));
needs_options = true;
}

Copilot uses AI. Check for mistakes.
Comment on lines +226 to +227
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The doc comment says cert/key I/O failures “bubble up as a … BootstrapError”, but this helper returns anyhow::Result and the produced errors are anyhow::Error strings. Consider adjusting the wording to avoid implying a specific error type here.

Suggested change
/// failure bubbles up as a nicely-contextualised BootstrapError at
/// the same point as other etcd connection errors.
/// failure bubbles up as a nicelycontextualised error at the same
/// point as other etcd connection errors.

Copilot uses AI. Check for mistakes.
Comment on lines +389 to +393
if self.admin.addr.parse::<std::net::SocketAddr>().is_err() {
return Err(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));

CopilotAIApr 23, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AdminConfig::default_addr() sets 127.0.0.1:0, and Config::validate() only checks that admin.addr parses (in standalone mode). This allows a standalone config that omits admin.addr (but sets admin_keys) to bind the admin listener on an ephemeral port, which contradicts the comment and is operationally surprising. Consider either (a) defaulting to the previous explicit port, or (b) rejecting port 0 / the default value in validate() when managed.enabled is false with a clear error message.

Suggested change
ifself.admin.addr.parse::<std::net::SocketAddr>().is_err(){
returnErr(BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
)));
let admin_addr = self
.admin
.addr
.parse::<std::net::SocketAddr>()
.map_err(|_| {
BootstrapError::Config(format!(
"admin.addr invalid socket address: {}",
self.admin.addr
))
})?;
if admin_addr.port() == 0{
returnErr(BootstrapError::Config(
"admin.addr must use an explicit non-zero port \
(required when managed.enabled is false)"
.into(),
));

Copilot uses AI. Check for mistakes.
moonming added a commit that referenced this pull request Apr 23, 2026
Free-tier Actions storage is 500 MB, shared across the whole repo.
Each \`aisix-bin\` artifact is ~72 MB and we publish one per CI run,
so storage saturates after <10 main-branch pushes and blocks
\`actions/upload-artifact\` on every subsequent PR (the failure that
paused #28 twice today).
Retention tightened per artifact by expected re-read horizon:
- aisix-bin 1 day (consumed by the same-day e2e job only)
- ui-dist 7 days (consumed by same-day e2e; light enough
to keep a week for manual inspection)
- coverage-* 7 days (manual download for debugging flaky
coverage gates; LCOV is tiny)
Nothing downstream relies on week+ old binaries — \`build-bin\` is
re-runnable from source and the \`needs:\` chain on \`build-aisix
(instrumented)\` → \`e2e\` already re-produces the artifact when an
earlier run has expired.
No behaviour change on passing runs; only limits how long stale
runs squat on quota.
@moonming
moonming merged commit 56d2d54 into mainApr 23, 2026
13 of 17 checks passed
@moonming
moonming deleted the feat/managed-mode-mtls branch April 23, 2026 08:57
jarvis9443 added a commit that referenced this pull request Jun 2, 2026
Output guardrails only inspected message.content, so client-visible
output that lives elsewhere bypassed content/DLP checks:
- tool_calls / Anthropic tool_use (normalized into message.extra) are
now folded into a single guardrail-inspected text view via
ChatResponse::guardrail_output_text(), used by the keyword, text-
moderation, Bedrock, and Prompt Shield output checks (#3/#18/#21).
Reasoning/thinking content is intentionally left out of scope.
- Non-streaming cache hits now run the resolved output guardrail chain
before returning the stored body, instead of replaying it unchecked
(#28). Streaming output guardrails already run end-of-stream.
Part of #448 (findings #3, #18, #21, #28)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@moonming