fix(aisix-etcd): deterministic wait for supervisor cache-write tests

[moonming]

Summary

• Replace the racy `tokio::time::sleep(50ms)` waits in the two supervisor cache-write tests with a real synchroniser
• `flush_cache` now records each spawned `JoinHandle` in a new `pending_writes` field
• Test-only `await_pending_cache_writes` drains and awaits those handles deterministically

Why

main has been red on these two tests for several days, and PR #39 inherited the breakage on rerun:

```
supervisor::tests::resync_writes_to_disk_cache_then_restore_replays_it FAILED
supervisor::tests::put_and_delete_keep_cache_in_sync FAILED
```

Both tests asserted on a disk read that happens after a 50ms sleep. The sleep was meant to give the `tokio::spawn`-ed cache writer time to flush, but on heavily loaded GitHub Actions runners 50ms isn't always enough — the spawn loses the race and the test reads an empty cache file. Bumping the sleep is a bandaid; this PR removes the timing guess entirely.

Approach

`Supervisor` gains one new field:

```rust
pending_writes: Mutex<Vec<JoinHandle<()>>>,
```

`flush_cache` pushes the spawned handle onto it. `await_pending_cache_writes` (test-only via documented usage) drains the vec and awaits every handle. Production code never reads the field; if a handle is dropped during shutdown the underlying write either completed or was cancelled, which is fine — the on-disk cache is best-effort and the next live cycle re-publishes from etcd.

Test plan

• `cargo test -p aisix-etcd --lib` → 36 passed, 0 failed
• `cargo clippy -p aisix-etcd --all-targets -- -D warnings` clean
• CI green on both `rust unit + coverage` and the rest of the matrix

Out of scope

• Other CI matrix items (UI build, e2e). Those failures, if any, are separate.

[Copilot]

Pull request overview

This PR removes racy, wall-clock-based sleeps from aisix-etcd supervisor disk-cache tests by introducing a deterministic test synchronizer that awaits spawned cache-write tasks.

Changes:

• Add pending_writes tracking for spawned flush_cache write tasks.
• Add a test-only await_pending_cache_writes() helper to deterministically await cache persistence.
• Update two flaky tests to use the new synchronizer instead of sleep(50ms).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

pending_writes retains every spawned cache-write JoinHandle for the lifetime of Supervisor, but production code never drains it. In a long-running process with frequent Put/Delete/Resync events, this will grow unbounded and keep completed tasks alive, leading to a memory leak. Consider making handle tracking #[cfg(test)] only (and drop(join) in non-test), or pruning finished handles (e.g., retain(|h| !h.is_finished())) before/after pushing new ones.

[Copilot]

await_pending_cache_writes awaits each JoinHandle but discards the result, which will silently swallow panics from cache.store(...). Since this is a test-only synchronizer, it's better to fail the test if a cache write task panics/cancels (e.g., by asserting the JoinError is OK), otherwise the test may pass/fail for the wrong reason and the underlying bug is harder to diagnose.

Suggested change

	// Failures here are not test failures — a write that
	// panicked is its own bug surfaced separately. We only
	// need the await to deterministically order against the
	// disk read that follows.
	let _ = handle.await;
	// In tests, a cache write task panic/cancellation should fail
	// loudly so the underlying bug is surfaced directly instead of
	// being masked by whatever assertion runs after the disk read.
	handle
	.await
	.expect("pending cache write task panicked or was cancelled");

[Copilot]

flush_cache now always pushes the spawned JoinHandle into pending_writes. Even if the intention is “test-only”, this path also runs in production and will accumulate handles unless drained. If you keep pending_writes in the struct, gate the push with #[cfg(test)] (and drop the handle otherwise) or drain/prune completed handles here.

Suggested change

	// Track the JoinHandle so tests can deterministically await
	// the write via [`Self::await_pending_cache_writes`] instead
	// of leaning on `tokio::time::sleep`, which under CI load
	// raced the spawn (~50ms wasn't enough on heavily loaded
	// GitHub Actions runners).
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	self.pending_writes.lock().unwrap().push(join);
	// Track the JoinHandle only in tests so they can
	// deterministically await the write via
	// [`Self::await_pending_cache_writes`] instead of leaning on
	// `tokio::time::sleep`, which under CI load raced the spawn
	// (~50ms wasn't enough on heavily loaded GitHub Actions
	// runners). In non-test builds, drop the handle immediately to
	// detach the task and avoid accumulating completed handles.
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	#[cfg(test)]
	self.pending_writes.lock().unwrap().push(join);
	#[cfg(not(test))]
	drop(join);

[moonming]

Summary

• Replace the racy `tokio::time::sleep(50ms)` waits in the two supervisor cache-write tests with a real synchroniser
• `flush_cache` now records each spawned `JoinHandle` in a new `pending_writes` field
• Test-only `await_pending_cache_writes` drains and awaits those handles deterministically

Why

main has been red on these two tests for several days, and PR #39 inherited the breakage on rerun:

```
supervisor::tests::resync_writes_to_disk_cache_then_restore_replays_it FAILED
supervisor::tests::put_and_delete_keep_cache_in_sync FAILED
```

Both tests asserted on a disk read that happens after a 50ms sleep. The sleep was meant to give the `tokio::spawn`-ed cache writer time to flush, but on heavily loaded GitHub Actions runners 50ms isn't always enough — the spawn loses the race and the test reads an empty cache file. Bumping the sleep is a bandaid; this PR removes the timing guess entirely.

Approach

`Supervisor` gains one new field:

```rust
pending_writes: Mutex<Vec<JoinHandle<()>>>,
```

`flush_cache` pushes the spawned handle onto it. `await_pending_cache_writes` (test-only via documented usage) drains the vec and awaits every handle. Production code never reads the field; if a handle is dropped during shutdown the underlying write either completed or was cancelled, which is fine — the on-disk cache is best-effort and the next live cycle re-publishes from etcd.

Test plan

• `cargo test -p aisix-etcd --lib` → 36 passed, 0 failed
• `cargo clippy -p aisix-etcd --all-targets -- -D warnings` clean
• CI green on both `rust unit + coverage` and the rest of the matrix

Out of scope

• Other CI matrix items (UI build, e2e). Those failures, if any, are separate.

[Copilot]

Pull request overview

This PR removes racy, wall-clock-based sleeps from aisix-etcd supervisor disk-cache tests by introducing a deterministic test synchronizer that awaits spawned cache-write tasks.

Changes:

• Add pending_writes tracking for spawned flush_cache write tasks.
• Add a test-only await_pending_cache_writes() helper to deterministically await cache persistence.
• Update two flaky tests to use the new synchronizer instead of sleep(50ms).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

pending_writes retains every spawned cache-write JoinHandle for the lifetime of Supervisor, but production code never drains it. In a long-running process with frequent Put/Delete/Resync events, this will grow unbounded and keep completed tasks alive, leading to a memory leak. Consider making handle tracking #[cfg(test)] only (and drop(join) in non-test), or pruning finished handles (e.g., retain(|h| !h.is_finished())) before/after pushing new ones.

[Copilot]

await_pending_cache_writes awaits each JoinHandle but discards the result, which will silently swallow panics from cache.store(...). Since this is a test-only synchronizer, it's better to fail the test if a cache write task panics/cancels (e.g., by asserting the JoinError is OK), otherwise the test may pass/fail for the wrong reason and the underlying bug is harder to diagnose.

Suggested change

	// Failures here are not test failures — a write that
	// panicked is its own bug surfaced separately. We only
	// need the await to deterministically order against the
	// disk read that follows.
	let _ = handle.await;
	// In tests, a cache write task panic/cancellation should fail
	// loudly so the underlying bug is surfaced directly instead of
	// being masked by whatever assertion runs after the disk read.
	handle
	.await
	.expect("pending cache write task panicked or was cancelled");

[Copilot]

flush_cache now always pushes the spawned JoinHandle into pending_writes. Even if the intention is “test-only”, this path also runs in production and will accumulate handles unless drained. If you keep pending_writes in the struct, gate the push with #[cfg(test)] (and drop the handle otherwise) or drain/prune completed handles here.

Suggested change

	// Track the JoinHandle so tests can deterministically await
	// the write via [`Self::await_pending_cache_writes`] instead
	// of leaning on `tokio::time::sleep`, which under CI load
	// raced the spawn (~50ms wasn't enough on heavily loaded
	// GitHub Actions runners).
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	self.pending_writes.lock().unwrap().push(join);
	// Track the JoinHandle only in tests so they can
	// deterministically await the write via
	// [`Self::await_pending_cache_writes`] instead of leaning on
	// `tokio::time::sleep`, which under CI load raced the spawn
	// (~50ms wasn't enough on heavily loaded GitHub Actions
	// runners). In non-test builds, drop the handle immediately to
	// detach the task and avoid accumulating completed handles.
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	#[cfg(test)]
	self.pending_writes.lock().unwrap().push(join);
	#[cfg(not(test))]
	drop(join);

[moonming]

Summary

• Replace the racy `tokio::time::sleep(50ms)` waits in the two supervisor cache-write tests with a real synchroniser
• `flush_cache` now records each spawned `JoinHandle` in a new `pending_writes` field
• Test-only `await_pending_cache_writes` drains and awaits those handles deterministically

Why

main has been red on these two tests for several days, and PR #39 inherited the breakage on rerun:

```
supervisor::tests::resync_writes_to_disk_cache_then_restore_replays_it FAILED
supervisor::tests::put_and_delete_keep_cache_in_sync FAILED
```

Both tests asserted on a disk read that happens after a 50ms sleep. The sleep was meant to give the `tokio::spawn`-ed cache writer time to flush, but on heavily loaded GitHub Actions runners 50ms isn't always enough — the spawn loses the race and the test reads an empty cache file. Bumping the sleep is a bandaid; this PR removes the timing guess entirely.

Approach

`Supervisor` gains one new field:

```rust
pending_writes: Mutex<Vec<JoinHandle<()>>>,
```

`flush_cache` pushes the spawned handle onto it. `await_pending_cache_writes` (test-only via documented usage) drains the vec and awaits every handle. Production code never reads the field; if a handle is dropped during shutdown the underlying write either completed or was cancelled, which is fine — the on-disk cache is best-effort and the next live cycle re-publishes from etcd.

Test plan

• `cargo test -p aisix-etcd --lib` → 36 passed, 0 failed
• `cargo clippy -p aisix-etcd --all-targets -- -D warnings` clean
• CI green on both `rust unit + coverage` and the rest of the matrix

Out of scope

• Other CI matrix items (UI build, e2e). Those failures, if any, are separate.

[Copilot]

Pull request overview

This PR removes racy, wall-clock-based sleeps from aisix-etcd supervisor disk-cache tests by introducing a deterministic test synchronizer that awaits spawned cache-write tasks.

Changes:

• Add pending_writes tracking for spawned flush_cache write tasks.
• Add a test-only await_pending_cache_writes() helper to deterministically await cache persistence.
• Update two flaky tests to use the new synchronizer instead of sleep(50ms).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

pending_writes retains every spawned cache-write JoinHandle for the lifetime of Supervisor, but production code never drains it. In a long-running process with frequent Put/Delete/Resync events, this will grow unbounded and keep completed tasks alive, leading to a memory leak. Consider making handle tracking #[cfg(test)] only (and drop(join) in non-test), or pruning finished handles (e.g., retain(|h| !h.is_finished())) before/after pushing new ones.

[Copilot]

await_pending_cache_writes awaits each JoinHandle but discards the result, which will silently swallow panics from cache.store(...). Since this is a test-only synchronizer, it's better to fail the test if a cache write task panics/cancels (e.g., by asserting the JoinError is OK), otherwise the test may pass/fail for the wrong reason and the underlying bug is harder to diagnose.

Suggested change

	// Failures here are not test failures — a write that
	// panicked is its own bug surfaced separately. We only
	// need the await to deterministically order against the
	// disk read that follows.
	let _ = handle.await;
	// In tests, a cache write task panic/cancellation should fail
	// loudly so the underlying bug is surfaced directly instead of
	// being masked by whatever assertion runs after the disk read.
	handle
	.await
	.expect("pending cache write task panicked or was cancelled");

[Copilot]

flush_cache now always pushes the spawned JoinHandle into pending_writes. Even if the intention is “test-only”, this path also runs in production and will accumulate handles unless drained. If you keep pending_writes in the struct, gate the push with #[cfg(test)] (and drop the handle otherwise) or drain/prune completed handles here.

Suggested change

	// Track the JoinHandle so tests can deterministically await
	// the write via [`Self::await_pending_cache_writes`] instead
	// of leaning on `tokio::time::sleep`, which under CI load
	// raced the spawn (~50ms wasn't enough on heavily loaded
	// GitHub Actions runners).
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	self.pending_writes.lock().unwrap().push(join);
	// Track the JoinHandle only in tests so they can
	// deterministically await the write via
	// [`Self::await_pending_cache_writes`] instead of leaning on
	// `tokio::time::sleep`, which under CI load raced the spawn
	// (~50ms wasn't enough on heavily loaded GitHub Actions
	// runners). In non-test builds, drop the handle immediately to
	// detach the task and avoid accumulating completed handles.
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	#[cfg(test)]
	self.pending_writes.lock().unwrap().push(join);
	#[cfg(not(test))]
	drop(join);

[moonming]

Summary

• Replace the racy `tokio::time::sleep(50ms)` waits in the two supervisor cache-write tests with a real synchroniser
• `flush_cache` now records each spawned `JoinHandle` in a new `pending_writes` field
• Test-only `await_pending_cache_writes` drains and awaits those handles deterministically

Why

main has been red on these two tests for several days, and PR #39 inherited the breakage on rerun:

```
supervisor::tests::resync_writes_to_disk_cache_then_restore_replays_it FAILED
supervisor::tests::put_and_delete_keep_cache_in_sync FAILED
```

Both tests asserted on a disk read that happens after a 50ms sleep. The sleep was meant to give the `tokio::spawn`-ed cache writer time to flush, but on heavily loaded GitHub Actions runners 50ms isn't always enough — the spawn loses the race and the test reads an empty cache file. Bumping the sleep is a bandaid; this PR removes the timing guess entirely.

Approach

`Supervisor` gains one new field:

```rust
pending_writes: Mutex<Vec<JoinHandle<()>>>,
```

`flush_cache` pushes the spawned handle onto it. `await_pending_cache_writes` (test-only via documented usage) drains the vec and awaits every handle. Production code never reads the field; if a handle is dropped during shutdown the underlying write either completed or was cancelled, which is fine — the on-disk cache is best-effort and the next live cycle re-publishes from etcd.

Test plan

• `cargo test -p aisix-etcd --lib` → 36 passed, 0 failed
• `cargo clippy -p aisix-etcd --all-targets -- -D warnings` clean
• CI green on both `rust unit + coverage` and the rest of the matrix

Out of scope

• Other CI matrix items (UI build, e2e). Those failures, if any, are separate.

[Copilot]

Pull request overview

This PR removes racy, wall-clock-based sleeps from aisix-etcd supervisor disk-cache tests by introducing a deterministic test synchronizer that awaits spawned cache-write tasks.

Changes:

• Add pending_writes tracking for spawned flush_cache write tasks.
• Add a test-only await_pending_cache_writes() helper to deterministically await cache persistence.
• Update two flaky tests to use the new synchronizer instead of sleep(50ms).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

pending_writes retains every spawned cache-write JoinHandle for the lifetime of Supervisor, but production code never drains it. In a long-running process with frequent Put/Delete/Resync events, this will grow unbounded and keep completed tasks alive, leading to a memory leak. Consider making handle tracking #[cfg(test)] only (and drop(join) in non-test), or pruning finished handles (e.g., retain(|h| !h.is_finished())) before/after pushing new ones.

[Copilot]

await_pending_cache_writes awaits each JoinHandle but discards the result, which will silently swallow panics from cache.store(...). Since this is a test-only synchronizer, it's better to fail the test if a cache write task panics/cancels (e.g., by asserting the JoinError is OK), otherwise the test may pass/fail for the wrong reason and the underlying bug is harder to diagnose.

Suggested change

	// Failures here are not test failures — a write that
	// panicked is its own bug surfaced separately. We only
	// need the await to deterministically order against the
	// disk read that follows.
	let _ = handle.await;
	// In tests, a cache write task panic/cancellation should fail
	// loudly so the underlying bug is surfaced directly instead of
	// being masked by whatever assertion runs after the disk read.
	handle
	.await
	.expect("pending cache write task panicked or was cancelled");

[Copilot]

flush_cache now always pushes the spawned JoinHandle into pending_writes. Even if the intention is “test-only”, this path also runs in production and will accumulate handles unless drained. If you keep pending_writes in the struct, gate the push with #[cfg(test)] (and drop the handle otherwise) or drain/prune completed handles here.

Suggested change

	// Track the JoinHandle so tests can deterministically await
	// the write via [`Self::await_pending_cache_writes`] instead
	// of leaning on `tokio::time::sleep`, which under CI load
	// raced the spawn (~50ms wasn't enough on heavily loaded
	// GitHub Actions runners).
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	self.pending_writes.lock().unwrap().push(join);
	// Track the JoinHandle only in tests so they can
	// deterministically await the write via
	// [`Self::await_pending_cache_writes`] instead of leaning on
	// `tokio::time::sleep`, which under CI load raced the spawn
	// (~50ms wasn't enough on heavily loaded GitHub Actions
	// runners). In non-test builds, drop the handle immediately to
	// detach the task and avoid accumulating completed handles.
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	#[cfg(test)]
	self.pending_writes.lock().unwrap().push(join);
	#[cfg(not(test))]
	drop(join);

[moonming]

Summary

• Replace the racy `tokio::time::sleep(50ms)` waits in the two supervisor cache-write tests with a real synchroniser
• `flush_cache` now records each spawned `JoinHandle` in a new `pending_writes` field
• Test-only `await_pending_cache_writes` drains and awaits those handles deterministically

Why

main has been red on these two tests for several days, and PR #39 inherited the breakage on rerun:

```
supervisor::tests::resync_writes_to_disk_cache_then_restore_replays_it FAILED
supervisor::tests::put_and_delete_keep_cache_in_sync FAILED
```

Both tests asserted on a disk read that happens after a 50ms sleep. The sleep was meant to give the `tokio::spawn`-ed cache writer time to flush, but on heavily loaded GitHub Actions runners 50ms isn't always enough — the spawn loses the race and the test reads an empty cache file. Bumping the sleep is a bandaid; this PR removes the timing guess entirely.

Approach

`Supervisor` gains one new field:

```rust
pending_writes: Mutex<Vec<JoinHandle<()>>>,
```

`flush_cache` pushes the spawned handle onto it. `await_pending_cache_writes` (test-only via documented usage) drains the vec and awaits every handle. Production code never reads the field; if a handle is dropped during shutdown the underlying write either completed or was cancelled, which is fine — the on-disk cache is best-effort and the next live cycle re-publishes from etcd.

Test plan

• `cargo test -p aisix-etcd --lib` → 36 passed, 0 failed
• `cargo clippy -p aisix-etcd --all-targets -- -D warnings` clean
• CI green on both `rust unit + coverage` and the rest of the matrix

Out of scope

• Other CI matrix items (UI build, e2e). Those failures, if any, are separate.

[Copilot]

Pull request overview

This PR removes racy, wall-clock-based sleeps from aisix-etcd supervisor disk-cache tests by introducing a deterministic test synchronizer that awaits spawned cache-write tasks.

Changes:

• Add pending_writes tracking for spawned flush_cache write tasks.
• Add a test-only await_pending_cache_writes() helper to deterministically await cache persistence.
• Update two flaky tests to use the new synchronizer instead of sleep(50ms).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

pending_writes retains every spawned cache-write JoinHandle for the lifetime of Supervisor, but production code never drains it. In a long-running process with frequent Put/Delete/Resync events, this will grow unbounded and keep completed tasks alive, leading to a memory leak. Consider making handle tracking #[cfg(test)] only (and drop(join) in non-test), or pruning finished handles (e.g., retain(|h| !h.is_finished())) before/after pushing new ones.

[Copilot]

await_pending_cache_writes awaits each JoinHandle but discards the result, which will silently swallow panics from cache.store(...). Since this is a test-only synchronizer, it's better to fail the test if a cache write task panics/cancels (e.g., by asserting the JoinError is OK), otherwise the test may pass/fail for the wrong reason and the underlying bug is harder to diagnose.

Suggested change

	// Failures here are not test failures — a write that
	// panicked is its own bug surfaced separately. We only
	// need the await to deterministically order against the
	// disk read that follows.
	let _ = handle.await;
	// In tests, a cache write task panic/cancellation should fail
	// loudly so the underlying bug is surfaced directly instead of
	// being masked by whatever assertion runs after the disk read.
	handle
	.await
	.expect("pending cache write task panicked or was cancelled");

[Copilot]

flush_cache now always pushes the spawned JoinHandle into pending_writes. Even if the intention is “test-only”, this path also runs in production and will accumulate handles unless drained. If you keep pending_writes in the struct, gate the push with #[cfg(test)] (and drop the handle otherwise) or drain/prune completed handles here.

Suggested change

	// Track the JoinHandle so tests can deterministically await
	// the write via [`Self::await_pending_cache_writes`] instead
	// of leaning on `tokio::time::sleep`, which under CI load
	// raced the spawn (~50ms wasn't enough on heavily loaded
	// GitHub Actions runners).
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	self.pending_writes.lock().unwrap().push(join);
	// Track the JoinHandle only in tests so they can
	// deterministically await the write via
	// [`Self::await_pending_cache_writes`] instead of leaning on
	// `tokio::time::sleep`, which under CI load raced the spawn
	// (~50ms wasn't enough on heavily loaded GitHub Actions
	// runners). In non-test builds, drop the handle immediately to
	// detach the task and avoid accumulating completed handles.
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	#[cfg(test)]
	self.pending_writes.lock().unwrap().push(join);
	#[cfg(not(test))]
	drop(join);

[moonming]

Summary

• Replace the racy `tokio::time::sleep(50ms)` waits in the two supervisor cache-write tests with a real synchroniser
• `flush_cache` now records each spawned `JoinHandle` in a new `pending_writes` field
• Test-only `await_pending_cache_writes` drains and awaits those handles deterministically

Why

main has been red on these two tests for several days, and PR #39 inherited the breakage on rerun:

```
supervisor::tests::resync_writes_to_disk_cache_then_restore_replays_it FAILED
supervisor::tests::put_and_delete_keep_cache_in_sync FAILED
```

Both tests asserted on a disk read that happens after a 50ms sleep. The sleep was meant to give the `tokio::spawn`-ed cache writer time to flush, but on heavily loaded GitHub Actions runners 50ms isn't always enough — the spawn loses the race and the test reads an empty cache file. Bumping the sleep is a bandaid; this PR removes the timing guess entirely.

Approach

`Supervisor` gains one new field:

```rust
pending_writes: Mutex<Vec<JoinHandle<()>>>,
```

`flush_cache` pushes the spawned handle onto it. `await_pending_cache_writes` (test-only via documented usage) drains the vec and awaits every handle. Production code never reads the field; if a handle is dropped during shutdown the underlying write either completed or was cancelled, which is fine — the on-disk cache is best-effort and the next live cycle re-publishes from etcd.

Test plan

• `cargo test -p aisix-etcd --lib` → 36 passed, 0 failed
• `cargo clippy -p aisix-etcd --all-targets -- -D warnings` clean
• CI green on both `rust unit + coverage` and the rest of the matrix

Out of scope

• Other CI matrix items (UI build, e2e). Those failures, if any, are separate.

[Copilot]

Pull request overview

This PR removes racy, wall-clock-based sleeps from aisix-etcd supervisor disk-cache tests by introducing a deterministic test synchronizer that awaits spawned cache-write tasks.

Changes:

• Add pending_writes tracking for spawned flush_cache write tasks.
• Add a test-only await_pending_cache_writes() helper to deterministically await cache persistence.
• Update two flaky tests to use the new synchronizer instead of sleep(50ms).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

pending_writes retains every spawned cache-write JoinHandle for the lifetime of Supervisor, but production code never drains it. In a long-running process with frequent Put/Delete/Resync events, this will grow unbounded and keep completed tasks alive, leading to a memory leak. Consider making handle tracking #[cfg(test)] only (and drop(join) in non-test), or pruning finished handles (e.g., retain(|h| !h.is_finished())) before/after pushing new ones.

[Copilot]

await_pending_cache_writes awaits each JoinHandle but discards the result, which will silently swallow panics from cache.store(...). Since this is a test-only synchronizer, it's better to fail the test if a cache write task panics/cancels (e.g., by asserting the JoinError is OK), otherwise the test may pass/fail for the wrong reason and the underlying bug is harder to diagnose.

Suggested change

	// Failures here are not test failures — a write that
	// panicked is its own bug surfaced separately. We only
	// need the await to deterministically order against the
	// disk read that follows.
	let _ = handle.await;
	// In tests, a cache write task panic/cancellation should fail
	// loudly so the underlying bug is surfaced directly instead of
	// being masked by whatever assertion runs after the disk read.
	handle
	.await
	.expect("pending cache write task panicked or was cancelled");

[Copilot]

flush_cache now always pushes the spawned JoinHandle into pending_writes. Even if the intention is “test-only”, this path also runs in production and will accumulate handles unless drained. If you keep pending_writes in the struct, gate the push with #[cfg(test)] (and drop the handle otherwise) or drain/prune completed handles here.

Suggested change

	// Track the JoinHandle so tests can deterministically await
	// the write via [`Self::await_pending_cache_writes`] instead
	// of leaning on `tokio::time::sleep`, which under CI load
	// raced the spawn (~50ms wasn't enough on heavily loaded
	// GitHub Actions runners).
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	self.pending_writes.lock().unwrap().push(join);
	// Track the JoinHandle only in tests so they can
	// deterministically await the write via
	// [`Self::await_pending_cache_writes`] instead of leaning on
	// `tokio::time::sleep`, which under CI load raced the spawn
	// (~50ms wasn't enough on heavily loaded GitHub Actions
	// runners). In non-test builds, drop the handle immediately to
	// detach the task and avoid accumulating completed handles.
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	#[cfg(test)]
	self.pending_writes.lock().unwrap().push(join);
	#[cfg(not(test))]
	drop(join);

[moonming]

Summary

• Replace the racy `tokio::time::sleep(50ms)` waits in the two supervisor cache-write tests with a real synchroniser
• `flush_cache` now records each spawned `JoinHandle` in a new `pending_writes` field
• Test-only `await_pending_cache_writes` drains and awaits those handles deterministically

Why

main has been red on these two tests for several days, and PR #39 inherited the breakage on rerun:

```
supervisor::tests::resync_writes_to_disk_cache_then_restore_replays_it FAILED
supervisor::tests::put_and_delete_keep_cache_in_sync FAILED
```

Both tests asserted on a disk read that happens after a 50ms sleep. The sleep was meant to give the `tokio::spawn`-ed cache writer time to flush, but on heavily loaded GitHub Actions runners 50ms isn't always enough — the spawn loses the race and the test reads an empty cache file. Bumping the sleep is a bandaid; this PR removes the timing guess entirely.

Approach

`Supervisor` gains one new field:

```rust
pending_writes: Mutex<Vec<JoinHandle<()>>>,
```

`flush_cache` pushes the spawned handle onto it. `await_pending_cache_writes` (test-only via documented usage) drains the vec and awaits every handle. Production code never reads the field; if a handle is dropped during shutdown the underlying write either completed or was cancelled, which is fine — the on-disk cache is best-effort and the next live cycle re-publishes from etcd.

Test plan

• `cargo test -p aisix-etcd --lib` → 36 passed, 0 failed
• `cargo clippy -p aisix-etcd --all-targets -- -D warnings` clean
• CI green on both `rust unit + coverage` and the rest of the matrix

Out of scope

• Other CI matrix items (UI build, e2e). Those failures, if any, are separate.

[Copilot]

Pull request overview

This PR removes racy, wall-clock-based sleeps from aisix-etcd supervisor disk-cache tests by introducing a deterministic test synchronizer that awaits spawned cache-write tasks.

Changes:

• Add pending_writes tracking for spawned flush_cache write tasks.
• Add a test-only await_pending_cache_writes() helper to deterministically await cache persistence.
• Update two flaky tests to use the new synchronizer instead of sleep(50ms).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

pending_writes retains every spawned cache-write JoinHandle for the lifetime of Supervisor, but production code never drains it. In a long-running process with frequent Put/Delete/Resync events, this will grow unbounded and keep completed tasks alive, leading to a memory leak. Consider making handle tracking #[cfg(test)] only (and drop(join) in non-test), or pruning finished handles (e.g., retain(|h| !h.is_finished())) before/after pushing new ones.

[Copilot]

await_pending_cache_writes awaits each JoinHandle but discards the result, which will silently swallow panics from cache.store(...). Since this is a test-only synchronizer, it's better to fail the test if a cache write task panics/cancels (e.g., by asserting the JoinError is OK), otherwise the test may pass/fail for the wrong reason and the underlying bug is harder to diagnose.

Suggested change

	// Failures here are not test failures — a write that
	// panicked is its own bug surfaced separately. We only
	// need the await to deterministically order against the
	// disk read that follows.
	let _ = handle.await;
	// In tests, a cache write task panic/cancellation should fail
	// loudly so the underlying bug is surfaced directly instead of
	// being masked by whatever assertion runs after the disk read.
	handle
	.await
	.expect("pending cache write task panicked or was cancelled");

[Copilot]

flush_cache now always pushes the spawned JoinHandle into pending_writes. Even if the intention is “test-only”, this path also runs in production and will accumulate handles unless drained. If you keep pending_writes in the struct, gate the push with #[cfg(test)] (and drop the handle otherwise) or drain/prune completed handles here.

Suggested change

	// Track the JoinHandle so tests can deterministically await
	// the write via [`Self::await_pending_cache_writes`] instead
	// of leaning on `tokio::time::sleep`, which under CI load
	// raced the spawn (~50ms wasn't enough on heavily loaded
	// GitHub Actions runners).
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	self.pending_writes.lock().unwrap().push(join);
	// Track the JoinHandle only in tests so they can
	// deterministically await the write via
	// [`Self::await_pending_cache_writes`] instead of leaning on
	// `tokio::time::sleep`, which under CI load raced the spawn
	// (~50ms wasn't enough on heavily loaded GitHub Actions
	// runners). In non-test builds, drop the handle immediately to
	// detach the task and avoid accumulating completed handles.
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	#[cfg(test)]
	self.pending_writes.lock().unwrap().push(join);
	#[cfg(not(test))]
	drop(join);

[moonming]

Summary

• Replace the racy `tokio::time::sleep(50ms)` waits in the two supervisor cache-write tests with a real synchroniser
• `flush_cache` now records each spawned `JoinHandle` in a new `pending_writes` field
• Test-only `await_pending_cache_writes` drains and awaits those handles deterministically

Why

main has been red on these two tests for several days, and PR #39 inherited the breakage on rerun:

```
supervisor::tests::resync_writes_to_disk_cache_then_restore_replays_it FAILED
supervisor::tests::put_and_delete_keep_cache_in_sync FAILED
```

Both tests asserted on a disk read that happens after a 50ms sleep. The sleep was meant to give the `tokio::spawn`-ed cache writer time to flush, but on heavily loaded GitHub Actions runners 50ms isn't always enough — the spawn loses the race and the test reads an empty cache file. Bumping the sleep is a bandaid; this PR removes the timing guess entirely.

Approach

`Supervisor` gains one new field:

```rust
pending_writes: Mutex<Vec<JoinHandle<()>>>,
```

`flush_cache` pushes the spawned handle onto it. `await_pending_cache_writes` (test-only via documented usage) drains the vec and awaits every handle. Production code never reads the field; if a handle is dropped during shutdown the underlying write either completed or was cancelled, which is fine — the on-disk cache is best-effort and the next live cycle re-publishes from etcd.

Test plan

• `cargo test -p aisix-etcd --lib` → 36 passed, 0 failed
• `cargo clippy -p aisix-etcd --all-targets -- -D warnings` clean
• CI green on both `rust unit + coverage` and the rest of the matrix

Out of scope

• Other CI matrix items (UI build, e2e). Those failures, if any, are separate.

[Copilot]

Pull request overview

This PR removes racy, wall-clock-based sleeps from aisix-etcd supervisor disk-cache tests by introducing a deterministic test synchronizer that awaits spawned cache-write tasks.

Changes:

• Add pending_writes tracking for spawned flush_cache write tasks.
• Add a test-only await_pending_cache_writes() helper to deterministically await cache persistence.
• Update two flaky tests to use the new synchronizer instead of sleep(50ms).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

pending_writes retains every spawned cache-write JoinHandle for the lifetime of Supervisor, but production code never drains it. In a long-running process with frequent Put/Delete/Resync events, this will grow unbounded and keep completed tasks alive, leading to a memory leak. Consider making handle tracking #[cfg(test)] only (and drop(join) in non-test), or pruning finished handles (e.g., retain(|h| !h.is_finished())) before/after pushing new ones.

[Copilot]

await_pending_cache_writes awaits each JoinHandle but discards the result, which will silently swallow panics from cache.store(...). Since this is a test-only synchronizer, it's better to fail the test if a cache write task panics/cancels (e.g., by asserting the JoinError is OK), otherwise the test may pass/fail for the wrong reason and the underlying bug is harder to diagnose.

Suggested change

	// Failures here are not test failures — a write that
	// panicked is its own bug surfaced separately. We only
	// need the await to deterministically order against the
	// disk read that follows.
	let _ = handle.await;
	// In tests, a cache write task panic/cancellation should fail
	// loudly so the underlying bug is surfaced directly instead of
	// being masked by whatever assertion runs after the disk read.
	handle
	.await
	.expect("pending cache write task panicked or was cancelled");

[Copilot]

flush_cache now always pushes the spawned JoinHandle into pending_writes. Even if the intention is “test-only”, this path also runs in production and will accumulate handles unless drained. If you keep pending_writes in the struct, gate the push with #[cfg(test)] (and drop the handle otherwise) or drain/prune completed handles here.

Suggested change

	// Track the JoinHandle so tests can deterministically await
	// the write via [`Self::await_pending_cache_writes`] instead
	// of leaning on `tokio::time::sleep`, which under CI load
	// raced the spawn (~50ms wasn't enough on heavily loaded
	// GitHub Actions runners).
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	self.pending_writes.lock().unwrap().push(join);
	// Track the JoinHandle only in tests so they can
	// deterministically await the write via
	// [`Self::await_pending_cache_writes`] instead of leaning on
	// `tokio::time::sleep`, which under CI load raced the spawn
	// (~50ms wasn't enough on heavily loaded GitHub Actions
	// runners). In non-test builds, drop the handle immediately to
	// detach the task and avoid accumulating completed handles.
	ifletOk(rt_handle) = tokio::runtime::Handle::try_current(){
	let join =
	rt_handle.spawn(asyncmove{ cache.store(&entries, revision).await});
	#[cfg(test)]
	self.pending_writes.lock().unwrap().push(join);
	#[cfg(not(test))]
	drop(join);