fix: prevent Docker network resource exhaustion with improved cleanup by ammario · Pull Request #41 · coder/httpjail

ammario · 2025-09-13T16:37:56Z

Problem

CI instance experiencing network interface pool exhaustion due to orphaned Docker networks accumulating over time. Investigation found 105+ stale networks.

Additionally identified a race condition where concurrent jail initialization could delete networks before their canary files were created.

Solution

Comprehensive orphan cleanup: Added cleanup_all_orphaned_docker_networks() to remove networks without canary files
Race condition fix: Create canary files BEFORE network setup to prevent concurrent deletion
Cleanup ordering: Ensure canaries are deleted LAST, only after successful cleanup
Persistent canaries: Moved from /tmp to user data directory to survive reboots
DRY refactoring: Extracted common patterns to reduce code duplication

Key Changes

Cleanup Improvements

Scan and remove orphaned Docker networks at jail setup
Only delete canary files after successful resource cleanup
Move canary files to ~/.local/share/httpjail/canaries/ for persistence

Race Condition Fix

Create canary BEFORE jail.setup() to protect resources during creation
Prevents concurrent jails from deleting each other's networks

Code Quality (DRY)

Extract network/table name generation with constants
Add is_not_found_error() and is_already_exists_error() helpers
Consolidate Docker flag handling with FLAGS_WITH_VALUES constant
Reduce code duplication while improving maintainability

Testing

Verified cleanup of 105 orphaned networks on ci-1
Builds successfully, passes clippy
Prevents future accumulation with proactive cleanup

Fixes network interface exhaustion on CI and production systems.

chatgpt-codex-connector

Codex Review: Here are some suggestions.

Reply with @codex fix comments to fix any unresolved comments.

About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".

Issues fixed: 1. Docker networks accumulating without cleanup (105+ orphaned networks on CI) 2. Race condition: concurrent jails deleting each other's networks during setup 3. Canary files in /tmp being cleared by system, breaking orphan detection 4. Canary files deleted even when cleanup fails, losing track of orphaned resources Critical race condition fix: - Create canary BEFORE jail.setup() to prevent concurrent deletion - Previously: Jail A creates network → Jail B runs cleanup → sees A's network has no canary → deletes it - Now: Canary created first, so network is protected during setup - Delete canary if setup fails to avoid orphaned canaries Other improvements: - Move canary files from /tmp to user data dir (~/.local/share/httpjail/canaries) - Prevents system tmp cleaners from removing canaries - Ensures canaries persist across reboots - Fix cleanup order in ManagedJail::drop() - Explicitly call jail.cleanup() BEFORE deleting canary - Only delete canary if cleanup succeeds - Leave canary for orphan cleanup if cleanup fails - Add cleanup_all_orphaned_docker_networks() to DockerLinux - Scans for and removes Docker networks without canary files - Called at setup to handle networks orphaned by crashes/kills The improved cleanup strategy: 1. On setup: Create canary, clean up orphaned networks, then create new network 2. On normal exit: Clean up resources, then delete canary only if successful 3. On cleanup failure: Leave canary so orphan cleanup can retry later 4. On orphan detection: Clean up resources for stale canaries This prevents network interface pool exhaustion and race conditions in CI/production.

…eanup

- Extract common network name generation with NETWORK_PREFIX constant - Add helper methods for jail ID extraction and error detection - Consolidate error checking patterns with is_not_found_error() and is_already_exists_error() - Extract Docker flag constants and image index finding logic - Reduce code duplication from ~540 lines to cleaner abstractions - Improve maintainability with centralized string patterns

ammario · 2025-09-13T16:37:56Z

Problem

CI instance experiencing network interface pool exhaustion due to orphaned Docker networks accumulating over time. Investigation found 105+ stale networks.

Additionally identified a race condition where concurrent jail initialization could delete networks before their canary files were created.

Solution

Comprehensive orphan cleanup: Added cleanup_all_orphaned_docker_networks() to remove networks without canary files
Race condition fix: Create canary files BEFORE network setup to prevent concurrent deletion
Cleanup ordering: Ensure canaries are deleted LAST, only after successful cleanup
Persistent canaries: Moved from /tmp to user data directory to survive reboots
DRY refactoring: Extracted common patterns to reduce code duplication

Key Changes

Cleanup Improvements

Scan and remove orphaned Docker networks at jail setup
Only delete canary files after successful resource cleanup
Move canary files to ~/.local/share/httpjail/canaries/ for persistence

Race Condition Fix

Create canary BEFORE jail.setup() to protect resources during creation
Prevents concurrent jails from deleting each other's networks

Code Quality (DRY)

Extract network/table name generation with constants
Add is_not_found_error() and is_already_exists_error() helpers
Consolidate Docker flag handling with FLAGS_WITH_VALUES constant
Reduce code duplication while improving maintainability

Testing

Verified cleanup of 105 orphaned networks on ci-1
Builds successfully, passes clippy
Prevents future accumulation with proactive cleanup

Fixes network interface exhaustion on CI and production systems.

chatgpt-codex-connector

Codex Review: Here are some suggestions.

Reply with @codex fix comments to fix any unresolved comments.

About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".

Issues fixed: 1. Docker networks accumulating without cleanup (105+ orphaned networks on CI) 2. Race condition: concurrent jails deleting each other's networks during setup 3. Canary files in /tmp being cleared by system, breaking orphan detection 4. Canary files deleted even when cleanup fails, losing track of orphaned resources Critical race condition fix: - Create canary BEFORE jail.setup() to prevent concurrent deletion - Previously: Jail A creates network → Jail B runs cleanup → sees A's network has no canary → deletes it - Now: Canary created first, so network is protected during setup - Delete canary if setup fails to avoid orphaned canaries Other improvements: - Move canary files from /tmp to user data dir (~/.local/share/httpjail/canaries) - Prevents system tmp cleaners from removing canaries - Ensures canaries persist across reboots - Fix cleanup order in ManagedJail::drop() - Explicitly call jail.cleanup() BEFORE deleting canary - Only delete canary if cleanup succeeds - Leave canary for orphan cleanup if cleanup fails - Add cleanup_all_orphaned_docker_networks() to DockerLinux - Scans for and removes Docker networks without canary files - Called at setup to handle networks orphaned by crashes/kills The improved cleanup strategy: 1. On setup: Create canary, clean up orphaned networks, then create new network 2. On normal exit: Clean up resources, then delete canary only if successful 3. On cleanup failure: Leave canary so orphan cleanup can retry later 4. On orphan detection: Clean up resources for stale canaries This prevents network interface pool exhaustion and race conditions in CI/production.

…eanup

- Extract common network name generation with NETWORK_PREFIX constant - Add helper methods for jail ID extraction and error detection - Consolidate error checking patterns with is_not_found_error() and is_already_exists_error() - Extract Docker flag constants and image index finding logic - Reduce code duplication from ~540 lines to cleaner abstractions - Improve maintainability with centralized string patterns

ammario · 2025-09-13T16:37:56Z

Problem

CI instance experiencing network interface pool exhaustion due to orphaned Docker networks accumulating over time. Investigation found 105+ stale networks.

Additionally identified a race condition where concurrent jail initialization could delete networks before their canary files were created.

Solution

Comprehensive orphan cleanup: Added cleanup_all_orphaned_docker_networks() to remove networks without canary files
Race condition fix: Create canary files BEFORE network setup to prevent concurrent deletion
Cleanup ordering: Ensure canaries are deleted LAST, only after successful cleanup
Persistent canaries: Moved from /tmp to user data directory to survive reboots
DRY refactoring: Extracted common patterns to reduce code duplication

Key Changes

Cleanup Improvements

Scan and remove orphaned Docker networks at jail setup
Only delete canary files after successful resource cleanup
Move canary files to ~/.local/share/httpjail/canaries/ for persistence

Race Condition Fix

Create canary BEFORE jail.setup() to protect resources during creation
Prevents concurrent jails from deleting each other's networks

Code Quality (DRY)

Extract network/table name generation with constants
Add is_not_found_error() and is_already_exists_error() helpers
Consolidate Docker flag handling with FLAGS_WITH_VALUES constant
Reduce code duplication while improving maintainability

Testing

Verified cleanup of 105 orphaned networks on ci-1
Builds successfully, passes clippy
Prevents future accumulation with proactive cleanup

Fixes network interface exhaustion on CI and production systems.

chatgpt-codex-connector

Codex Review: Here are some suggestions.

Reply with @codex fix comments to fix any unresolved comments.

About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".

Issues fixed: 1. Docker networks accumulating without cleanup (105+ orphaned networks on CI) 2. Race condition: concurrent jails deleting each other's networks during setup 3. Canary files in /tmp being cleared by system, breaking orphan detection 4. Canary files deleted even when cleanup fails, losing track of orphaned resources Critical race condition fix: - Create canary BEFORE jail.setup() to prevent concurrent deletion - Previously: Jail A creates network → Jail B runs cleanup → sees A's network has no canary → deletes it - Now: Canary created first, so network is protected during setup - Delete canary if setup fails to avoid orphaned canaries Other improvements: - Move canary files from /tmp to user data dir (~/.local/share/httpjail/canaries) - Prevents system tmp cleaners from removing canaries - Ensures canaries persist across reboots - Fix cleanup order in ManagedJail::drop() - Explicitly call jail.cleanup() BEFORE deleting canary - Only delete canary if cleanup succeeds - Leave canary for orphan cleanup if cleanup fails - Add cleanup_all_orphaned_docker_networks() to DockerLinux - Scans for and removes Docker networks without canary files - Called at setup to handle networks orphaned by crashes/kills The improved cleanup strategy: 1. On setup: Create canary, clean up orphaned networks, then create new network 2. On normal exit: Clean up resources, then delete canary only if successful 3. On cleanup failure: Leave canary so orphan cleanup can retry later 4. On orphan detection: Clean up resources for stale canaries This prevents network interface pool exhaustion and race conditions in CI/production.

…eanup

- Extract common network name generation with NETWORK_PREFIX constant - Add helper methods for jail ID extraction and error detection - Consolidate error checking patterns with is_not_found_error() and is_already_exists_error() - Extract Docker flag constants and image index finding logic - Reduce code duplication from ~540 lines to cleaner abstractions - Improve maintainability with centralized string patterns

ammario · 2025-09-13T16:37:56Z

Problem

CI instance experiencing network interface pool exhaustion due to orphaned Docker networks accumulating over time. Investigation found 105+ stale networks.

Additionally identified a race condition where concurrent jail initialization could delete networks before their canary files were created.

Solution

Comprehensive orphan cleanup: Added cleanup_all_orphaned_docker_networks() to remove networks without canary files
Race condition fix: Create canary files BEFORE network setup to prevent concurrent deletion
Cleanup ordering: Ensure canaries are deleted LAST, only after successful cleanup
Persistent canaries: Moved from /tmp to user data directory to survive reboots
DRY refactoring: Extracted common patterns to reduce code duplication

Key Changes

Cleanup Improvements

Scan and remove orphaned Docker networks at jail setup
Only delete canary files after successful resource cleanup
Move canary files to ~/.local/share/httpjail/canaries/ for persistence

Race Condition Fix

Create canary BEFORE jail.setup() to protect resources during creation
Prevents concurrent jails from deleting each other's networks

Code Quality (DRY)

Extract network/table name generation with constants
Add is_not_found_error() and is_already_exists_error() helpers
Consolidate Docker flag handling with FLAGS_WITH_VALUES constant
Reduce code duplication while improving maintainability

Testing

Verified cleanup of 105 orphaned networks on ci-1
Builds successfully, passes clippy
Prevents future accumulation with proactive cleanup

Fixes network interface exhaustion on CI and production systems.

chatgpt-codex-connector

Codex Review: Here are some suggestions.

Reply with @codex fix comments to fix any unresolved comments.

About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".

Issues fixed: 1. Docker networks accumulating without cleanup (105+ orphaned networks on CI) 2. Race condition: concurrent jails deleting each other's networks during setup 3. Canary files in /tmp being cleared by system, breaking orphan detection 4. Canary files deleted even when cleanup fails, losing track of orphaned resources Critical race condition fix: - Create canary BEFORE jail.setup() to prevent concurrent deletion - Previously: Jail A creates network → Jail B runs cleanup → sees A's network has no canary → deletes it - Now: Canary created first, so network is protected during setup - Delete canary if setup fails to avoid orphaned canaries Other improvements: - Move canary files from /tmp to user data dir (~/.local/share/httpjail/canaries) - Prevents system tmp cleaners from removing canaries - Ensures canaries persist across reboots - Fix cleanup order in ManagedJail::drop() - Explicitly call jail.cleanup() BEFORE deleting canary - Only delete canary if cleanup succeeds - Leave canary for orphan cleanup if cleanup fails - Add cleanup_all_orphaned_docker_networks() to DockerLinux - Scans for and removes Docker networks without canary files - Called at setup to handle networks orphaned by crashes/kills The improved cleanup strategy: 1. On setup: Create canary, clean up orphaned networks, then create new network 2. On normal exit: Clean up resources, then delete canary only if successful 3. On cleanup failure: Leave canary so orphan cleanup can retry later 4. On orphan detection: Clean up resources for stale canaries This prevents network interface pool exhaustion and race conditions in CI/production.

…eanup

- Extract common network name generation with NETWORK_PREFIX constant - Add helper methods for jail ID extraction and error detection - Consolidate error checking patterns with is_not_found_error() and is_already_exists_error() - Extract Docker flag constants and image index finding logic - Reduce code duplication from ~540 lines to cleaner abstractions - Improve maintainability with centralized string patterns

ammario · 2025-09-13T16:37:56Z

Problem

CI instance experiencing network interface pool exhaustion due to orphaned Docker networks accumulating over time. Investigation found 105+ stale networks.

Additionally identified a race condition where concurrent jail initialization could delete networks before their canary files were created.

Solution

Comprehensive orphan cleanup: Added cleanup_all_orphaned_docker_networks() to remove networks without canary files
Race condition fix: Create canary files BEFORE network setup to prevent concurrent deletion
Cleanup ordering: Ensure canaries are deleted LAST, only after successful cleanup
Persistent canaries: Moved from /tmp to user data directory to survive reboots
DRY refactoring: Extracted common patterns to reduce code duplication

Key Changes

Cleanup Improvements

Scan and remove orphaned Docker networks at jail setup
Only delete canary files after successful resource cleanup
Move canary files to ~/.local/share/httpjail/canaries/ for persistence

Race Condition Fix

Create canary BEFORE jail.setup() to protect resources during creation
Prevents concurrent jails from deleting each other's networks

Code Quality (DRY)

Extract network/table name generation with constants
Add is_not_found_error() and is_already_exists_error() helpers
Consolidate Docker flag handling with FLAGS_WITH_VALUES constant
Reduce code duplication while improving maintainability

Testing

Verified cleanup of 105 orphaned networks on ci-1
Builds successfully, passes clippy
Prevents future accumulation with proactive cleanup

Fixes network interface exhaustion on CI and production systems.

chatgpt-codex-connector

Codex Review: Here are some suggestions.

Reply with @codex fix comments to fix any unresolved comments.

About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".

Issues fixed: 1. Docker networks accumulating without cleanup (105+ orphaned networks on CI) 2. Race condition: concurrent jails deleting each other's networks during setup 3. Canary files in /tmp being cleared by system, breaking orphan detection 4. Canary files deleted even when cleanup fails, losing track of orphaned resources Critical race condition fix: - Create canary BEFORE jail.setup() to prevent concurrent deletion - Previously: Jail A creates network → Jail B runs cleanup → sees A's network has no canary → deletes it - Now: Canary created first, so network is protected during setup - Delete canary if setup fails to avoid orphaned canaries Other improvements: - Move canary files from /tmp to user data dir (~/.local/share/httpjail/canaries) - Prevents system tmp cleaners from removing canaries - Ensures canaries persist across reboots - Fix cleanup order in ManagedJail::drop() - Explicitly call jail.cleanup() BEFORE deleting canary - Only delete canary if cleanup succeeds - Leave canary for orphan cleanup if cleanup fails - Add cleanup_all_orphaned_docker_networks() to DockerLinux - Scans for and removes Docker networks without canary files - Called at setup to handle networks orphaned by crashes/kills The improved cleanup strategy: 1. On setup: Create canary, clean up orphaned networks, then create new network 2. On normal exit: Clean up resources, then delete canary only if successful 3. On cleanup failure: Leave canary so orphan cleanup can retry later 4. On orphan detection: Clean up resources for stale canaries This prevents network interface pool exhaustion and race conditions in CI/production.

…eanup

- Extract common network name generation with NETWORK_PREFIX constant - Add helper methods for jail ID extraction and error detection - Consolidate error checking patterns with is_not_found_error() and is_already_exists_error() - Extract Docker flag constants and image index finding logic - Reduce code duplication from ~540 lines to cleaner abstractions - Improve maintainability with centralized string patterns

ammario · 2025-09-13T16:37:56Z

Problem

CI instance experiencing network interface pool exhaustion due to orphaned Docker networks accumulating over time. Investigation found 105+ stale networks.

Additionally identified a race condition where concurrent jail initialization could delete networks before their canary files were created.

Solution

Comprehensive orphan cleanup: Added cleanup_all_orphaned_docker_networks() to remove networks without canary files
Race condition fix: Create canary files BEFORE network setup to prevent concurrent deletion
Cleanup ordering: Ensure canaries are deleted LAST, only after successful cleanup
Persistent canaries: Moved from /tmp to user data directory to survive reboots
DRY refactoring: Extracted common patterns to reduce code duplication

Key Changes

Cleanup Improvements

Scan and remove orphaned Docker networks at jail setup
Only delete canary files after successful resource cleanup
Move canary files to ~/.local/share/httpjail/canaries/ for persistence

Race Condition Fix

Create canary BEFORE jail.setup() to protect resources during creation
Prevents concurrent jails from deleting each other's networks

Code Quality (DRY)

Extract network/table name generation with constants
Add is_not_found_error() and is_already_exists_error() helpers
Consolidate Docker flag handling with FLAGS_WITH_VALUES constant
Reduce code duplication while improving maintainability

Testing

Verified cleanup of 105 orphaned networks on ci-1
Builds successfully, passes clippy
Prevents future accumulation with proactive cleanup

Fixes network interface exhaustion on CI and production systems.

chatgpt-codex-connector

Codex Review: Here are some suggestions.

Reply with @codex fix comments to fix any unresolved comments.

About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".

Issues fixed: 1. Docker networks accumulating without cleanup (105+ orphaned networks on CI) 2. Race condition: concurrent jails deleting each other's networks during setup 3. Canary files in /tmp being cleared by system, breaking orphan detection 4. Canary files deleted even when cleanup fails, losing track of orphaned resources Critical race condition fix: - Create canary BEFORE jail.setup() to prevent concurrent deletion - Previously: Jail A creates network → Jail B runs cleanup → sees A's network has no canary → deletes it - Now: Canary created first, so network is protected during setup - Delete canary if setup fails to avoid orphaned canaries Other improvements: - Move canary files from /tmp to user data dir (~/.local/share/httpjail/canaries) - Prevents system tmp cleaners from removing canaries - Ensures canaries persist across reboots - Fix cleanup order in ManagedJail::drop() - Explicitly call jail.cleanup() BEFORE deleting canary - Only delete canary if cleanup succeeds - Leave canary for orphan cleanup if cleanup fails - Add cleanup_all_orphaned_docker_networks() to DockerLinux - Scans for and removes Docker networks without canary files - Called at setup to handle networks orphaned by crashes/kills The improved cleanup strategy: 1. On setup: Create canary, clean up orphaned networks, then create new network 2. On normal exit: Clean up resources, then delete canary only if successful 3. On cleanup failure: Leave canary so orphan cleanup can retry later 4. On orphan detection: Clean up resources for stale canaries This prevents network interface pool exhaustion and race conditions in CI/production.

…eanup

- Extract common network name generation with NETWORK_PREFIX constant - Add helper methods for jail ID extraction and error detection - Consolidate error checking patterns with is_not_found_error() and is_already_exists_error() - Extract Docker flag constants and image index finding logic - Reduce code duplication from ~540 lines to cleaner abstractions - Improve maintainability with centralized string patterns

ammario · 2025-09-13T16:37:56Z

Problem

CI instance experiencing network interface pool exhaustion due to orphaned Docker networks accumulating over time. Investigation found 105+ stale networks.

Additionally identified a race condition where concurrent jail initialization could delete networks before their canary files were created.

Solution

Comprehensive orphan cleanup: Added cleanup_all_orphaned_docker_networks() to remove networks without canary files
Race condition fix: Create canary files BEFORE network setup to prevent concurrent deletion
Cleanup ordering: Ensure canaries are deleted LAST, only after successful cleanup
Persistent canaries: Moved from /tmp to user data directory to survive reboots
DRY refactoring: Extracted common patterns to reduce code duplication

Key Changes

Cleanup Improvements

Scan and remove orphaned Docker networks at jail setup
Only delete canary files after successful resource cleanup
Move canary files to ~/.local/share/httpjail/canaries/ for persistence

Race Condition Fix

Create canary BEFORE jail.setup() to protect resources during creation
Prevents concurrent jails from deleting each other's networks

Code Quality (DRY)

Extract network/table name generation with constants
Add is_not_found_error() and is_already_exists_error() helpers
Consolidate Docker flag handling with FLAGS_WITH_VALUES constant
Reduce code duplication while improving maintainability

Testing

Verified cleanup of 105 orphaned networks on ci-1
Builds successfully, passes clippy
Prevents future accumulation with proactive cleanup

Fixes network interface exhaustion on CI and production systems.

chatgpt-codex-connector

Codex Review: Here are some suggestions.

Reply with @codex fix comments to fix any unresolved comments.

About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".

Issues fixed: 1. Docker networks accumulating without cleanup (105+ orphaned networks on CI) 2. Race condition: concurrent jails deleting each other's networks during setup 3. Canary files in /tmp being cleared by system, breaking orphan detection 4. Canary files deleted even when cleanup fails, losing track of orphaned resources Critical race condition fix: - Create canary BEFORE jail.setup() to prevent concurrent deletion - Previously: Jail A creates network → Jail B runs cleanup → sees A's network has no canary → deletes it - Now: Canary created first, so network is protected during setup - Delete canary if setup fails to avoid orphaned canaries Other improvements: - Move canary files from /tmp to user data dir (~/.local/share/httpjail/canaries) - Prevents system tmp cleaners from removing canaries - Ensures canaries persist across reboots - Fix cleanup order in ManagedJail::drop() - Explicitly call jail.cleanup() BEFORE deleting canary - Only delete canary if cleanup succeeds - Leave canary for orphan cleanup if cleanup fails - Add cleanup_all_orphaned_docker_networks() to DockerLinux - Scans for and removes Docker networks without canary files - Called at setup to handle networks orphaned by crashes/kills The improved cleanup strategy: 1. On setup: Create canary, clean up orphaned networks, then create new network 2. On normal exit: Clean up resources, then delete canary only if successful 3. On cleanup failure: Leave canary so orphan cleanup can retry later 4. On orphan detection: Clean up resources for stale canaries This prevents network interface pool exhaustion and race conditions in CI/production.

…eanup

- Extract common network name generation with NETWORK_PREFIX constant - Add helper methods for jail ID extraction and error detection - Consolidate error checking patterns with is_not_found_error() and is_already_exists_error() - Extract Docker flag constants and image index finding logic - Reduce code duplication from ~540 lines to cleaner abstractions - Improve maintainability with centralized string patterns

ammario · 2025-09-13T16:37:56Z

Problem

CI instance experiencing network interface pool exhaustion due to orphaned Docker networks accumulating over time. Investigation found 105+ stale networks.

Additionally identified a race condition where concurrent jail initialization could delete networks before their canary files were created.

Solution

Comprehensive orphan cleanup: Added cleanup_all_orphaned_docker_networks() to remove networks without canary files
Race condition fix: Create canary files BEFORE network setup to prevent concurrent deletion
Cleanup ordering: Ensure canaries are deleted LAST, only after successful cleanup
Persistent canaries: Moved from /tmp to user data directory to survive reboots
DRY refactoring: Extracted common patterns to reduce code duplication

Key Changes

Cleanup Improvements

Scan and remove orphaned Docker networks at jail setup
Only delete canary files after successful resource cleanup
Move canary files to ~/.local/share/httpjail/canaries/ for persistence

Race Condition Fix

Create canary BEFORE jail.setup() to protect resources during creation
Prevents concurrent jails from deleting each other's networks

Code Quality (DRY)

Extract network/table name generation with constants
Add is_not_found_error() and is_already_exists_error() helpers
Consolidate Docker flag handling with FLAGS_WITH_VALUES constant
Reduce code duplication while improving maintainability

Testing

Verified cleanup of 105 orphaned networks on ci-1
Builds successfully, passes clippy
Prevents future accumulation with proactive cleanup

Fixes network interface exhaustion on CI and production systems.

chatgpt-codex-connector

Codex Review: Here are some suggestions.

Reply with @codex fix comments to fix any unresolved comments.

About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you open a pull request for review, mark a draft as ready, or comment "@codex review". If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex fix this CI failure" or "@codex address that feedback".

Issues fixed: 1. Docker networks accumulating without cleanup (105+ orphaned networks on CI) 2. Race condition: concurrent jails deleting each other's networks during setup 3. Canary files in /tmp being cleared by system, breaking orphan detection 4. Canary files deleted even when cleanup fails, losing track of orphaned resources Critical race condition fix: - Create canary BEFORE jail.setup() to prevent concurrent deletion - Previously: Jail A creates network → Jail B runs cleanup → sees A's network has no canary → deletes it - Now: Canary created first, so network is protected during setup - Delete canary if setup fails to avoid orphaned canaries Other improvements: - Move canary files from /tmp to user data dir (~/.local/share/httpjail/canaries) - Prevents system tmp cleaners from removing canaries - Ensures canaries persist across reboots - Fix cleanup order in ManagedJail::drop() - Explicitly call jail.cleanup() BEFORE deleting canary - Only delete canary if cleanup succeeds - Leave canary for orphan cleanup if cleanup fails - Add cleanup_all_orphaned_docker_networks() to DockerLinux - Scans for and removes Docker networks without canary files - Called at setup to handle networks orphaned by crashes/kills The improved cleanup strategy: 1. On setup: Create canary, clean up orphaned networks, then create new network 2. On normal exit: Clean up resources, then delete canary only if successful 3. On cleanup failure: Leave canary so orphan cleanup can retry later 4. On orphan detection: Clean up resources for stale canaries This prevents network interface pool exhaustion and race conditions in CI/production.

…eanup

- Extract common network name generation with NETWORK_PREFIX constant - Add helper methods for jail ID extraction and error detection - Consolidate error checking patterns with is_not_found_error() and is_already_exists_error() - Extract Docker flag constants and image index finding logic - Reduce code duplication from ~540 lines to cleaner abstractions - Improve maintainability with centralized string patterns

chatgpt-codex-connectorBot reviewed Sep 13, 2025
View reviewed changes

Comment threadsrc/jail/linux/docker.rs

ammario force-pushed the fix-docker-network-cleanup branch 2 times, most recently from 038382b to 285e85eCompare September 13, 2025 16:46

ammario force-pushed the fix-docker-network-cleanup branch from 696e0e7 to 145b617Compare September 13, 2025 16:50

ammario added 3 commits September 13, 2025 11:51

Merge remote-tracking branch 'origin/main' into fix-docker-network-cl…
b752f52
…eanup

add pr-compress
6032a10

ammario changed the title ~~fix: improve Docker network cleanup to prevent resource exhaustion~~fix: prevent Docker network resource exhaustion with improved cleanupSep 13, 2025

ammario merged commit 542eece into mainSep 13, 2025
6 checks passed

ammario deleted the fix-docker-network-cleanup branch September 13, 2025 16:58

chatgpt-codex-connectorBot reviewed Sep 13, 2025
View reviewed changes

Comment threadsrc/jail/linux/docker.rs

ammario force-pushed the fix-docker-network-cleanup branch 2 times, most recently from 038382b to 285e85eCompare September 13, 2025 16:46

ammario force-pushed the fix-docker-network-cleanup branch from 696e0e7 to 145b617Compare September 13, 2025 16:50

ammario added 3 commits September 13, 2025 11:51

Merge remote-tracking branch 'origin/main' into fix-docker-network-cl…
b752f52
…eanup

add pr-compress
6032a10

ammario changed the title ~~fix: improve Docker network cleanup to prevent resource exhaustion~~fix: prevent Docker network resource exhaustion with improved cleanupSep 13, 2025

ammario merged commit 542eece into mainSep 13, 2025
6 checks passed

ammario deleted the fix-docker-network-cleanup branch September 13, 2025 16:58

chatgpt-codex-connectorBot reviewed Sep 13, 2025
View reviewed changes

Comment threadsrc/jail/linux/docker.rs

ammario force-pushed the fix-docker-network-cleanup branch 2 times, most recently from 038382b to 285e85eCompare September 13, 2025 16:46

ammario force-pushed the fix-docker-network-cleanup branch from 696e0e7 to 145b617Compare September 13, 2025 16:50

ammario added 3 commits September 13, 2025 11:51

Merge remote-tracking branch 'origin/main' into fix-docker-network-cl…
b752f52
…eanup

add pr-compress
6032a10

ammario changed the title ~~fix: improve Docker network cleanup to prevent resource exhaustion~~fix: prevent Docker network resource exhaustion with improved cleanupSep 13, 2025

ammario merged commit 542eece into mainSep 13, 2025
6 checks passed

ammario deleted the fix-docker-network-cleanup branch September 13, 2025 16:58

chatgpt-codex-connectorBot reviewed Sep 13, 2025
View reviewed changes

Comment threadsrc/jail/linux/docker.rs

ammario force-pushed the fix-docker-network-cleanup branch 2 times, most recently from 038382b to 285e85eCompare September 13, 2025 16:46

ammario force-pushed the fix-docker-network-cleanup branch from 696e0e7 to 145b617Compare September 13, 2025 16:50

ammario added 3 commits September 13, 2025 11:51

Merge remote-tracking branch 'origin/main' into fix-docker-network-cl…
b752f52
…eanup

add pr-compress
6032a10

ammario changed the title ~~fix: improve Docker network cleanup to prevent resource exhaustion~~fix: prevent Docker network resource exhaustion with improved cleanupSep 13, 2025

ammario merged commit 542eece into mainSep 13, 2025
6 checks passed

ammario deleted the fix-docker-network-cleanup branch September 13, 2025 16:58

chatgpt-codex-connectorBot reviewed Sep 13, 2025
View reviewed changes

Comment threadsrc/jail/linux/docker.rs

ammario force-pushed the fix-docker-network-cleanup branch 2 times, most recently from 038382b to 285e85eCompare September 13, 2025 16:46

ammario force-pushed the fix-docker-network-cleanup branch from 696e0e7 to 145b617Compare September 13, 2025 16:50

ammario added 3 commits September 13, 2025 11:51

Merge remote-tracking branch 'origin/main' into fix-docker-network-cl…
b752f52
…eanup

add pr-compress
6032a10

ammario changed the title ~~fix: improve Docker network cleanup to prevent resource exhaustion~~fix: prevent Docker network resource exhaustion with improved cleanupSep 13, 2025

ammario merged commit 542eece into mainSep 13, 2025
6 checks passed

ammario deleted the fix-docker-network-cleanup branch September 13, 2025 16:58

chatgpt-codex-connectorBot reviewed Sep 13, 2025
View reviewed changes

Comment threadsrc/jail/linux/docker.rs

ammario force-pushed the fix-docker-network-cleanup branch 2 times, most recently from 038382b to 285e85eCompare September 13, 2025 16:46

ammario force-pushed the fix-docker-network-cleanup branch from 696e0e7 to 145b617Compare September 13, 2025 16:50

ammario added 3 commits September 13, 2025 11:51

Merge remote-tracking branch 'origin/main' into fix-docker-network-cl…
b752f52
…eanup

add pr-compress
6032a10

ammario changed the title ~~fix: improve Docker network cleanup to prevent resource exhaustion~~fix: prevent Docker network resource exhaustion with improved cleanupSep 13, 2025

ammario merged commit 542eece into mainSep 13, 2025
6 checks passed

ammario deleted the fix-docker-network-cleanup branch September 13, 2025 16:58

chatgpt-codex-connectorBot reviewed Sep 13, 2025
View reviewed changes

Comment threadsrc/jail/linux/docker.rs

ammario force-pushed the fix-docker-network-cleanup branch 2 times, most recently from 038382b to 285e85eCompare September 13, 2025 16:46

ammario force-pushed the fix-docker-network-cleanup branch from 696e0e7 to 145b617Compare September 13, 2025 16:50

ammario added 3 commits September 13, 2025 11:51

Merge remote-tracking branch 'origin/main' into fix-docker-network-cl…
b752f52
…eanup

add pr-compress
6032a10

ammario changed the title ~~fix: improve Docker network cleanup to prevent resource exhaustion~~fix: prevent Docker network resource exhaustion with improved cleanupSep 13, 2025

ammario merged commit 542eece into mainSep 13, 2025
6 checks passed

ammario deleted the fix-docker-network-cleanup branch September 13, 2025 16:58

chatgpt-codex-connectorBot reviewed Sep 13, 2025
View reviewed changes

Comment threadsrc/jail/linux/docker.rs

ammario force-pushed the fix-docker-network-cleanup branch 2 times, most recently from 038382b to 285e85eCompare September 13, 2025 16:46

ammario force-pushed the fix-docker-network-cleanup branch from 696e0e7 to 145b617Compare September 13, 2025 16:50

ammario added 3 commits September 13, 2025 11:51

Merge remote-tracking branch 'origin/main' into fix-docker-network-cl…
b752f52
…eanup

add pr-compress
6032a10

ammario changed the title ~~fix: improve Docker network cleanup to prevent resource exhaustion~~fix: prevent Docker network resource exhaustion with improved cleanupSep 13, 2025

ammario merged commit 542eece into mainSep 13, 2025
6 checks passed

ammario deleted the fix-docker-network-cleanup branch September 13, 2025 16:58

Uh oh!

Conversation

ammario commented Sep 13, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Key Changes

Cleanup Improvements

Race Condition Fix

Code Quality (DRY)

Testing

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ammario commented Sep 13, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Key Changes

Cleanup Improvements

Race Condition Fix

Code Quality (DRY)

Testing

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ammario commented Sep 13, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Key Changes

Cleanup Improvements

Race Condition Fix

Code Quality (DRY)

Testing

Uh oh!

chatgpt-codex-connectorBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ammario commented Sep 13, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

ammario commented Sep 13, 2025•
edited
Loading

ammario commented Sep 13, 2025•
edited
Loading

ammario commented Sep 13, 2025•
edited
Loading

ammario commented Sep 13, 2025•
edited
Loading

ammario commented Sep 13, 2025•
edited
Loading