FedGuard -- A Fault-Tolerant Cluster Layer for Federated Learning

Abstract: Most Federated Learning (FL) frameworks rely on a centralized client-server architecture, where the server coordinates training and aggregation. This setup introduces a single point of failure (SPOF), making the system fragile in the face of server outages and requiring manual intervention to recover. We present a leader-aware cluster layer that enhances fault tolerance in FL deployments by enabling automatic leader election and recovery. Built on Apache ZooKeeper, the system monitors node health and elects a new leader when failures occur, allowing the FL federation to reconfigure itself without downtime. Unlike fully decentralized frameworks that impose new APIs and peer-to-peer models, our approach is designed to work with existing tools like Flower, preserving their rich interfaces and ecosystem compatibility. This makes it easier to adopt fault- tolerant FL without rewriting applications. Using chaos tests, we show that running a Flower training run in our leader-aware cluster can automatically recover from different types of node failures.

Overview

The implementation of the leader-aware cluster, as well as the abstract flower application can be found under src/fedguard.

The flower project containing the network to train, as well as the implementation of the flower Client App and federation strategy are located in flwr-project.

To test the application, check out the experiment-local directory. To run the default deployment, run:

bash run.sh.

To run the chaos tests, run:

python chaos_simulation.py

from within the experiment-local directory.

Running the simulations

• If you are executing a simulation for the first time, it will take some time for the Docker images to build. This is currently not logged in the application.
• You might get errors like The container name "/node3" is already in use by container. In that case, manually remove the containers. Docker Desktop helps a lot in that case
• Simulations run 100 rounds. As such, if you want intermediary results or stats, access the /checkpoints/super_run.json of the containers.

Requirements

• Python
• Docker and Docker Compose

Tested on

• Machine: MacBook M4 Pro
• OS: macOS Sequoia 15.5
• Python: 3.9.6
• Docker: 28.0.4, build b8034c0
• Docker Compose: v2.34.0-desktop.1

To run the simulation for the leader-aware implementation, run:

python chaos_simulation_fedguard.py

To run the default client-server version, run:

python chaos_simulation_central.py

To deploy the leader-aware implementation without the chaos simulation, run:

bash ./run.sh

Development

For development, install the required Python packages:

pip install -e .