Data Balance Simulator CLI

To run Swift REPL in a docker container run:

docker run --rm -it --cap-add=SYS_PTRACE --security-opt seccomp=unconfined swift:5.10.1 swift repl

Run

To run the project in a container, then:

make run CONFIG_FILE_PATH=config-files/base_config.json SIMULATOR_ARGS=[...]

If the environment already have Swift installed (e.g. when you are developing using VSCode devcontainer feature):

make run IS_DEVCONTAINER=true CONFIG_FILE_PATH=config-files/base_config.json SIMULATOR_ARGS=[...]

---

Simulation Complexity

The number of simulations is determined by the execution parameters:

$\sum_{s = MinServices}^{MaxServices} \sum_{n = MinNodes}^{MaxNodes} \sum_{w = 1}^{min(n, MaxWindowSize)} s^{w} * (n - w + 1) + n$

An execution with:

$ nodes = 5 \newline services = 6 \newline maxWindowSize = 4 \newline $

Includes the following number of samplings:

$ winSize = 1 \to samplings = (6^{1}) * 5 + 5\newline winSize = 2 \to samplings = (6^{2}) * 4 + 5\newline winSize = 3 \to samplings = (6^{3}) * 3 + 5\newline winSize = 4 \to samplings = (6^{4}) * 2 + 5\newline $

$6^{x}$ represents the number of combinations in a window, which is multiplied by the number of windows in a simulation. After we choose the service, it is executed and the resulting dataset is stored and cached. This is the meaning of $+ n$ (one service for each node).

---

Datasets

Datasets are located in the datasets folder. The following table describes the characteristics of each dataset:

Dataset	Average of Columns entropy	Variance of Columns entropy	Std Dev of Columns entropy
high_variability	11.80	0.24	0.49
low_variability	1.7	0.2	0.45
inmates_enriched_10k	5.35	13.09	3.62
IBM_HR_Analytics_employee_attrition	3.13	8.56	2.93
red_wine_quality	5.61	2.01	1.42
avocado	9.36	22.13	4.7

To compute the entropy of each column:

importpandasaspdimportnumpyasnpfromtypingimportDictdataset=pd.read_csv(dataset_name+".csv")
dataset_size=len(dataset)
defget_column_frequency(column: pd.Series) ->pd.Series:
returncolumn.value_counts()
defget_column_probability(column: pd.Series) ->pd.Series:
returncolumn.value_counts(normalize=True)
defget_column_entropy(column: pd.Series) ->float:
column_probability=get_column_probability(column)
return-sum(column_probability*np.log2(column_probability))
entropies= [get_column_entropy(dataset[column]) forcolumnindataset.columns ]
print(f"{round(np.mean(entropies), 2)}, {round(np.var(entropies), 2)}, {round(np.std(entropies), 2)}")

---

Logging

To set the logger level, create an env variable called LOGGER_LEVEL with one of the following values: trace, debug, info, notice, warning, error, critical ( default is info). The alternative is to pass this variable to make run.

---

DB Migrations and DB queries

For DB migration, run make migrate-db SQL_CODE="your_migration_sql".

To run queries on DB, run make run-query SQL_CODE="your_plain_sql".

---

K8s

Inside the k8s/ folder, there are all the resources to run a simulation on k8s. After cd k8s, here are some Makefile recipees:

• install: deploy the setup resources
• uninstall: uninstall the setup resources
• run-simulation: run a simulation. Example:

  make run-simulation NAME=test-sim VALUES_FILE=./run-simulation/files/base-params.yaml`

• delete-simulation: uninstall the resources created for the simulation
• copy-dataset: copy a dataset into the volume used by running a simulation. Example:

  make copy-dataset FILE_PATH=path/to/dataset.csv

• query-db: open a sqlite connection with the db specified (defaults to simulations.db). Example:

  make query-db DB_PATH=simulations.db

---

Deepnote experiments