This Projects contains all parts of the UpStream scaling framework. It makes use of domain specific metrics from the Flink application cluster to make a scaling decision using MPC.
Tip
This repo makes use of git submodules. To clone with all included submodules at once you can run:
git clone --recurse-submodules git@github.com:DPS-Stream-Processing-Applications/UpStream.gitThis project contains a flake.nix file to manage all the dependencies that would usually need to be installed through a regular package manager
with the Nix package manager instead. The Nix Shell will provide an installation of Flink as well as the OpenJDK needed for the Gradle wrapper to work.
To install Nix follow the official instructions for your platform.
Following this, you need to enable flakes and nix-command for the Nix package manager that you just installed.
The "Other Distros, without Home-Manager" section of the Flake Wiki will explain how to do this.
Tip
If the ~/.config/nix folder and nix.conf file do not already exist after installing, you need to create them manually.
After successfully installing Nix and enabling Flakes, you will be able to use the nix develop command in the root of the project to enter a
development shell managed by Nix. To exit the dev shell, use the exit command or hit Ctrl+d.
Note
This step is entirely optional but may improve your development experience.
Using Direnv will allow you to automatically launch the Nix devShell whenever you change into the project directory.
Direnv can also be detected by your IDE if a plugin exists.
You might also want to install nix-direnv to improve the Direnv experience with Nix.
For all the applications in this repository external resources are required. These resources are all managed within a Kubernetes cluster.
Important
Make sure you have docker installed on your system before reading further.
The nix dev shell provides k3d as means to spin up a local Kubernetes cluster.
Use the k3d-cluster-config.yaml file of this project to set up a preconfigured test-cluster.
See the following guide on how to use a config file.
Follow the quick start guide to set up an empty cluster.
This project uses the Chameleon test bed more specificaly CHI@TACC to test and evaluate the application.
The experimental setup requires 1 Jump Host as well as 3 cluster nodes.
Required resources: 1 floating IP, One instance of type skylake, as well as three instances of a cascadelake node.
To provision instances on Chameleon Cloud you first have to create a lease.
After that you can launch the instances.
Make sure you use the official Chameleon Cloud version of the Ubuntu 22.04 image for consistency.
Tip
regarding ssh access to the host, it might be easiest to upload your existing global ssh key,
such that all following ssh commands work without modification.
To make connecting to the test bed easier, this project provides an ssh config located at.ssh/config.
Make sure you edit the IP addresses of the nodes once they are running in Chameleon.
This project uses Ansible to set up the Kubernetes cluster on the remote nodes.
Follow the setup instructions in the ansible directory.
kubectl uses port 6443 to communicate with a cluster.
The k3s-server node is only accessible via the jump host. This means if we want to use kubectl locally we need to modify the .kube/config.
The fix_kubeconfig utility does this for you. The next step is to forward traffic from localhost to k3s-server.
This project provides the forward_kubectl utility to achieve port forwarding from localhost:6443 to the k3s-server node.
fix_kubeconfig && forward_kubectlWarning
This command will only work once the Ansible setup has been completed.
You can verify connectivity to the Kubernetes cluster by running
kubectl get nodes
All the applications of this repository depend on external resources like an Apache Kafka cluster as well as a mongoDB database.
All dependencies for the applications are managed and deployed via a custom helm chart in the helm-charts directory.
Refer to the README for an installation walkthrough.