run

Run FrameworkController

We provide various approaches to run FrameworkController:

• Run By Kubernetes StatefulSet
• Run By Docker Container
• Run By OS Process

Notes:

• For a single k8s cluster, one instance of FrameworkController orchestrates all Frameworks in all namespaces.
• For a single k8s cluster, ensure at most one instance of FrameworkController is running at any point in time.
• For the full FrameworkController configuration, see Config Usage and Config Example.

Run By Kubernetes StatefulSet

• This approach is better for production, since StatefulSet by itself provides self-healing and can ensure at most one instance of FrameworkController is running at any point in time.
• Using official image to demonstrate this example.

Prerequisite

If the k8s cluster enforces Authorization, you need to first create a ServiceAccount with granted permission for FrameworkController. For example, if the cluster enforces RBAC:

kubectl create serviceaccount frameworkcontroller --namespace default
kubectl create clusterrolebinding frameworkcontroller \
--clusterrole=cluster-admin \
--user=system:serviceaccount:default:frameworkcontroller

Run

Run FrameworkController with above ServiceAccount and the k8s inClusterConfig:

Run with default config

kubectl create -f frameworkcontroller-with-default-config.yaml

frameworkcontroller-with-default-config.yaml:

apiVersion: apps/v1kind: StatefulSetmetadata:
name: frameworkcontrollernamespace: defaultspec:
serviceName: frameworkcontrollerselector:
matchLabels:
app: frameworkcontrollerreplicas: 1template:
metadata:
labels:
app: frameworkcontrollerspec:
# Using the ServiceAccount with granted permission# if the k8s cluster enforces authorization.serviceAccountName: frameworkcontrollercontainers:
- name: frameworkcontrollerimage: frameworkcontroller/frameworkcontroller# Using k8s inClusterConfig, so usually, no need to specify# KUBE_APISERVER_ADDRESS or KUBECONFIG#env:#- name: KUBE_APISERVER_ADDRESS# value: {http[s]://host:port}#- name: KUBECONFIG# value: {Pod Local KubeConfig File Path}

Run with customized config

kubectl create -f frameworkcontroller-customized-config.yaml
kubectl create -f frameworkcontroller-with-customized-config.yaml

frameworkcontroller-customized-config.yaml:

apiVersion: v1kind: ConfigMapmetadata:
name: frameworkcontroller-confignamespace: defaultdata:
frameworkcontroller.yaml: | kubeClientQps: 200 kubeClientBurst: 300 workerNumber: 500 largeFrameworkCompression: true frameworkCompletedRetainSec: 2592000 #podFailureSpec: #- code: 221 # phrase: ContainerTensorflowOOMKilled # type: # attributes: [Permanent] # podPatterns: # - containers: # - messageRegex: '(?msi)tensorflow.*ResourceExhaustedError.*OOM.*' # codeRange: {min: 1} # nameRegex: '(?ms).*' #- {More customized podFailureSpec, better to also include these in the default config}

frameworkcontroller-with-customized-config.yaml:

apiVersion: apps/v1kind: StatefulSetmetadata:
name: frameworkcontrollernamespace: defaultspec:
serviceName: frameworkcontrollerselector:
matchLabels:
app: frameworkcontrollerreplicas: 1template:
metadata:
labels:
app: frameworkcontrollerspec:
# Using the ServiceAccount with granted permission# if the k8s cluster enforces authorization.serviceAccountName: frameworkcontrollercontainers:
- name: frameworkcontrollerimage: frameworkcontroller/frameworkcontroller# Using k8s inClusterConfig, so usually, no need to specify# KUBE_APISERVER_ADDRESS or KUBECONFIG#env:#- name: KUBE_APISERVER_ADDRESS# value: {http[s]://host:port}#- name: KUBECONFIG# value: {Pod Local KubeConfig File Path}command: ["bash", "-c","cp /frameworkcontroller-config/frameworkcontroller.yaml . && ./start.sh"]volumeMounts:
- name: frameworkcontroller-configmountPath: /frameworkcontroller-configvolumes:
- name: frameworkcontroller-configconfigMap:
name: frameworkcontroller-config

Run By Docker Container

• This approach may be better for development sometimes.
• Using official image to demonstrate this example.

Run

If you have an insecure ApiServer address (can be got from Insecure ApiServer or kubectl proxy) which does not enforce authentication, you only need to provide the address:

docker run -e KUBE_APISERVER_ADDRESS={http[s]://host:port} \
frameworkcontroller/frameworkcontroller

Otherwise, you need to provide your KubeConfig File which inlines or refers the ApiServer Credential Files with granted permission:

docker run -e KUBECONFIG=/mnt/.kube/config \
-v {Host Local KubeConfig File Path}:/mnt/.kube/config \
-v {Host Local ApiServer Credential File Path}:{Container Local ApiServer Credential File Path} \
frameworkcontroller/frameworkcontroller

For example, if the k8s cluster is created by Minikube:

docker run -e KUBECONFIG=/mnt/.kube/config \
-v ${HOME}/.kube/config:/mnt/.kube/config \
-v ${HOME}/.minikube:${HOME}/.minikube \
frameworkcontroller/frameworkcontroller

Run By OS Process

• This approach may be better for development sometimes.
• Using local built binary distribution to demonstrate this example.

Prerequisite

Ensure you have installed Golang 1.12.6 or above and the ${GOPATH} is valid.

Then build the FrameworkController binary distribution:

export PROJECT_DIR=${GOPATH}/src/github.com/microsoft/frameworkcontroller
rm -rf ${PROJECT_DIR}
mkdir -p ${PROJECT_DIR}
git clone https://github.com/microsoft/frameworkcontroller.git ${PROJECT_DIR}cd${PROJECT_DIR}
./build/frameworkcontroller/go-build.sh

Run

If you have an insecure ApiServer address (can be got from Insecure ApiServer or kubectl proxy) which does not enforce authentication, you only need to provide the address:

KUBE_APISERVER_ADDRESS={http[s]://host:port} \
./dist/frameworkcontroller/start.sh

Otherwise, you need to provide your KubeConfig File which inlines or refers the ApiServer Credential Files with granted permission:

KUBECONFIG={Process Local KubeConfig File Path} \
./dist/frameworkcontroller/start.sh

For example:

KUBECONFIG=${HOME}/.kube/config \
./dist/frameworkcontroller/start.sh

And in above example, ${HOME}/.kube/config is the default value of KUBECONFIG, so you can skip it:

./dist/frameworkcontroller/start.sh

Next

1. Submit Framework