EKS GitOps Workshop

For this example we assume a scenario with two clusters: dev and production. The end goal is to leverage Flux and Kustomize to manage both clusters while minimizing duplicated declarations.

We will configure Flux to install, test and upgrade a demo app using HelmRepository and HelmRelease custom resources. Flux will monitor the Helm repository, and it will automatically upgrade the Helm releases to their latest chart version based on semver ranges.

Prerequisites

You will need a Kubernetes cluster version 1.16 or newer and kubectl version 1.18. For a quick local test, you can use Kubernetes kind. Any other Kubernetes setup will work as well though.

In order to follow the guide you'll need a GitHub account and a personal access token that can create repositories (check all permissions under repo).

Install the Flux CLI on MacOS and Linux using Homebrew:

brew install fluxcd/tap/flux

Or install the CLI by downloading precompiled binaries using a Bash script:

curl -s https://fluxcd.io/install.sh | sudo bash

Repository structure

The Git repository contains the following top directories:

• apps dir contains Helm releases with a custom configuration per cluster
• infrastructure dir contains infra tools such as NGINX ingress controller and Helm repository definitions
• clusters dir contains the Flux configuration per cluster

├── apps
│ ├── base
│ ├── production │ └── dev
├── infrastructure
│ ├── nginx
│ ├── redis
│ └── sources
└── clusters
├── production
└── dev

The apps configuration is structured into:

• apps/base/ dir contains namespaces and Helm release definitions
• apps/production/ dir contains the production Helm release values
• apps/dev/ dir contains the dev values

./apps/
├── base
│ └── podinfo
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ └── release.yaml
├── production
│ ├── kustomization.yaml
│ └── podinfo-patch.yaml
└── dev
├── kustomization.yaml
└── podinfo-patch.yaml

In apps/base/podinfo/ dir we have a HelmRelease with common values for both clusters:

apiVersion: helm.toolkit.fluxcd.io/v2beta1kind: HelmReleasemetadata:
name: podinfonamespace: podinfospec:
releaseName: podinfochart:
spec:
chart: podinfosourceRef:
kind: HelmRepositoryname: podinfonamespace: flux-systeminterval: 5mvalues:
cache: redis-master.redis:6379ingress:
enabled: trueannotations:
kubernetes.io/ingress.class: nginx

In apps/dev/ dir we have a Kustomize patch with the dev specific values:

apiVersion: helm.toolkit.fluxcd.io/v2beta1kind: HelmReleasemetadata:
name: podinfospec:
chart:
spec:
version: ">=1.0.0-alpha"test:
enable: truevalues:
ingress:
hosts:
- host: podinfo.dev

Note that with version: ">=1.0.0-alpha" we configure Flux to automatically upgrade the HelmRelease to the latest chart version including alpha, beta and pre-releases.

In apps/production/ dir we have a Kustomize patch with the production specific values:

apiVersion: helm.toolkit.fluxcd.io/v2beta1kind: HelmReleasemetadata:
name: podinfonamespace: podinfospec:
chart:
spec:
version: ">=1.0.0"values:
ingress:
hosts:
- host: podinfo.production

Note that with version: ">=1.0.0" we configure Flux to automatically upgrade the HelmRelease to the latest stable chart version (alpha, beta and pre-releases will be ignored).

In infrastructure/dev/ dir we have infrastructure toold with dev specific values:

./infrastructure/dev
├── nginx
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ └── release.yaml
├── redis
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ └── release.yaml
└── sources
├── bitnami.yaml
├── kustomization.yaml
└── podinfo.yaml

In infrastructure/production/ dir we have infrastructure toold with dev specific values:

./infrastructure/production
├── nginx
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ └── release.yaml
├── redis
│ ├── kustomization.yaml
│ ├── namespace.yaml
│ └── release.yaml
└── sources
├── bitnami.yaml
├── kustomization.yaml
└── podinfo.yaml

In infrastructure/*/sources/ dir we have the Helm repositories definitions:

apiVersion: source.toolkit.fluxcd.io/v1beta1kind: HelmRepositorymetadata:
name: podinfospec:
interval: 5murl: https://stefanprodan.github.io/podinfo
---
apiVersion: source.toolkit.fluxcd.io/v1beta1kind: HelmRepositorymetadata:
name: bitnamispec:
interval: 30murl: https://charts.bitnami.com/bitnami

Note that with interval: 5m we configure Flux to pull the Helm repository index every five minutes. If the index contains a new chart version that matches a HelmRelease semver range, Flux will upgrade the release.

Bootstrap dev and production

The clusters dir contains the Flux configuration:

./clusters/
├── production
│ ├── apps.yaml
│ └── infrastructure.yaml
└── dev
├── apps.yaml
└── infrastructure.yaml

In clusters/dev/ dir we have the Kustomization definitions:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1kind: Kustomizationmetadata:
name: appsnamespace: flux-systemspec:
interval: 10m0sdependsOn:
- name: infrastructuresourceRef:
kind: GitRepositoryname: flux-systempath: ./apps/devprune: truevalidation: client
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1kind: Kustomizationmetadata:
name: infrastructurenamespace: flux-systemspec:
interval: 10m0ssourceRef:
kind: GitRepositoryname: flux-systempath: ./infrastructure/dev

Note that with path: ./apps/dev we configure Flux to sync the dev Kustomize overlay and with dependsOn we tell Flux to create the infrastructure items before deploying the apps.

Fork this repository on your personal GitHub account and export your GitHub access token, username and repo name:

export GITHUB_TOKEN=<your-token>export GITHUB_USER=<your-username>export GITHUB_REPO=<repository-name>

Verify that your dev cluster satisfies the prerequisites with:

flux check --pre

Set the kubectl context to your dev cluster and bootstrap Flux:

flux bootstrap github \
--context=dev \
--owner=${GITHUB_USER} \
--repository=${GITHUB_REPO} \
--branch=main \
--private=false \
--personal \
--path=clusters/dev

The bootstrap command commits the manifests for the Flux components in clusters/dev/flux-system dir and creates a deploy key with read-only access on GitHub, so it can pull changes inside the cluster.

Watch for the Helm releases being install on dev:

$ watch flux get helmreleases --all-namespaces NAMESPACE	NAME REVISION	SUSPENDED	READY	MESSAGE nginx nginx 5.6.14 False True release reconciliation succeeded	podinfo podinfo	5.0.3 False True release reconciliation succeeded	redis redis 11.3.4 False True release reconciliation succeeded

Verify that the demo app can be accessed via ingress:

$ kubectl -n nginx port-forward svc/nginx-ingress-controller 8080:80 &
$ curl -H "Host: podinfo.dev" http://localhost:8080{ "hostname": "podinfo-59489db7b5-lmwpn", "version": "5.0.3"}

Bootstrap Flux on production by setting the context and path to your production cluster:

flux bootstrap github \
--context=production \
--owner=${GITHUB_USER} \
--repository=${GITHUB_REPO} \
--private=false \
--branch=main \
--personal \
--path=clusters/production

Watch the production reconciliation:

$ watch flux get kustomizationsNAME REVISION READYapps main/797cd90cc8e81feb30cfe471a5186b86daf2758d	Trueflux-system main/797cd90cc8e81feb30cfe471a5186b86daf2758d	Trueinfrastructure	main/797cd90cc8e81feb30cfe471a5186b86daf2758d	True