Fix airflow-webserver cannot create resource pods for k8s exceutor deployed using helm chart

[snjypl]

we see the fellowing error in the webserver log on trying to run the task manually from the UI for k8s executor.

Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/kubernetes_executor.py", line 264, in run_pod_async
body=sanitized_pod, namespace=pod.metadata.namespace, **kwargs
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 7356, in create_namespaced_pod
return self.create_namespaced_pod_with_http_info(namespace, body, **kwargs) # noqa: E501
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api/core_v1_api.py", line 7469, in create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 353, in call_api
_preload_content, _request_timeout, _host)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 184, in __call_api
_request_timeout=_request_timeout)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 397, in request
body=body)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 281, in POST
body=body)
File "/home/airflow/.local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 234, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '5a631062-cbc1-49f6-ae20-3581626a6249', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': 'e6bfeec0-ef68-4420-88d8-b7024eed9fea', 'X-Kubernetes-Pf-Prioritylevel-Uid': '0e598fd5-a817-46f7-be85-6ac94aeb7872', 'Date': 'Wed, 18 Jan 2023 10:52:05 GMT', 'Content-Length': '381'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:airflow:airflow-webserver\" cannot create resource \"pods\" in API group \"\" in the namespace \"airflow\": RBAC: clusterrole.rbac.authorization.k8s.io \"system:openshift:scc:anyuid\" not found","reason":"Forbidden","details":{"kind":"pods"},"code":403}

in case of k8s executor, the webserver role should be able to launch the pod for manual task trigger. we need to add the airflow-webserver service account to pod-launcher role.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[arjunanan6]

Alright, so I attempted this locally where I have no restrictions. As discussed on #28394, scheduled tasks run just fine, but there is an exception when running a task manually:

[2023-01-20T11:57:44.263+0000] {kubernetes_executor.py:527} INFO - Start Kubernetes executor
[2023-01-20T11:57:44.306+0000] {kubernetes_executor.py:476} INFO - Found 0 queued task instances
[2023-01-20T11:57:44.309+0000] {base_executor.py:95} INFO - Adding to queue: ['airflow', 'tasks', 'run', 'HELLO_WORLD', 'hello', 'scheduled__2023-01-20T11:40:00+00:00', '--ignore-all-dependencies', '--ignore-dependencies', '--force', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/hello.py']
[2023-01-20T11:57:44.310+0000] {kubernetes_executor.py:559} INFO - Add task TaskInstanceKey(dag_id='HELLO_WORLD', task_id='hello', run_id='scheduled__2023-01-20T11:40:00+00:00', try_number=4, map_index=-1) with command ['airflow', 'tasks', 'run', 'HELLO_WORLD', 'hello', 'scheduled__2023-01-20T11:40:00+00:00', '--ignore-all-dependencies', '--ignore-dependencies', '--force', '--local', '--pool', 'default_pool', '--subdir', 'DAGS_FOLDER/hello.py']
[2023-01-20T11:57:44.310+0000] {kubernetes_executor.py:130} INFO - Event: and now my watch begins starting at resource_version: 0
[2023-01-20T11:57:44.383+0000] {kubernetes_executor.py:339} INFO - Creating kubernetes pod for job is TaskInstanceKey(dag_id='HELLO_WORLD', task_id='hello', run_id='scheduled__2023-01-20T11:40:00+00:00', try_number=2, map_index=-1), with pod name hello-world-hello-be2bad2bd8dc4568bd1ba73082ecef4a
[2023-01-20T11:57:44.392+0000] {kubernetes_executor.py:274} ERROR - Exception when attempting to create Namespaced Pod: {
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"annotations": {
"dag_id": "HELLO_WORLD",
"task_id": "hello",
"try_number": "2",
"run_id": "scheduled__2023-01-20T11:40:00+00:00"
},
"labels": {
"tier": "airflow",
"component": "worker",
"release": "airflowlocal",
"airflow-worker": "None",
"dag_id": "HELLO_WORLD",
"task_id": "hello",
"try_number": "2",
"airflow_version": "2.5.0",
"kubernetes_executor": "True",
"run_id": "scheduled__2023-01-20T1140000000-c15690dab"
},
"name": "hello-world-hello-be2bad2bd8dc4568bd1ba73082ecef4a",
"namespace": "airflow"
},
"spec": {
"affinity": {},
"containers": [
{
"args": [
"airflow",
"tasks",
"run",
"HELLO_WORLD",
"hello",
"scheduled__2023-01-20T11:40:00+00:00",
"--ignore-all-dependencies",
"--ignore-dependencies",
"--force",
"--local",
"--pool",
"default_pool",
"--subdir",
"DAGS_FOLDER/hello.py"
],
"env": [
{
"name": "AIRFLOW__CORE__EXECUTOR",
"value": "LocalExecutor"
},
{
"name": "AIRFLOW__CORE__FERNET_KEY",
"valueFrom": {
"secretKeyRef": {
"key": "fernet-key",
"name": "airflowlocal-fernet-key"
}
}
},
{
"name": "AIRFLOW__CORE__SQL_ALCHEMY_CONN",
"valueFrom": {
"secretKeyRef": {
"key": "connection",
"name": "airflowlocal-airflow-metadata"
}
}
},
{
"name": "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN",
"valueFrom": {
"secretKeyRef": {
"key": "connection",
"name": "airflowlocal-airflow-metadata"
}
}
},
{
"name": "AIRFLOW_CONN_AIRFLOW_DB",
"valueFrom": {
"secretKeyRef": {
"key": "connection",
"name": "airflowlocal-airflow-metadata"
}
}
},
{
"name": "AIRFLOW__WEBSERVER__SECRET_KEY",
"valueFrom": {
"secretKeyRef": {
"key": "webserver-secret-key",
"name": "airflowlocal-webserver-secret-key"
}
}
},
{
"name": "AIRFLOW_IS_K8S_EXECUTOR_POD",
"value": "True"
}
],
"image": "my-dags:0.0.1",
"imagePullPolicy": "IfNotPresent",
"name": "base",
"resources": {},
"volumeMounts": [
{
"mountPath": "/opt/airflow/logs",
"name": "logs"
},
{
"mountPath": "/opt/airflow/airflow.cfg",
"name": "config",
"readOnly": true,
"subPath": "airflow.cfg"
},
{
"mountPath": "/opt/airflow/config/airflow_local_settings.py",
"name": "config",
"readOnly": true,
"subPath": "airflow_local_settings.py"
}
]
}
],
"restartPolicy": "Never",
"securityContext": {
"fsGroup": 0,
"runAsUser": 50000
},
"serviceAccountName": "airflowlocal-worker",
"volumes": [
{
"emptyDir": {},
"name": "logs"
},
{
"configMap": {
"name": "airflowlocal-airflow-config"
},
"name": "config"
}
]
}
}
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/executors/kubernetes_executor.py", line 269, in run_pod_async
resp = self.kube_client.create_namespaced_pod(
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 7356, in create_namespaced_pod
return self.create_namespaced_pod_with_http_info(namespace, body, **kwargs) # noqa: E501
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 7455, in create_namespaced_pod_with_http_info
return self.api_client.call_api(
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/api_client.py", line 391, in request
return self.rest_client.POST(url,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 275, in POST
return self.request("POST", url,
File "/home/airflow/.local/lib/python3.9/site-packages/kubernetes/client/rest.py", line 234, in request
raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'c2225504-16da-4966-ae42-36241a0d49cb', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '3dd6f880-05b3-46ee-a0f2-a45a4c663a50', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'b8f0d7a8-7a33-4a3e-8e49-112cfe10ad1d', 'Date': 'Fri, 20 Jan 2023 11:57:44 GMT', 'Content-Length': '299'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:airflow:airflowlocal-webserver\" cannot create resource \"pods\" in API group \"\" in the namespace \"airflow\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}

Which is strange, since this is the same service account that is spinning up pods to execute a scheduled task, here is a yaml of that pod definition:

apiVersion: v1
kind: Pod
metadata:
annotations:
dag_id: HELLO_WORLD
run_id: scheduled__2023-01-20T11:40:00+00:00
task_id: hello
try_number: "3"
creationTimestamp: "2023-01-20T11:57:11Z"
labels:
airflow-worker: "80"
airflow_version: 2.5.0
component: worker
dag_id: HELLO_WORLD
kubernetes_executor: "True"
release: airflowlocal
run_id: scheduled__2023-01-20T1140000000-c15690dab
task_id: hello
tier: airflow
try_number: "3"
name: hello-world-hello-3c579ea84688467bab7036d3bc940c64
namespace: airflow
resourceVersion: "40588"
uid: 53017417-d794-4532-bdb8-fa92db4f97fd
spec:
affinity: {}
containers:
- args:
- airflow
- tasks
- run
- HELLO_WORLD
- hello
- scheduled__2023-01-20T11:40:00+00:00
- --local
- --subdir
- DAGS_FOLDER/hello.py
env:
- name: AIRFLOW__CORE__EXECUTOR
value: LocalExecutor
- name: AIRFLOW__CORE__FERNET_KEY
valueFrom:
secretKeyRef:
key: fernet-key
name: airflowlocal-fernet-key
- name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
key: connection
name: airflowlocal-airflow-metadata
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
valueFrom:
secretKeyRef:
key: connection
name: airflowlocal-airflow-metadata
- name: AIRFLOW_CONN_AIRFLOW_DB
valueFrom:
secretKeyRef:
key: connection
name: airflowlocal-airflow-metadata
- name: AIRFLOW__WEBSERVER__SECRET_KEY
valueFrom:
secretKeyRef:
key: webserver-secret-key
name: airflowlocal-webserver-secret-key
- name: AIRFLOW_IS_K8S_EXECUTOR_POD
value: "True"
image: my-dags:0.0.1
imagePullPolicy: IfNotPresent
name: base
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/airflow/logs
name: logs
- mountPath: /opt/airflow/airflow.cfg
name: config
readOnly: true
subPath: airflow.cfg
- mountPath: /opt/airflow/config/airflow_local_settings.py
name: config
readOnly: true
subPath: airflow_local_settings.py
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-w5gcp
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: kind-control-plane
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Never
schedulerName: default-scheduler
securityContext:
fsGroup: 0
runAsUser: 50000
serviceAccount: airflowlocal-worker
serviceAccountName: airflowlocal-worker
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- emptyDir: {}
name: logs
- configMap:
defaultMode: 420
name: airflowlocal-airflow-config
name: config
- name: kube-api-access-w5gcp
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2023-01-20T11:57:11Z"
reason: PodCompleted
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2023-01-20T11:57:20Z"
reason: PodCompleted
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2023-01-20T11:57:20Z"
reason: PodCompleted
status: "False"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2023-01-20T11:57:11Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://a81155f6103c7fee21ab8b06298d8ba04a112f91356685f2d6414dd68136eb3b
image: docker.io/library/my-dags:0.0.1
imageID: docker.io/library/import-2023-01-20@sha256:9298bf3504bc8279b2270f7d41e9d2c1244a39e22e0ac0c534c5e881d5621ca9
lastState: {}
name: base
ready: false
restartCount: 0
started: false
state:
terminated:
containerID: containerd://a81155f6103c7fee21ab8b06298d8ba04a112f91356685f2d6414dd68136eb3b
exitCode: 0
finishedAt: "2023-01-20T11:57:19Z"
reason: Completed
startedAt: "2023-01-20T11:57:11Z"
hostIP: 172.18.0.2
phase: Running
podIP: 10.244.0.42
podIPs:
- ip: 10.244.0.42
qosClass: BestEffort
startTime: "2023-01-20T11:57:11Z"

I verified whether this SA is able to spin up a pod, and that checks out too:

kubectl auth can-i get pods --as system:serviceaccount:airflow:airflowlocal-webserver
yes

[snjypl]

@arjunanan6 please share the output of kubectl get rolebinding airflow-pod-launcher-rolebinding -o yaml

[arjunanan6]

@snjypl Sure, here you go:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
annotations:
meta.helm.sh/release-name: airflowlocal
meta.helm.sh/release-namespace: airflow
creationTimestamp: "2023-01-18T09:52:13Z"
labels:
app.kubernetes.io/managed-by: Helm
chart: airflow-1.7.0
heritage: Helm
release: airflowlocal
tier: airflow
name: airflowlocal-pod-launcher-rolebinding
namespace: airflow
resourceVersion: "13279"
uid: 05617ca4-ebf3-424a-990b-8b59274702f0
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: airflowlocal-pod-launcher-role
subjects:
- kind: ServiceAccount
name: airflowlocal-scheduler
namespace: airflow
- kind: ServiceAccount
name: airflowlocal-worker
namespace: airflow

[snjypl]

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
annotations:
meta.helm.sh/release-name: airflowlocal
meta.helm.sh/release-namespace: airflow
creationTimestamp: "2023-01-18T09:52:13Z"
labels:
app.kubernetes.io/managed-by: Helm
chart: airflow-1.7.0
heritage: Helm
release: airflowlocal
tier: airflow
name: airflowlocal-pod-launcher-rolebinding
namespace: airflow
resourceVersion: "13279"
uid: 05617ca4-ebf3-424a-990b-8b59274702f0
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: airflowlocal-pod-launcher-role
subjects:

• kind: ServiceAccount
  name: airflowlocal-scheduler
  namespace: airflow
• kind: ServiceAccount
  name: airflowlocal-worker
  namespace: airflow

@arjunanan6 it seems like you have not applied the patch in this PR correctly,
as you can see airflowlocal-webserver service account does not have pod laucher role. it is not in the list of subjects.

you can try to manually apply the bellow gist
https://gist.github.com/snjypl/aeebe582be0190e483163224f9c966e7

reason why the scheduled task are not having issue is, the scheduled task pod are launched by the scheduler. the scheduler sa has pod launcher role.

in case of manual trigger, the k8s pod is launched by the webserver.

how are you deploying the helm chart from this PR? if you can share the steps i might be able to help you with it.
anyway, manually doing a kubectl apply -f <the gist file> should fix it for you. later you can try testing the helm chart.

[snjypl]

@arjunanan6 also,
you should check
kubectl auth can-i create pods --as system:serviceaccount:airflow:airflowlocal-webserver

[arjunanan6]

@snjypl Yeah, you're right. Not sure what happened there, but now the patch has been added in properly and I can confirm that the the change works. Running a task individually works. :)

[potiuk]

LGTM. But would love @jedcunningham or @dstandish to verify it.

[jedcunningham]

So this PR does fix the chart, I can't argue there.

However, I'd never actually looked at the manual run code, and it turns out the webserver endpoint kicks off an executor and abandons it! KE pods don't even get cleaned up by it, that falls on a scheduler that ends up adopting the pod. That's the happy path! If you have something happen that causes your worker to not be successful, it may not ever be adopted?

So, in short, I'm a little hesitant to "fix" this chart issue when the underlying feature it relies on is itself flawed. Meaning, I think it's correct in the big picture that the webserver isn't able to launch pods. I'd rather explore fixing the manual run behavior first.

[jedcunningham]

Suggested change

print(docs)

[potiuk]

So @snjypl are you going to work on it as explained by @jedcunningham ? Or maybe you would like to create theissue to fix the manual behaviour if you are not sure how to fix it?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.