Kubernetes decorator

[subkanthi]

Decorator to execute functions in k8s using KubernetesPodOperator

closes: #19135

@task.kubernetes(image='python:3.8-slim-buster', name='k8s_test', namespace='default')
def k8s_decorator_func():
print("decorator func")

If no parameters passed, defaults are set in kubernetes.py

 # Image, name and namespace are all required.
if not 'image' in kwargs:
kwargs['image'] = 'python:3.8-slim-buster'
if not 'name' in kwargs:
kwargs['name'] = f'k8s_airflow_pod_{uuid.uuid4().hex}'
if not 'namespace' in kwargs:
kwargs['namespace'] = 'default'

This version sets the function as an environment variable - PYTHON_SCRIPT and in the command , the logic is to retrieve the environment variable and write to '/tmp/script.py' to execute the function(very similar to Docker)

1. Initially proceeded with trying to mount a local file using the hostPath, the problem is that requires Airflow also to run in the same node. For example running airflow locally and connecting to the kind cluster would need a mount.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Also need to add image to the argument list of _KubernetesDecoratedOperator

[ashb]

This has to be in the kube provider folder, else it won't get released with the provider, and it won't work.

[subkanthi]

@ashb , currently the decorator uses the write_python_script function thats in the utils/python_virtualenv.py file and thats why the template(python_kubernetes_script.jinja2) was also placed in the same folder. If we want to remove the dependency on the utils folder, I guess we can make a copy of the write_python_script in the cncf.providers.kubernetes folder. Question is if there a benefit of having a central place for the function and all templates, Docker decorator and python operator also call this function.

def write_python_script(
jinja_context: dict,
filename: str,
render_template_as_native_obj: bool = False,
template_file: str = 'python_virtualenv_script.jinja2',
):

[Asciotti]

Bump? :)

[kaxil]

Please also rebase your PR?

[kaxil]

Can you move this @subkanthi to the cncf kube providers please?

[uranusjr]

What can go into kwargs here? It seems unnecessary.

[uranusjr]

Nit, it’s easier to use setdefault for these

[uranusjr]

Unused?

[potiuk]

Closing, as this has been already implemented and merged in #25663

[subkanthi]

Decorator to execute functions in k8s using KubernetesPodOperator

closes: #19135

@task.kubernetes(image='python:3.8-slim-buster', name='k8s_test', namespace='default')
def k8s_decorator_func():
print("decorator func")

If no parameters passed, defaults are set in kubernetes.py

 # Image, name and namespace are all required.
if not 'image' in kwargs:
kwargs['image'] = 'python:3.8-slim-buster'
if not 'name' in kwargs:
kwargs['name'] = f'k8s_airflow_pod_{uuid.uuid4().hex}'
if not 'namespace' in kwargs:
kwargs['namespace'] = 'default'

This version sets the function as an environment variable - PYTHON_SCRIPT and in the command , the logic is to retrieve the environment variable and write to '/tmp/script.py' to execute the function(very similar to Docker)

1. Initially proceeded with trying to mount a local file using the hostPath, the problem is that requires Airflow also to run in the same node. For example running airflow locally and connecting to the kind cluster would need a mount.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Also need to add image to the argument list of _KubernetesDecoratedOperator

[ashb]

This has to be in the kube provider folder, else it won't get released with the provider, and it won't work.

[subkanthi]

@ashb , currently the decorator uses the write_python_script function thats in the utils/python_virtualenv.py file and thats why the template(python_kubernetes_script.jinja2) was also placed in the same folder. If we want to remove the dependency on the utils folder, I guess we can make a copy of the write_python_script in the cncf.providers.kubernetes folder. Question is if there a benefit of having a central place for the function and all templates, Docker decorator and python operator also call this function.

def write_python_script(
jinja_context: dict,
filename: str,
render_template_as_native_obj: bool = False,
template_file: str = 'python_virtualenv_script.jinja2',
):

[Asciotti]

Bump? :)

[kaxil]

Please also rebase your PR?

[kaxil]

Can you move this @subkanthi to the cncf kube providers please?

[uranusjr]

What can go into kwargs here? It seems unnecessary.

[uranusjr]

Nit, it’s easier to use setdefault for these

[uranusjr]

Unused?

[potiuk]

Closing, as this has been already implemented and merged in #25663

[subkanthi]

Decorator to execute functions in k8s using KubernetesPodOperator

closes: #19135

@task.kubernetes(image='python:3.8-slim-buster', name='k8s_test', namespace='default')
def k8s_decorator_func():
print("decorator func")

If no parameters passed, defaults are set in kubernetes.py

 # Image, name and namespace are all required.
if not 'image' in kwargs:
kwargs['image'] = 'python:3.8-slim-buster'
if not 'name' in kwargs:
kwargs['name'] = f'k8s_airflow_pod_{uuid.uuid4().hex}'
if not 'namespace' in kwargs:
kwargs['namespace'] = 'default'

This version sets the function as an environment variable - PYTHON_SCRIPT and in the command , the logic is to retrieve the environment variable and write to '/tmp/script.py' to execute the function(very similar to Docker)

1. Initially proceeded with trying to mount a local file using the hostPath, the problem is that requires Airflow also to run in the same node. For example running airflow locally and connecting to the kind cluster would need a mount.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Also need to add image to the argument list of _KubernetesDecoratedOperator

[ashb]

This has to be in the kube provider folder, else it won't get released with the provider, and it won't work.

[subkanthi]

@ashb , currently the decorator uses the write_python_script function thats in the utils/python_virtualenv.py file and thats why the template(python_kubernetes_script.jinja2) was also placed in the same folder. If we want to remove the dependency on the utils folder, I guess we can make a copy of the write_python_script in the cncf.providers.kubernetes folder. Question is if there a benefit of having a central place for the function and all templates, Docker decorator and python operator also call this function.

def write_python_script(
jinja_context: dict,
filename: str,
render_template_as_native_obj: bool = False,
template_file: str = 'python_virtualenv_script.jinja2',
):

[Asciotti]

Bump? :)

[kaxil]

Please also rebase your PR?

[kaxil]

Can you move this @subkanthi to the cncf kube providers please?

[uranusjr]

What can go into kwargs here? It seems unnecessary.

[uranusjr]

Nit, it’s easier to use setdefault for these

[uranusjr]

Unused?

[potiuk]

Closing, as this has been already implemented and merged in #25663

[subkanthi]

Decorator to execute functions in k8s using KubernetesPodOperator

closes: #19135

@task.kubernetes(image='python:3.8-slim-buster', name='k8s_test', namespace='default')
def k8s_decorator_func():
print("decorator func")

If no parameters passed, defaults are set in kubernetes.py

 # Image, name and namespace are all required.
if not 'image' in kwargs:
kwargs['image'] = 'python:3.8-slim-buster'
if not 'name' in kwargs:
kwargs['name'] = f'k8s_airflow_pod_{uuid.uuid4().hex}'
if not 'namespace' in kwargs:
kwargs['namespace'] = 'default'

This version sets the function as an environment variable - PYTHON_SCRIPT and in the command , the logic is to retrieve the environment variable and write to '/tmp/script.py' to execute the function(very similar to Docker)

1. Initially proceeded with trying to mount a local file using the hostPath, the problem is that requires Airflow also to run in the same node. For example running airflow locally and connecting to the kind cluster would need a mount.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Also need to add image to the argument list of _KubernetesDecoratedOperator

[ashb]

This has to be in the kube provider folder, else it won't get released with the provider, and it won't work.

[subkanthi]

@ashb , currently the decorator uses the write_python_script function thats in the utils/python_virtualenv.py file and thats why the template(python_kubernetes_script.jinja2) was also placed in the same folder. If we want to remove the dependency on the utils folder, I guess we can make a copy of the write_python_script in the cncf.providers.kubernetes folder. Question is if there a benefit of having a central place for the function and all templates, Docker decorator and python operator also call this function.

def write_python_script(
jinja_context: dict,
filename: str,
render_template_as_native_obj: bool = False,
template_file: str = 'python_virtualenv_script.jinja2',
):

[Asciotti]

Bump? :)

[kaxil]

Please also rebase your PR?

[kaxil]

Can you move this @subkanthi to the cncf kube providers please?

[uranusjr]

What can go into kwargs here? It seems unnecessary.

[uranusjr]

Nit, it’s easier to use setdefault for these

[uranusjr]

Unused?

[potiuk]

Closing, as this has been already implemented and merged in #25663

[subkanthi]

Decorator to execute functions in k8s using KubernetesPodOperator

closes: #19135

@task.kubernetes(image='python:3.8-slim-buster', name='k8s_test', namespace='default')
def k8s_decorator_func():
print("decorator func")

If no parameters passed, defaults are set in kubernetes.py

 # Image, name and namespace are all required.
if not 'image' in kwargs:
kwargs['image'] = 'python:3.8-slim-buster'
if not 'name' in kwargs:
kwargs['name'] = f'k8s_airflow_pod_{uuid.uuid4().hex}'
if not 'namespace' in kwargs:
kwargs['namespace'] = 'default'

This version sets the function as an environment variable - PYTHON_SCRIPT and in the command , the logic is to retrieve the environment variable and write to '/tmp/script.py' to execute the function(very similar to Docker)

1. Initially proceeded with trying to mount a local file using the hostPath, the problem is that requires Airflow also to run in the same node. For example running airflow locally and connecting to the kind cluster would need a mount.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Also need to add image to the argument list of _KubernetesDecoratedOperator

[ashb]

This has to be in the kube provider folder, else it won't get released with the provider, and it won't work.

[subkanthi]

@ashb , currently the decorator uses the write_python_script function thats in the utils/python_virtualenv.py file and thats why the template(python_kubernetes_script.jinja2) was also placed in the same folder. If we want to remove the dependency on the utils folder, I guess we can make a copy of the write_python_script in the cncf.providers.kubernetes folder. Question is if there a benefit of having a central place for the function and all templates, Docker decorator and python operator also call this function.

def write_python_script(
jinja_context: dict,
filename: str,
render_template_as_native_obj: bool = False,
template_file: str = 'python_virtualenv_script.jinja2',
):

[Asciotti]

Bump? :)

[kaxil]

Please also rebase your PR?

[kaxil]

Can you move this @subkanthi to the cncf kube providers please?

[uranusjr]

What can go into kwargs here? It seems unnecessary.

[uranusjr]

Nit, it’s easier to use setdefault for these

[uranusjr]

Unused?

[potiuk]

Closing, as this has been already implemented and merged in #25663

[subkanthi]

Decorator to execute functions in k8s using KubernetesPodOperator

closes: #19135

@task.kubernetes(image='python:3.8-slim-buster', name='k8s_test', namespace='default')
def k8s_decorator_func():
print("decorator func")

If no parameters passed, defaults are set in kubernetes.py

 # Image, name and namespace are all required.
if not 'image' in kwargs:
kwargs['image'] = 'python:3.8-slim-buster'
if not 'name' in kwargs:
kwargs['name'] = f'k8s_airflow_pod_{uuid.uuid4().hex}'
if not 'namespace' in kwargs:
kwargs['namespace'] = 'default'

This version sets the function as an environment variable - PYTHON_SCRIPT and in the command , the logic is to retrieve the environment variable and write to '/tmp/script.py' to execute the function(very similar to Docker)

1. Initially proceeded with trying to mount a local file using the hostPath, the problem is that requires Airflow also to run in the same node. For example running airflow locally and connecting to the kind cluster would need a mount.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Also need to add image to the argument list of _KubernetesDecoratedOperator

[ashb]

This has to be in the kube provider folder, else it won't get released with the provider, and it won't work.

[subkanthi]

@ashb , currently the decorator uses the write_python_script function thats in the utils/python_virtualenv.py file and thats why the template(python_kubernetes_script.jinja2) was also placed in the same folder. If we want to remove the dependency on the utils folder, I guess we can make a copy of the write_python_script in the cncf.providers.kubernetes folder. Question is if there a benefit of having a central place for the function and all templates, Docker decorator and python operator also call this function.

def write_python_script(
jinja_context: dict,
filename: str,
render_template_as_native_obj: bool = False,
template_file: str = 'python_virtualenv_script.jinja2',
):

[Asciotti]

Bump? :)

[kaxil]

Please also rebase your PR?

[kaxil]

Can you move this @subkanthi to the cncf kube providers please?

[uranusjr]

What can go into kwargs here? It seems unnecessary.

[uranusjr]

Nit, it’s easier to use setdefault for these

[uranusjr]

Unused?

[potiuk]

Closing, as this has been already implemented and merged in #25663

[subkanthi]

Decorator to execute functions in k8s using KubernetesPodOperator

closes: #19135

@task.kubernetes(image='python:3.8-slim-buster', name='k8s_test', namespace='default')
def k8s_decorator_func():
print("decorator func")

If no parameters passed, defaults are set in kubernetes.py

 # Image, name and namespace are all required.
if not 'image' in kwargs:
kwargs['image'] = 'python:3.8-slim-buster'
if not 'name' in kwargs:
kwargs['name'] = f'k8s_airflow_pod_{uuid.uuid4().hex}'
if not 'namespace' in kwargs:
kwargs['namespace'] = 'default'

This version sets the function as an environment variable - PYTHON_SCRIPT and in the command , the logic is to retrieve the environment variable and write to '/tmp/script.py' to execute the function(very similar to Docker)

1. Initially proceeded with trying to mount a local file using the hostPath, the problem is that requires Airflow also to run in the same node. For example running airflow locally and connecting to the kind cluster would need a mount.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Also need to add image to the argument list of _KubernetesDecoratedOperator

[ashb]

This has to be in the kube provider folder, else it won't get released with the provider, and it won't work.

[subkanthi]

@ashb , currently the decorator uses the write_python_script function thats in the utils/python_virtualenv.py file and thats why the template(python_kubernetes_script.jinja2) was also placed in the same folder. If we want to remove the dependency on the utils folder, I guess we can make a copy of the write_python_script in the cncf.providers.kubernetes folder. Question is if there a benefit of having a central place for the function and all templates, Docker decorator and python operator also call this function.

def write_python_script(
jinja_context: dict,
filename: str,
render_template_as_native_obj: bool = False,
template_file: str = 'python_virtualenv_script.jinja2',
):

[Asciotti]

Bump? :)

[kaxil]

Please also rebase your PR?

[kaxil]

Can you move this @subkanthi to the cncf kube providers please?

[uranusjr]

What can go into kwargs here? It seems unnecessary.

[uranusjr]

Nit, it’s easier to use setdefault for these

[uranusjr]

Unused?

[potiuk]

Closing, as this has been already implemented and merged in #25663

[subkanthi]

Decorator to execute functions in k8s using KubernetesPodOperator

closes: #19135

@task.kubernetes(image='python:3.8-slim-buster', name='k8s_test', namespace='default')
def k8s_decorator_func():
print("decorator func")

If no parameters passed, defaults are set in kubernetes.py

 # Image, name and namespace are all required.
if not 'image' in kwargs:
kwargs['image'] = 'python:3.8-slim-buster'
if not 'name' in kwargs:
kwargs['name'] = f'k8s_airflow_pod_{uuid.uuid4().hex}'
if not 'namespace' in kwargs:
kwargs['namespace'] = 'default'

This version sets the function as an environment variable - PYTHON_SCRIPT and in the command , the logic is to retrieve the environment variable and write to '/tmp/script.py' to execute the function(very similar to Docker)

1. Initially proceeded with trying to mount a local file using the hostPath, the problem is that requires Airflow also to run in the same node. For example running airflow locally and connecting to the kind cluster would need a mount.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[uranusjr]

Also need to add image to the argument list of _KubernetesDecoratedOperator

[ashb]

This has to be in the kube provider folder, else it won't get released with the provider, and it won't work.

[subkanthi]

@ashb , currently the decorator uses the write_python_script function thats in the utils/python_virtualenv.py file and thats why the template(python_kubernetes_script.jinja2) was also placed in the same folder. If we want to remove the dependency on the utils folder, I guess we can make a copy of the write_python_script in the cncf.providers.kubernetes folder. Question is if there a benefit of having a central place for the function and all templates, Docker decorator and python operator also call this function.

def write_python_script(
jinja_context: dict,
filename: str,
render_template_as_native_obj: bool = False,
template_file: str = 'python_virtualenv_script.jinja2',
):

[Asciotti]

Bump? :)

[kaxil]

Please also rebase your PR?

[kaxil]

Can you move this @subkanthi to the cncf kube providers please?

[uranusjr]

What can go into kwargs here? It seems unnecessary.

[uranusjr]

Nit, it’s easier to use setdefault for these

[uranusjr]

Unused?

[potiuk]

Closing, as this has been already implemented and merged in #25663