Skip to content

[WIP] Improve Kubernetes System Tests Iteration Experience - #47595

Closed
jason810496 wants to merge 9 commits into
apache:mainfrom
jason810496:refactor/improve-k8s-tests-iteration-experience
Closed

[WIP] Improve Kubernetes System Tests Iteration Experience#47595
jason810496 wants to merge 9 commits into
apache:mainfrom
jason810496:refactor/improve-k8s-tests-iteration-experience

Conversation

@jason810496

@jason810496jason810496 commented Mar 11, 2025

Copy link
Copy Markdown
Member

related: #44508

Why

Improving the Kubernetes development experience, as described in #44508.

How

  • Introduce dev mode in the Helm chart
    • Add airflow-code-pvc PVC
    • Mount airflow-code-pvc as $AIRFLOW_HOME/airflow to include the source code
  • Introduce the breeze k8s reload command
    • Copy the current source code from the local machine to airflow-code-pvc
    • Roll out a restart for all Airflow components

What

  • Significantly improves the k8s test iteration process
    • Don't need to rebuild image and upload to kind cluster repeatedly.
  • Could potentially speed up CI in k8s system tests
    • Eliminates the need to rebuild the k8s image with breeze k8s build-k8s-image --rebuild-base-image for every change (only required when dependencies change)
    • Allows loading the latest code changes using breeze k8s reload

@boring-cyborgboring-cyborgBot added the area:helm-chart Airflow Helm Chart label Mar 11, 2025
@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actionsgithub-actionsBot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Apr 26, 2025
@potiukpotiuk removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Apr 26, 2025
@potiuk

Copy link
Copy Markdown
Member

I 'd encourage to continue it :)

@jason810496

Copy link
Copy Markdown
MemberAuthor

I 'd encourage to continue it :)

So do I, but I still working on #49470 ( rework resolving OOM for api-server when reading large logs ) and almost finish it, and will continue on this one.

@jason810496
jason810496force-pushed the refactor/improve-k8s-tests-iteration-experience branch 4 times, most recently from eae849f to 78ff971CompareMay 13, 2025 02:49
@jason810496
jason810496force-pushed the refactor/improve-k8s-tests-iteration-experience branch from 78ff971 to 75c0cd7CompareMay 28, 2025 13:47
@jason810496
jason810496force-pushed the refactor/improve-k8s-tests-iteration-experience branch from 75c0cd7 to d074196CompareJune 1, 2025 07:07
@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actionsgithub-actionsBot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Jul 27, 2025
@jason810496

Copy link
Copy Markdown
MemberAuthor

up.

I still stuck with a few conference recently but I will keep work this later.
Before this PR, I will add --dev flag for hot-reload support for all components startup commands.

@github-actionsgithub-actionsBot removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Jul 28, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:helm-chartAirflow Helm Chart

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@jason810496@potiuk