Add support for forwarding CeleryExecutor task logs to stdout

[jayachandrakasarla]

Closes#69554

Currently, CeleryExecutor task logs are not forwarded to stdout because this behavior is hardcoded to false in BaseExecutor.run_workload().

This PR allows users to enable log forwarding by setting AIRFLOW__CELERY__TASK_LOGS_TO_STDOUT=True. By default, this setting remains false to preserve the existing behavior. When enabled, task logs are forwarded to stdout, allowing container-level log collectors to capture them.

Below is a screenshot of the Celery worker logs showing the task's stdout output.

Was generative AI tooling used to co-author this PR?

• Yes (Claude Sonnet 4.6)

[multimeric]

Note that this has been attempted before, so worth checking that your PR satisfies all the review here: #64481

[github-actions]

uv.lock on main just moved via #69854 ("Bump datamodel-code-generator pin from 0.33.0 to 0.41.0"), commit 15a4478 and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

[jayachandrakasarla]

@potiuk@ashb could you please review this PR?

[jayachandrakasarla]

@kaxil could you please review this?

[potiuk]

The feature is worth having and the shape is right — a core default with a worker-level override, mirroring [celery] json_logs / [logging] json_logs down to the != "none" guard, which matches the existing precedent in celery_command.py rather than inventing a new convention. The version gating on the tests is also done properly: the pre-3.3 and 3.3+ paths each have their own skipif, so both compat legs actually exercise their branch.

Two things to fix before merge, and one doc line.

[logging] task_logs_to_stdout silently does nothing on Airflow 3.3

The None sentinel only means "fall back to the core setting" on an Airflow that carries this PR's BaseExecutor.run_workload() change. Against the versions this provider supports, that lands unevenly:

Airflow	Path	[celery] task_logs_to_stdout	[logging] task_logs_to_stdout
3.0–3.2	_execute_workload_pre_3_3	works	works — the provider resolves the fallback itself
3.3	execute_workload → run_workload	works	ignored — 3.3's run_workload signature is subprocess_logs_to_stdout: bool = False, so the None is simply falsy and the core key is never consulted
3.4+	execute_workload → run_workload	works	works

A user on 3.3 who sets the global key sees nothing happen, with no warning — and it worked on 3.2 and will work again on 3.4. From their side that reads as a bug rather than a version boundary.

The fix is the pattern you already use in the pre-3.3 branch: resolve the fallback in execute_workload and pass a concrete bool to run_workload. The core None handling then stays as a convenience for other executors, instead of being load-bearing for Celery.

Missing newsfragment

This adds a new user-facing core config option ([logging] task_logs_to_stdout, version_added: 3.4.0), so it needs airflow-core/newsfragments/69597.feature.rst. Nothing is required on the provider side — provider changelogs are regenerated from git log.

Smaller observations

• docs/celery_executor.rst — the note says "Airflow 3+ only", which is right for the [celery] key but not for the [logging] one. Worth stating that the global key requires Airflow 3.4+ (and, if you take the fix above, that it works on 3.0+ through the provider). The json_logs section sets the precedent with its "[logging] json_logs was added in Airflow 3.2.0" note.
• The two-level lookup now exists twice in this provider — inline in celery_command.py for json_logs, and as _celery_task_logs_to_stdout_override() here. A small shared helper taking the option name would collapse both. Not blocking.

---

This review was drafted by an AI-assisted tool and
confirmed by an Airflow maintainer. After you've
addressed the points above and pushed an update, an Airflow
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

[jayachandrakasarla]

@potiuk I've made the changes, please review

[jayachandrakasarla]

@kaxil@potiuk could you please review this PR?