Extract Cloud Logging labels from AF3 log path when task_instance is missing in supervisor context by goingforstudying-ctrl · Pull Request #68246 · apache/airflow

goingforstudying-ctrl · 2026-06-08T18:13:24Z

Ran into this while debugging a Cloud Logging setup on GKE — log entries were landing in Stackdriver with empty labels, making it impossible to filter by dag_id or task_id.

Turns out the StackdriverRemoteLogIO.processors proc() closure reads record.task_instance to populate labels, but in AF3's supervisor model the REMOTE_TASK_LOG handler runs in the supervisor process where that attribute is never set. So every log entry from the supervisor just gets empty labels.

This grabs dag_id, task_id, and try_number from the log path instead. AF3's log path template is dag_id=<x>/run_id=<x>/task_id=<x>/attempt=<N>.log — all four fields are already in the path with zero DB access needed.

The fallback only kicks in when task_instance is genuinely missing, so the task-subprocess code path (where task_instance is available) is untouched.

Not sure if run_id should also be turned into a label here — left it out for now since the existing label set doesn't include it and the read-side filtering (bug 2) will need its own fix anyway. Happy to add it if maintainers think it belongs.

relates to #68240

boring-cyborg · 2026-06-08T18:13:30Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

shahar1 · 2026-06-17T07:14:39Z

+ # Parse labels from the structured log path as additional fallback.
+ path_labels = _labels_from_path(str(relative))
+ labels.update(path_labels)


It overrides event-derived labels rather than acting as a "fallback"

Fixed. Path labels now only fill in missing keys rather than calling update() on the whole dict.

Fixed — reworded the comment to avoid the 'fallback' framing. The path labels are now described as filling in missing keys rather than overriding event-derived ones.

shahar1 · 2026-06-17T07:15:01Z

 return "\n".join(messages), page.next_page_token


+def _labels_from_path(relative_path: str) -> dict[str, str]:


should be renamed to _extract_labels_from_path

Renamed to _extract_labels_from_path as requested.

Fixed — renamed the function to _extract_labels_from_path as requested.

It is being referenced from the test using the old name.

shahar1 · 2026-06-17T07:18:03Z

+ elif key == "run_id":
+ # run_id is NOT a standard Stackdriver label yet, but it is used
+ # on the write side via the log path template. Store it so the
+ # read path can filter on it (Bug 2 will wire this up).
+ pass


Currently it seems like a no-op, so I'd rather drop this branch

Removed the no-op pass branch. run_id is now stored in the labels dict like the other keys.

Fixed — removed the no-op branch as requested.

In Airflow 3 the supervisor process runs REMOTE_TASK_LOG handlers, but record.task_instance is never set in supervisor context (it is a task-subprocess concept). When task_instance is missing the proc() closure shipped log entries with empty labels, making Cloud Logging entries unsearchable by dag_id / task_id. Parse dag_id, task_id, and try_number from the structured AF3 log path (dag_id=<x>/run_id=<x>/task_id=<x>/attempt=<N>.log) instead. This requires zero DB access and works regardless of whether the handler runs in a task subprocess or the supervisor. relates to apache#68240 Signed-off-by: goingforstudying-ctrl <goingforstudying-ctrl@users.noreply.github.com>

- Rename _labels_from_path to _extract_labels_from_path - Use path labels as fallback instead of overriding event-derived labels - Actually store run_id instead of no-op pass branch Signed-off-by: goingforstudying-ctrl <goingforstudying-ctrl@users.noreply.github.com>

goingforstudying-ctrl · 2026-06-20T03:27:30Z

Closing as the fix has already been merged via #68292. Thanks for the review.

goingforstudying-ctrl requested a review from shahar1 as a code owner June 8, 2026 18:13

boring-cyborgBot added area:logging area:providers provider:google Google (including GCP) related issues labels Jun 8, 2026

This was referenced Jun 8, 2026
Use run_id instead of logical_date in StackdriverRemoteLogIO.read() filter #68249
Closed
StackdriverRemoteLogIO: three bugs in AF3 supervisor context — empty labels, broken read filter, unguarded send crash #68240
Open

goingforstudying-ctrl force-pushed the fix/stackdriver-labels-from-path branch 23 times, most recently from dc7f1d1 to 6ba2072Compare June 11, 2026 14:50

goingforstudying-ctrl force-pushed the fix/stackdriver-labels-from-path branch 15 times, most recently from c3298d7 to 85ec4b8Compare June 17, 2026 05:26

shahar1 changed the title ~~fix(google): extract Cloud Logging labels from AF3 log path when task_instance is missing in supervisor context~~Extract Cloud Logging labels from AF3 log path when task_instance is missing in supervisor contextJun 17, 2026

shahar1 requested changes Jun 17, 2026
View reviewed changes

goingforstudying-ctrl force-pushed the fix/stackdriver-labels-from-path branch 10 times, most recently from a0c2c9b to 8add617Compare June 18, 2026 02:07

goingforstudying-ctrland others added 2 commits June 19, 2026 20:16

Navigation Menu

Uh oh!

Extract Cloud Logging labels from AF3 log path when task_instance is missing in supervisor context - #68246

Extract Cloud Logging labels from AF3 log path when task_instance is missing in supervisor context#68246
goingforstudying-ctrl wants to merge 2 commits into
apache:mainfrom
goingforstudying-ctrl:fix/stackdriver-labels-from-path

goingforstudying-ctrl commented Jun 8, 2026

Uh oh!

boring-cyborgBot commented Jun 8, 2026

Uh oh!

shahar1Jun 17, 2026

Uh oh!

goingforstudying-ctrlJun 17, 2026

Uh oh!

goingforstudying-ctrlJun 18, 2026

Uh oh!

shahar1Jun 17, 2026

Uh oh!

goingforstudying-ctrlJun 17, 2026

Uh oh!

goingforstudying-ctrlJun 18, 2026

Uh oh!

shahar1Jun 19, 2026

Uh oh!

shahar1Jun 17, 2026

Uh oh!

goingforstudying-ctrlJun 17, 2026

Uh oh!

goingforstudying-ctrlJun 18, 2026

Uh oh!

goingforstudying-ctrl commented Jun 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		return "\n".join(messages), page.next_page_token


		def _labels_from_path(relative_path: str) -> dict[str, str]:

Uh oh!

Conversation

goingforstudying-ctrl commented Jun 8, 2026

Uh oh!

boring-cyborgBot commented Jun 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

goingforstudying-ctrl commented Jun 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants