Add BigQuery routine operators and existence sensor

[alamashir]

Summary

Adds first-class Airflow operators for BigQuery routines (user-defined functions, stored procedures, table-valued and aggregate functions), so DAGs can own their routine definitions declaratively instead of embedding CREATE FUNCTION / CREATE PROCEDURE DDL inside BigQueryInsertJobOperator.

New operators (providers/google/src/airflow/providers/google/cloud/operators/bigquery.py):

• BigQueryCreateRoutineOperator — creates any routine type; if_exists of fail | skip | replace
• BigQueryUpdateRoutineOperator — updates selected fields on an existing routine. Because BigQuery's routines.update REST API is a full-resource PUT (not a PATCH), the hook fetches the current routine, merges the requested changes, and writes the complete resource back.
• BigQueryDeleteRoutineOperator — ignore_if_missing toggle
• BigQueryGetRoutineOperator — pushes the serialized Routine resource to XCom
• BigQueryListRoutinesOperator — dataset-scoped list via XCom

New sensor (sensors/bigquery.py):

• BigQueryRoutineExistenceSensor

New hook methods on BigQueryHook: create_routine, update_routine, delete_routine, get_routine, list_routines. They wrap google-cloud-bigquery's existing routine client APIs — no new provider dependency.

Docs: new bigquery_routines.rst guide with per-routine-type examples, cross-linked from bigquery.rst.

System test: example_bigquery_routines.py exercises the full lifecycle (create scalar UDF / procedure / TVF → sensor → update → get → list → delete).

Unit tests: 24 tests covering success, failure and edge cases (e.g. if_exists branches, missing routineReference, not_found_ok).

closes: #65467
related: #47579

Test plan

• uv run --project providers/google pytest providers/google/tests/unit/google/cloud/{hooks,operators,sensors}/test_bigquery.py -k Routine — 24 passed
• ruff format + ruff check clean on all touched files
• System test (example_bigquery_routines.py) run end-to-end against a live GCP project — passed (DAG state success, ~5 min run). The system run exercised every operator and the sensor against real BigQuery:
  • create_dataset → created a fresh dataset
  • create_scalar_routine / create_procedure / create_tvf → created SCALAR_FUNCTION, PROCEDURE, and TABLE_VALUED_FUNCTION routines via BigQueryCreateRoutineOperator
  • wait_for_routine → BigQueryRoutineExistenceSensor polled and returned True once the scalar UDF was visible
  • update_routine → mutated the scalar UDF's description via BigQueryUpdateRoutineOperator (this is what flushed out the PUT-vs-PATCH semantic and the retry-sentinel bug fixed in this PR)
  • get_routine / list_routines → fetched the updated routine and listed all three routines
  • delete_scalar_routine / delete_procedure / delete_tvf → deleted each routine via BigQueryDeleteRoutineOperator
  • delete_dataset → tore down the dataset
  • The DAG's watcher task (trigger_rule=ONE_FAILED) did not fire, confirming no upstream failures

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Opus 4.7 (Claude Code)

Generated-by: Claude Opus 4.7 (Claude Code) following the guidelines

[potiuk]

@alamashir Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.

• ❌ Unresolved review comments (3 threads from maintainers): please walk through each unresolved review thread. Even if a suggestion looks incorrect or irrelevant — and some of them will be, especially any comments left by automated reviewers like GitHub Copilot — it is still the author's responsibility to respond: apply the fix, reply in-thread with a brief explanation of why the suggestion does not apply, or resolve the thread if the feedback is no longer relevant. Leaving threads unaddressed for weeks blocks the PR from moving forward.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[potiuk]

Quick follow-up to the triage comment above — one clarification on the "Unresolved review comments" item:

Once you believe a thread has been addressed — whether by pushing a fix, or by replying in-thread with an explanation of why the suggestion doesn't apply — please mark the thread as resolved yourself by clicking the "Resolve conversation" button at the bottom of each thread. Reviewers don't auto-close their own threads, so an addressed-but-unresolved thread reads as "still waiting on the author" and keeps the PR from moving forward. The author doing the resolve-click is the expected convention on this project.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[alamashir]

Hi @potiuk — thanks for the triage and the clarification on thread-resolution convention.

Addressed:

• All 3 review threads resolved (each had an in-thread reply).
• prek run --from-ref main --stage pre-commit run locally — all hooks passed.

Marking ready for review.

[shahar1]

Overall: nearly there, two things to address before merge.

---

Bug: BigQueryUpdateRoutineOperator — fields uses the wrong namespace, causing silent no-ops

The fields parameter is documented and tested with Python SDK attribute names ("body", "type_"), but the merge loop in update_routine operates on the camelCase REST API repr (updates_repr):

forfieldinfields:
iffieldinupdates_repr: # camelCase keys only ("definitionBody", not "body")merged_repr[field] =updates_repr[field]
else:
merged_repr.pop(field, None) # also camelCase; Python names are no-ops here

A user following the operator docstring example and passing fields=["body"] will get a silent no-op: the key "body" isn't in updates_repr (the correct key is "definitionBody"), so neither branch does anything, and the routine is unchanged. The API call succeeds with no error.

The system test uses only fields=["description"], which is identical in both namespaces and therefore masks the issue. The operator unit test uses fields=["description", "body"] — the "body" entry would silently do nothing on a real hook.

Fix: the fields parameter should use camelCase REST names throughout ("definitionBody", "routineType", etc.). Update the operator docstring, the operator unit test, and the hook docstring to reflect this. _ROUTINE_WRITABLE_PROPERTIES is an internal constant passed to client.update_routine() and correctly uses Python attribute names — that part is fine.

---

Missing changelog entry

AGENTS.md asks that new user-visible provider changes be recorded directly in providers/google/docs/changelog.rst. This PR adds 5 operators and a sensor — no changelog entry is present.

---

Minor (non-blocking)

• first, second = MagicMock(), MagicMock() in TestBigQueryListRoutinesOperator.test_execute lacks spec=. Project guidelines require spec/autospec on all mocks.

---

Drafted-by: GitHub Copilot (Claude Sonnet 4.6); reviewed by @shahar1 before posting

[alamashir]

Thanks for the review @shahar1!

Fields namespace bug — Fixed in 4fbc662. Updated the operator docstring to use ["definitionBody", "description"] instead of ["body", "description"], and updated the unit test accordingly.

Changelog entry — Added in 810e499.

Missing spec= — Fixed in 4fbc662. Added spec=Routine to the mocks in TestBigQueryListRoutinesOperator.

[shahar1]

LGTM, good to merge once CI is green

[alamashir]

Summary

Adds first-class Airflow operators for BigQuery routines (user-defined functions, stored procedures, table-valued and aggregate functions), so DAGs can own their routine definitions declaratively instead of embedding CREATE FUNCTION / CREATE PROCEDURE DDL inside BigQueryInsertJobOperator.

New operators (providers/google/src/airflow/providers/google/cloud/operators/bigquery.py):

• BigQueryCreateRoutineOperator — creates any routine type; if_exists of fail | skip | replace
• BigQueryUpdateRoutineOperator — updates selected fields on an existing routine. Because BigQuery's routines.update REST API is a full-resource PUT (not a PATCH), the hook fetches the current routine, merges the requested changes, and writes the complete resource back.
• BigQueryDeleteRoutineOperator — ignore_if_missing toggle
• BigQueryGetRoutineOperator — pushes the serialized Routine resource to XCom
• BigQueryListRoutinesOperator — dataset-scoped list via XCom

New sensor (sensors/bigquery.py):

• BigQueryRoutineExistenceSensor

New hook methods on BigQueryHook: create_routine, update_routine, delete_routine, get_routine, list_routines. They wrap google-cloud-bigquery's existing routine client APIs — no new provider dependency.

Docs: new bigquery_routines.rst guide with per-routine-type examples, cross-linked from bigquery.rst.

System test: example_bigquery_routines.py exercises the full lifecycle (create scalar UDF / procedure / TVF → sensor → update → get → list → delete).

Unit tests: 24 tests covering success, failure and edge cases (e.g. if_exists branches, missing routineReference, not_found_ok).

closes: #65467
related: #47579

Test plan

• uv run --project providers/google pytest providers/google/tests/unit/google/cloud/{hooks,operators,sensors}/test_bigquery.py -k Routine — 24 passed
• ruff format + ruff check clean on all touched files
• System test (example_bigquery_routines.py) run end-to-end against a live GCP project — passed (DAG state success, ~5 min run). The system run exercised every operator and the sensor against real BigQuery:
  • create_dataset → created a fresh dataset
  • create_scalar_routine / create_procedure / create_tvf → created SCALAR_FUNCTION, PROCEDURE, and TABLE_VALUED_FUNCTION routines via BigQueryCreateRoutineOperator
  • wait_for_routine → BigQueryRoutineExistenceSensor polled and returned True once the scalar UDF was visible
  • update_routine → mutated the scalar UDF's description via BigQueryUpdateRoutineOperator (this is what flushed out the PUT-vs-PATCH semantic and the retry-sentinel bug fixed in this PR)
  • get_routine / list_routines → fetched the updated routine and listed all three routines
  • delete_scalar_routine / delete_procedure / delete_tvf → deleted each routine via BigQueryDeleteRoutineOperator
  • delete_dataset → tore down the dataset
  • The DAG's watcher task (trigger_rule=ONE_FAILED) did not fire, confirming no upstream failures

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Opus 4.7 (Claude Code)

Generated-by: Claude Opus 4.7 (Claude Code) following the guidelines

[potiuk]

@alamashir Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.

• ❌ Unresolved review comments (3 threads from maintainers): please walk through each unresolved review thread. Even if a suggestion looks incorrect or irrelevant — and some of them will be, especially any comments left by automated reviewers like GitHub Copilot — it is still the author's responsibility to respond: apply the fix, reply in-thread with a brief explanation of why the suggestion does not apply, or resolve the thread if the feedback is no longer relevant. Leaving threads unaddressed for weeks blocks the PR from moving forward.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[potiuk]

Quick follow-up to the triage comment above — one clarification on the "Unresolved review comments" item:

Once you believe a thread has been addressed — whether by pushing a fix, or by replying in-thread with an explanation of why the suggestion doesn't apply — please mark the thread as resolved yourself by clicking the "Resolve conversation" button at the bottom of each thread. Reviewers don't auto-close their own threads, so an addressed-but-unresolved thread reads as "still waiting on the author" and keeps the PR from moving forward. The author doing the resolve-click is the expected convention on this project.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[alamashir]

Hi @potiuk — thanks for the triage and the clarification on thread-resolution convention.

Addressed:

• All 3 review threads resolved (each had an in-thread reply).
• prek run --from-ref main --stage pre-commit run locally — all hooks passed.

Marking ready for review.

[shahar1]

Overall: nearly there, two things to address before merge.

---

Bug: BigQueryUpdateRoutineOperator — fields uses the wrong namespace, causing silent no-ops

The fields parameter is documented and tested with Python SDK attribute names ("body", "type_"), but the merge loop in update_routine operates on the camelCase REST API repr (updates_repr):

forfieldinfields:
iffieldinupdates_repr: # camelCase keys only ("definitionBody", not "body")merged_repr[field] =updates_repr[field]
else:
merged_repr.pop(field, None) # also camelCase; Python names are no-ops here

A user following the operator docstring example and passing fields=["body"] will get a silent no-op: the key "body" isn't in updates_repr (the correct key is "definitionBody"), so neither branch does anything, and the routine is unchanged. The API call succeeds with no error.

The system test uses only fields=["description"], which is identical in both namespaces and therefore masks the issue. The operator unit test uses fields=["description", "body"] — the "body" entry would silently do nothing on a real hook.

Fix: the fields parameter should use camelCase REST names throughout ("definitionBody", "routineType", etc.). Update the operator docstring, the operator unit test, and the hook docstring to reflect this. _ROUTINE_WRITABLE_PROPERTIES is an internal constant passed to client.update_routine() and correctly uses Python attribute names — that part is fine.

---

Missing changelog entry

AGENTS.md asks that new user-visible provider changes be recorded directly in providers/google/docs/changelog.rst. This PR adds 5 operators and a sensor — no changelog entry is present.

---

Minor (non-blocking)

• first, second = MagicMock(), MagicMock() in TestBigQueryListRoutinesOperator.test_execute lacks spec=. Project guidelines require spec/autospec on all mocks.

---

Drafted-by: GitHub Copilot (Claude Sonnet 4.6); reviewed by @shahar1 before posting

[alamashir]

Thanks for the review @shahar1!

Fields namespace bug — Fixed in 4fbc662. Updated the operator docstring to use ["definitionBody", "description"] instead of ["body", "description"], and updated the unit test accordingly.

Changelog entry — Added in 810e499.

Missing spec= — Fixed in 4fbc662. Added spec=Routine to the mocks in TestBigQueryListRoutinesOperator.

[shahar1]

LGTM, good to merge once CI is green

[alamashir]

Summary

Adds first-class Airflow operators for BigQuery routines (user-defined functions, stored procedures, table-valued and aggregate functions), so DAGs can own their routine definitions declaratively instead of embedding CREATE FUNCTION / CREATE PROCEDURE DDL inside BigQueryInsertJobOperator.

New operators (providers/google/src/airflow/providers/google/cloud/operators/bigquery.py):

• BigQueryCreateRoutineOperator — creates any routine type; if_exists of fail | skip | replace
• BigQueryUpdateRoutineOperator — updates selected fields on an existing routine. Because BigQuery's routines.update REST API is a full-resource PUT (not a PATCH), the hook fetches the current routine, merges the requested changes, and writes the complete resource back.
• BigQueryDeleteRoutineOperator — ignore_if_missing toggle
• BigQueryGetRoutineOperator — pushes the serialized Routine resource to XCom
• BigQueryListRoutinesOperator — dataset-scoped list via XCom

New sensor (sensors/bigquery.py):

• BigQueryRoutineExistenceSensor

New hook methods on BigQueryHook: create_routine, update_routine, delete_routine, get_routine, list_routines. They wrap google-cloud-bigquery's existing routine client APIs — no new provider dependency.

Docs: new bigquery_routines.rst guide with per-routine-type examples, cross-linked from bigquery.rst.

System test: example_bigquery_routines.py exercises the full lifecycle (create scalar UDF / procedure / TVF → sensor → update → get → list → delete).

Unit tests: 24 tests covering success, failure and edge cases (e.g. if_exists branches, missing routineReference, not_found_ok).

closes: #65467
related: #47579

Test plan

• uv run --project providers/google pytest providers/google/tests/unit/google/cloud/{hooks,operators,sensors}/test_bigquery.py -k Routine — 24 passed
• ruff format + ruff check clean on all touched files
• System test (example_bigquery_routines.py) run end-to-end against a live GCP project — passed (DAG state success, ~5 min run). The system run exercised every operator and the sensor against real BigQuery:
  • create_dataset → created a fresh dataset
  • create_scalar_routine / create_procedure / create_tvf → created SCALAR_FUNCTION, PROCEDURE, and TABLE_VALUED_FUNCTION routines via BigQueryCreateRoutineOperator
  • wait_for_routine → BigQueryRoutineExistenceSensor polled and returned True once the scalar UDF was visible
  • update_routine → mutated the scalar UDF's description via BigQueryUpdateRoutineOperator (this is what flushed out the PUT-vs-PATCH semantic and the retry-sentinel bug fixed in this PR)
  • get_routine / list_routines → fetched the updated routine and listed all three routines
  • delete_scalar_routine / delete_procedure / delete_tvf → deleted each routine via BigQueryDeleteRoutineOperator
  • delete_dataset → tore down the dataset
  • The DAG's watcher task (trigger_rule=ONE_FAILED) did not fire, confirming no upstream failures

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Opus 4.7 (Claude Code)

Generated-by: Claude Opus 4.7 (Claude Code) following the guidelines

[potiuk]

@alamashir Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.

• ❌ Unresolved review comments (3 threads from maintainers): please walk through each unresolved review thread. Even if a suggestion looks incorrect or irrelevant — and some of them will be, especially any comments left by automated reviewers like GitHub Copilot — it is still the author's responsibility to respond: apply the fix, reply in-thread with a brief explanation of why the suggestion does not apply, or resolve the thread if the feedback is no longer relevant. Leaving threads unaddressed for weeks blocks the PR from moving forward.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[potiuk]

Quick follow-up to the triage comment above — one clarification on the "Unresolved review comments" item:

Once you believe a thread has been addressed — whether by pushing a fix, or by replying in-thread with an explanation of why the suggestion doesn't apply — please mark the thread as resolved yourself by clicking the "Resolve conversation" button at the bottom of each thread. Reviewers don't auto-close their own threads, so an addressed-but-unresolved thread reads as "still waiting on the author" and keeps the PR from moving forward. The author doing the resolve-click is the expected convention on this project.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[alamashir]

Hi @potiuk — thanks for the triage and the clarification on thread-resolution convention.

Addressed:

• All 3 review threads resolved (each had an in-thread reply).
• prek run --from-ref main --stage pre-commit run locally — all hooks passed.

Marking ready for review.

[shahar1]

Overall: nearly there, two things to address before merge.

---

Bug: BigQueryUpdateRoutineOperator — fields uses the wrong namespace, causing silent no-ops

The fields parameter is documented and tested with Python SDK attribute names ("body", "type_"), but the merge loop in update_routine operates on the camelCase REST API repr (updates_repr):

forfieldinfields:
iffieldinupdates_repr: # camelCase keys only ("definitionBody", not "body")merged_repr[field] =updates_repr[field]
else:
merged_repr.pop(field, None) # also camelCase; Python names are no-ops here

A user following the operator docstring example and passing fields=["body"] will get a silent no-op: the key "body" isn't in updates_repr (the correct key is "definitionBody"), so neither branch does anything, and the routine is unchanged. The API call succeeds with no error.

The system test uses only fields=["description"], which is identical in both namespaces and therefore masks the issue. The operator unit test uses fields=["description", "body"] — the "body" entry would silently do nothing on a real hook.

Fix: the fields parameter should use camelCase REST names throughout ("definitionBody", "routineType", etc.). Update the operator docstring, the operator unit test, and the hook docstring to reflect this. _ROUTINE_WRITABLE_PROPERTIES is an internal constant passed to client.update_routine() and correctly uses Python attribute names — that part is fine.

---

Missing changelog entry

AGENTS.md asks that new user-visible provider changes be recorded directly in providers/google/docs/changelog.rst. This PR adds 5 operators and a sensor — no changelog entry is present.

---

Minor (non-blocking)

• first, second = MagicMock(), MagicMock() in TestBigQueryListRoutinesOperator.test_execute lacks spec=. Project guidelines require spec/autospec on all mocks.

---

Drafted-by: GitHub Copilot (Claude Sonnet 4.6); reviewed by @shahar1 before posting

[alamashir]

Thanks for the review @shahar1!

Fields namespace bug — Fixed in 4fbc662. Updated the operator docstring to use ["definitionBody", "description"] instead of ["body", "description"], and updated the unit test accordingly.

Changelog entry — Added in 810e499.

Missing spec= — Fixed in 4fbc662. Added spec=Routine to the mocks in TestBigQueryListRoutinesOperator.

[shahar1]

LGTM, good to merge once CI is green

[alamashir]

Summary

Adds first-class Airflow operators for BigQuery routines (user-defined functions, stored procedures, table-valued and aggregate functions), so DAGs can own their routine definitions declaratively instead of embedding CREATE FUNCTION / CREATE PROCEDURE DDL inside BigQueryInsertJobOperator.

New operators (providers/google/src/airflow/providers/google/cloud/operators/bigquery.py):

• BigQueryCreateRoutineOperator — creates any routine type; if_exists of fail | skip | replace
• BigQueryUpdateRoutineOperator — updates selected fields on an existing routine. Because BigQuery's routines.update REST API is a full-resource PUT (not a PATCH), the hook fetches the current routine, merges the requested changes, and writes the complete resource back.
• BigQueryDeleteRoutineOperator — ignore_if_missing toggle
• BigQueryGetRoutineOperator — pushes the serialized Routine resource to XCom
• BigQueryListRoutinesOperator — dataset-scoped list via XCom

New sensor (sensors/bigquery.py):

• BigQueryRoutineExistenceSensor

New hook methods on BigQueryHook: create_routine, update_routine, delete_routine, get_routine, list_routines. They wrap google-cloud-bigquery's existing routine client APIs — no new provider dependency.

Docs: new bigquery_routines.rst guide with per-routine-type examples, cross-linked from bigquery.rst.

System test: example_bigquery_routines.py exercises the full lifecycle (create scalar UDF / procedure / TVF → sensor → update → get → list → delete).

Unit tests: 24 tests covering success, failure and edge cases (e.g. if_exists branches, missing routineReference, not_found_ok).

closes: #65467
related: #47579

Test plan

• uv run --project providers/google pytest providers/google/tests/unit/google/cloud/{hooks,operators,sensors}/test_bigquery.py -k Routine — 24 passed
• ruff format + ruff check clean on all touched files
• System test (example_bigquery_routines.py) run end-to-end against a live GCP project — passed (DAG state success, ~5 min run). The system run exercised every operator and the sensor against real BigQuery:
  • create_dataset → created a fresh dataset
  • create_scalar_routine / create_procedure / create_tvf → created SCALAR_FUNCTION, PROCEDURE, and TABLE_VALUED_FUNCTION routines via BigQueryCreateRoutineOperator
  • wait_for_routine → BigQueryRoutineExistenceSensor polled and returned True once the scalar UDF was visible
  • update_routine → mutated the scalar UDF's description via BigQueryUpdateRoutineOperator (this is what flushed out the PUT-vs-PATCH semantic and the retry-sentinel bug fixed in this PR)
  • get_routine / list_routines → fetched the updated routine and listed all three routines
  • delete_scalar_routine / delete_procedure / delete_tvf → deleted each routine via BigQueryDeleteRoutineOperator
  • delete_dataset → tore down the dataset
  • The DAG's watcher task (trigger_rule=ONE_FAILED) did not fire, confirming no upstream failures

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Opus 4.7 (Claude Code)

Generated-by: Claude Opus 4.7 (Claude Code) following the guidelines

[potiuk]

@alamashir Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.

• ❌ Unresolved review comments (3 threads from maintainers): please walk through each unresolved review thread. Even if a suggestion looks incorrect or irrelevant — and some of them will be, especially any comments left by automated reviewers like GitHub Copilot — it is still the author's responsibility to respond: apply the fix, reply in-thread with a brief explanation of why the suggestion does not apply, or resolve the thread if the feedback is no longer relevant. Leaving threads unaddressed for weeks blocks the PR from moving forward.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[potiuk]

Quick follow-up to the triage comment above — one clarification on the "Unresolved review comments" item:

Once you believe a thread has been addressed — whether by pushing a fix, or by replying in-thread with an explanation of why the suggestion doesn't apply — please mark the thread as resolved yourself by clicking the "Resolve conversation" button at the bottom of each thread. Reviewers don't auto-close their own threads, so an addressed-but-unresolved thread reads as "still waiting on the author" and keeps the PR from moving forward. The author doing the resolve-click is the expected convention on this project.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[alamashir]

Hi @potiuk — thanks for the triage and the clarification on thread-resolution convention.

Addressed:

• All 3 review threads resolved (each had an in-thread reply).
• prek run --from-ref main --stage pre-commit run locally — all hooks passed.

Marking ready for review.

[shahar1]

Overall: nearly there, two things to address before merge.

---

Bug: BigQueryUpdateRoutineOperator — fields uses the wrong namespace, causing silent no-ops

The fields parameter is documented and tested with Python SDK attribute names ("body", "type_"), but the merge loop in update_routine operates on the camelCase REST API repr (updates_repr):

forfieldinfields:
iffieldinupdates_repr: # camelCase keys only ("definitionBody", not "body")merged_repr[field] =updates_repr[field]
else:
merged_repr.pop(field, None) # also camelCase; Python names are no-ops here

A user following the operator docstring example and passing fields=["body"] will get a silent no-op: the key "body" isn't in updates_repr (the correct key is "definitionBody"), so neither branch does anything, and the routine is unchanged. The API call succeeds with no error.

The system test uses only fields=["description"], which is identical in both namespaces and therefore masks the issue. The operator unit test uses fields=["description", "body"] — the "body" entry would silently do nothing on a real hook.

Fix: the fields parameter should use camelCase REST names throughout ("definitionBody", "routineType", etc.). Update the operator docstring, the operator unit test, and the hook docstring to reflect this. _ROUTINE_WRITABLE_PROPERTIES is an internal constant passed to client.update_routine() and correctly uses Python attribute names — that part is fine.

---

Missing changelog entry

AGENTS.md asks that new user-visible provider changes be recorded directly in providers/google/docs/changelog.rst. This PR adds 5 operators and a sensor — no changelog entry is present.

---

Minor (non-blocking)

• first, second = MagicMock(), MagicMock() in TestBigQueryListRoutinesOperator.test_execute lacks spec=. Project guidelines require spec/autospec on all mocks.

---

Drafted-by: GitHub Copilot (Claude Sonnet 4.6); reviewed by @shahar1 before posting

[alamashir]

Thanks for the review @shahar1!

Fields namespace bug — Fixed in 4fbc662. Updated the operator docstring to use ["definitionBody", "description"] instead of ["body", "description"], and updated the unit test accordingly.

Changelog entry — Added in 810e499.

Missing spec= — Fixed in 4fbc662. Added spec=Routine to the mocks in TestBigQueryListRoutinesOperator.

[shahar1]

LGTM, good to merge once CI is green

[alamashir]

Summary

Adds first-class Airflow operators for BigQuery routines (user-defined functions, stored procedures, table-valued and aggregate functions), so DAGs can own their routine definitions declaratively instead of embedding CREATE FUNCTION / CREATE PROCEDURE DDL inside BigQueryInsertJobOperator.

New operators (providers/google/src/airflow/providers/google/cloud/operators/bigquery.py):

• BigQueryCreateRoutineOperator — creates any routine type; if_exists of fail | skip | replace
• BigQueryUpdateRoutineOperator — updates selected fields on an existing routine. Because BigQuery's routines.update REST API is a full-resource PUT (not a PATCH), the hook fetches the current routine, merges the requested changes, and writes the complete resource back.
• BigQueryDeleteRoutineOperator — ignore_if_missing toggle
• BigQueryGetRoutineOperator — pushes the serialized Routine resource to XCom
• BigQueryListRoutinesOperator — dataset-scoped list via XCom

New sensor (sensors/bigquery.py):

• BigQueryRoutineExistenceSensor

New hook methods on BigQueryHook: create_routine, update_routine, delete_routine, get_routine, list_routines. They wrap google-cloud-bigquery's existing routine client APIs — no new provider dependency.

Docs: new bigquery_routines.rst guide with per-routine-type examples, cross-linked from bigquery.rst.

System test: example_bigquery_routines.py exercises the full lifecycle (create scalar UDF / procedure / TVF → sensor → update → get → list → delete).

Unit tests: 24 tests covering success, failure and edge cases (e.g. if_exists branches, missing routineReference, not_found_ok).

closes: #65467
related: #47579

Test plan

• uv run --project providers/google pytest providers/google/tests/unit/google/cloud/{hooks,operators,sensors}/test_bigquery.py -k Routine — 24 passed
• ruff format + ruff check clean on all touched files
• System test (example_bigquery_routines.py) run end-to-end against a live GCP project — passed (DAG state success, ~5 min run). The system run exercised every operator and the sensor against real BigQuery:
  • create_dataset → created a fresh dataset
  • create_scalar_routine / create_procedure / create_tvf → created SCALAR_FUNCTION, PROCEDURE, and TABLE_VALUED_FUNCTION routines via BigQueryCreateRoutineOperator
  • wait_for_routine → BigQueryRoutineExistenceSensor polled and returned True once the scalar UDF was visible
  • update_routine → mutated the scalar UDF's description via BigQueryUpdateRoutineOperator (this is what flushed out the PUT-vs-PATCH semantic and the retry-sentinel bug fixed in this PR)
  • get_routine / list_routines → fetched the updated routine and listed all three routines
  • delete_scalar_routine / delete_procedure / delete_tvf → deleted each routine via BigQueryDeleteRoutineOperator
  • delete_dataset → tore down the dataset
  • The DAG's watcher task (trigger_rule=ONE_FAILED) did not fire, confirming no upstream failures

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Opus 4.7 (Claude Code)

Generated-by: Claude Opus 4.7 (Claude Code) following the guidelines

[potiuk]

@alamashir Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.

• ❌ Unresolved review comments (3 threads from maintainers): please walk through each unresolved review thread. Even if a suggestion looks incorrect or irrelevant — and some of them will be, especially any comments left by automated reviewers like GitHub Copilot — it is still the author's responsibility to respond: apply the fix, reply in-thread with a brief explanation of why the suggestion does not apply, or resolve the thread if the feedback is no longer relevant. Leaving threads unaddressed for weeks blocks the PR from moving forward.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[potiuk]

Quick follow-up to the triage comment above — one clarification on the "Unresolved review comments" item:

Once you believe a thread has been addressed — whether by pushing a fix, or by replying in-thread with an explanation of why the suggestion doesn't apply — please mark the thread as resolved yourself by clicking the "Resolve conversation" button at the bottom of each thread. Reviewers don't auto-close their own threads, so an addressed-but-unresolved thread reads as "still waiting on the author" and keeps the PR from moving forward. The author doing the resolve-click is the expected convention on this project.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[alamashir]

Hi @potiuk — thanks for the triage and the clarification on thread-resolution convention.

Addressed:

• All 3 review threads resolved (each had an in-thread reply).
• prek run --from-ref main --stage pre-commit run locally — all hooks passed.

Marking ready for review.

[shahar1]

Overall: nearly there, two things to address before merge.

---

Bug: BigQueryUpdateRoutineOperator — fields uses the wrong namespace, causing silent no-ops

The fields parameter is documented and tested with Python SDK attribute names ("body", "type_"), but the merge loop in update_routine operates on the camelCase REST API repr (updates_repr):

forfieldinfields:
iffieldinupdates_repr: # camelCase keys only ("definitionBody", not "body")merged_repr[field] =updates_repr[field]
else:
merged_repr.pop(field, None) # also camelCase; Python names are no-ops here

A user following the operator docstring example and passing fields=["body"] will get a silent no-op: the key "body" isn't in updates_repr (the correct key is "definitionBody"), so neither branch does anything, and the routine is unchanged. The API call succeeds with no error.

The system test uses only fields=["description"], which is identical in both namespaces and therefore masks the issue. The operator unit test uses fields=["description", "body"] — the "body" entry would silently do nothing on a real hook.

Fix: the fields parameter should use camelCase REST names throughout ("definitionBody", "routineType", etc.). Update the operator docstring, the operator unit test, and the hook docstring to reflect this. _ROUTINE_WRITABLE_PROPERTIES is an internal constant passed to client.update_routine() and correctly uses Python attribute names — that part is fine.

---

Missing changelog entry

AGENTS.md asks that new user-visible provider changes be recorded directly in providers/google/docs/changelog.rst. This PR adds 5 operators and a sensor — no changelog entry is present.

---

Minor (non-blocking)

• first, second = MagicMock(), MagicMock() in TestBigQueryListRoutinesOperator.test_execute lacks spec=. Project guidelines require spec/autospec on all mocks.

---

Drafted-by: GitHub Copilot (Claude Sonnet 4.6); reviewed by @shahar1 before posting

[alamashir]

Thanks for the review @shahar1!

Fields namespace bug — Fixed in 4fbc662. Updated the operator docstring to use ["definitionBody", "description"] instead of ["body", "description"], and updated the unit test accordingly.

Changelog entry — Added in 810e499.

Missing spec= — Fixed in 4fbc662. Added spec=Routine to the mocks in TestBigQueryListRoutinesOperator.

[shahar1]

LGTM, good to merge once CI is green

[alamashir]

Summary

Adds first-class Airflow operators for BigQuery routines (user-defined functions, stored procedures, table-valued and aggregate functions), so DAGs can own their routine definitions declaratively instead of embedding CREATE FUNCTION / CREATE PROCEDURE DDL inside BigQueryInsertJobOperator.

New operators (providers/google/src/airflow/providers/google/cloud/operators/bigquery.py):

• BigQueryCreateRoutineOperator — creates any routine type; if_exists of fail | skip | replace
• BigQueryUpdateRoutineOperator — updates selected fields on an existing routine. Because BigQuery's routines.update REST API is a full-resource PUT (not a PATCH), the hook fetches the current routine, merges the requested changes, and writes the complete resource back.
• BigQueryDeleteRoutineOperator — ignore_if_missing toggle
• BigQueryGetRoutineOperator — pushes the serialized Routine resource to XCom
• BigQueryListRoutinesOperator — dataset-scoped list via XCom

New sensor (sensors/bigquery.py):

• BigQueryRoutineExistenceSensor

New hook methods on BigQueryHook: create_routine, update_routine, delete_routine, get_routine, list_routines. They wrap google-cloud-bigquery's existing routine client APIs — no new provider dependency.

Docs: new bigquery_routines.rst guide with per-routine-type examples, cross-linked from bigquery.rst.

System test: example_bigquery_routines.py exercises the full lifecycle (create scalar UDF / procedure / TVF → sensor → update → get → list → delete).

Unit tests: 24 tests covering success, failure and edge cases (e.g. if_exists branches, missing routineReference, not_found_ok).

closes: #65467
related: #47579

Test plan

• uv run --project providers/google pytest providers/google/tests/unit/google/cloud/{hooks,operators,sensors}/test_bigquery.py -k Routine — 24 passed
• ruff format + ruff check clean on all touched files
• System test (example_bigquery_routines.py) run end-to-end against a live GCP project — passed (DAG state success, ~5 min run). The system run exercised every operator and the sensor against real BigQuery:
  • create_dataset → created a fresh dataset
  • create_scalar_routine / create_procedure / create_tvf → created SCALAR_FUNCTION, PROCEDURE, and TABLE_VALUED_FUNCTION routines via BigQueryCreateRoutineOperator
  • wait_for_routine → BigQueryRoutineExistenceSensor polled and returned True once the scalar UDF was visible
  • update_routine → mutated the scalar UDF's description via BigQueryUpdateRoutineOperator (this is what flushed out the PUT-vs-PATCH semantic and the retry-sentinel bug fixed in this PR)
  • get_routine / list_routines → fetched the updated routine and listed all three routines
  • delete_scalar_routine / delete_procedure / delete_tvf → deleted each routine via BigQueryDeleteRoutineOperator
  • delete_dataset → tore down the dataset
  • The DAG's watcher task (trigger_rule=ONE_FAILED) did not fire, confirming no upstream failures

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Opus 4.7 (Claude Code)

Generated-by: Claude Opus 4.7 (Claude Code) following the guidelines

[potiuk]

@alamashir Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.

• ❌ Unresolved review comments (3 threads from maintainers): please walk through each unresolved review thread. Even if a suggestion looks incorrect or irrelevant — and some of them will be, especially any comments left by automated reviewers like GitHub Copilot — it is still the author's responsibility to respond: apply the fix, reply in-thread with a brief explanation of why the suggestion does not apply, or resolve the thread if the feedback is no longer relevant. Leaving threads unaddressed for weeks blocks the PR from moving forward.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[potiuk]

Quick follow-up to the triage comment above — one clarification on the "Unresolved review comments" item:

Once you believe a thread has been addressed — whether by pushing a fix, or by replying in-thread with an explanation of why the suggestion doesn't apply — please mark the thread as resolved yourself by clicking the "Resolve conversation" button at the bottom of each thread. Reviewers don't auto-close their own threads, so an addressed-but-unresolved thread reads as "still waiting on the author" and keeps the PR from moving forward. The author doing the resolve-click is the expected convention on this project.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[alamashir]

Hi @potiuk — thanks for the triage and the clarification on thread-resolution convention.

Addressed:

• All 3 review threads resolved (each had an in-thread reply).
• prek run --from-ref main --stage pre-commit run locally — all hooks passed.

Marking ready for review.

[shahar1]

Overall: nearly there, two things to address before merge.

---

Bug: BigQueryUpdateRoutineOperator — fields uses the wrong namespace, causing silent no-ops

The fields parameter is documented and tested with Python SDK attribute names ("body", "type_"), but the merge loop in update_routine operates on the camelCase REST API repr (updates_repr):

forfieldinfields:
iffieldinupdates_repr: # camelCase keys only ("definitionBody", not "body")merged_repr[field] =updates_repr[field]
else:
merged_repr.pop(field, None) # also camelCase; Python names are no-ops here

A user following the operator docstring example and passing fields=["body"] will get a silent no-op: the key "body" isn't in updates_repr (the correct key is "definitionBody"), so neither branch does anything, and the routine is unchanged. The API call succeeds with no error.

The system test uses only fields=["description"], which is identical in both namespaces and therefore masks the issue. The operator unit test uses fields=["description", "body"] — the "body" entry would silently do nothing on a real hook.

Fix: the fields parameter should use camelCase REST names throughout ("definitionBody", "routineType", etc.). Update the operator docstring, the operator unit test, and the hook docstring to reflect this. _ROUTINE_WRITABLE_PROPERTIES is an internal constant passed to client.update_routine() and correctly uses Python attribute names — that part is fine.

---

Missing changelog entry

AGENTS.md asks that new user-visible provider changes be recorded directly in providers/google/docs/changelog.rst. This PR adds 5 operators and a sensor — no changelog entry is present.

---

Minor (non-blocking)

• first, second = MagicMock(), MagicMock() in TestBigQueryListRoutinesOperator.test_execute lacks spec=. Project guidelines require spec/autospec on all mocks.

---

Drafted-by: GitHub Copilot (Claude Sonnet 4.6); reviewed by @shahar1 before posting

[alamashir]

Thanks for the review @shahar1!

Fields namespace bug — Fixed in 4fbc662. Updated the operator docstring to use ["definitionBody", "description"] instead of ["body", "description"], and updated the unit test accordingly.

Changelog entry — Added in 810e499.

Missing spec= — Fixed in 4fbc662. Added spec=Routine to the mocks in TestBigQueryListRoutinesOperator.

[shahar1]

LGTM, good to merge once CI is green

[alamashir]

Summary

Adds first-class Airflow operators for BigQuery routines (user-defined functions, stored procedures, table-valued and aggregate functions), so DAGs can own their routine definitions declaratively instead of embedding CREATE FUNCTION / CREATE PROCEDURE DDL inside BigQueryInsertJobOperator.

New operators (providers/google/src/airflow/providers/google/cloud/operators/bigquery.py):

• BigQueryCreateRoutineOperator — creates any routine type; if_exists of fail | skip | replace
• BigQueryUpdateRoutineOperator — updates selected fields on an existing routine. Because BigQuery's routines.update REST API is a full-resource PUT (not a PATCH), the hook fetches the current routine, merges the requested changes, and writes the complete resource back.
• BigQueryDeleteRoutineOperator — ignore_if_missing toggle
• BigQueryGetRoutineOperator — pushes the serialized Routine resource to XCom
• BigQueryListRoutinesOperator — dataset-scoped list via XCom

New sensor (sensors/bigquery.py):

• BigQueryRoutineExistenceSensor

New hook methods on BigQueryHook: create_routine, update_routine, delete_routine, get_routine, list_routines. They wrap google-cloud-bigquery's existing routine client APIs — no new provider dependency.

Docs: new bigquery_routines.rst guide with per-routine-type examples, cross-linked from bigquery.rst.

System test: example_bigquery_routines.py exercises the full lifecycle (create scalar UDF / procedure / TVF → sensor → update → get → list → delete).

Unit tests: 24 tests covering success, failure and edge cases (e.g. if_exists branches, missing routineReference, not_found_ok).

closes: #65467
related: #47579

Test plan

• uv run --project providers/google pytest providers/google/tests/unit/google/cloud/{hooks,operators,sensors}/test_bigquery.py -k Routine — 24 passed
• ruff format + ruff check clean on all touched files
• System test (example_bigquery_routines.py) run end-to-end against a live GCP project — passed (DAG state success, ~5 min run). The system run exercised every operator and the sensor against real BigQuery:
  • create_dataset → created a fresh dataset
  • create_scalar_routine / create_procedure / create_tvf → created SCALAR_FUNCTION, PROCEDURE, and TABLE_VALUED_FUNCTION routines via BigQueryCreateRoutineOperator
  • wait_for_routine → BigQueryRoutineExistenceSensor polled and returned True once the scalar UDF was visible
  • update_routine → mutated the scalar UDF's description via BigQueryUpdateRoutineOperator (this is what flushed out the PUT-vs-PATCH semantic and the retry-sentinel bug fixed in this PR)
  • get_routine / list_routines → fetched the updated routine and listed all three routines
  • delete_scalar_routine / delete_procedure / delete_tvf → deleted each routine via BigQueryDeleteRoutineOperator
  • delete_dataset → tore down the dataset
  • The DAG's watcher task (trigger_rule=ONE_FAILED) did not fire, confirming no upstream failures

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Opus 4.7 (Claude Code)

Generated-by: Claude Opus 4.7 (Claude Code) following the guidelines

[potiuk]

@alamashir Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.

• ❌ Unresolved review comments (3 threads from maintainers): please walk through each unresolved review thread. Even if a suggestion looks incorrect or irrelevant — and some of them will be, especially any comments left by automated reviewers like GitHub Copilot — it is still the author's responsibility to respond: apply the fix, reply in-thread with a brief explanation of why the suggestion does not apply, or resolve the thread if the feedback is no longer relevant. Leaving threads unaddressed for weeks blocks the PR from moving forward.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[potiuk]

Quick follow-up to the triage comment above — one clarification on the "Unresolved review comments" item:

Once you believe a thread has been addressed — whether by pushing a fix, or by replying in-thread with an explanation of why the suggestion doesn't apply — please mark the thread as resolved yourself by clicking the "Resolve conversation" button at the bottom of each thread. Reviewers don't auto-close their own threads, so an addressed-but-unresolved thread reads as "still waiting on the author" and keeps the PR from moving forward. The author doing the resolve-click is the expected convention on this project.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[alamashir]

Hi @potiuk — thanks for the triage and the clarification on thread-resolution convention.

Addressed:

• All 3 review threads resolved (each had an in-thread reply).
• prek run --from-ref main --stage pre-commit run locally — all hooks passed.

Marking ready for review.

[shahar1]

Overall: nearly there, two things to address before merge.

---

Bug: BigQueryUpdateRoutineOperator — fields uses the wrong namespace, causing silent no-ops

The fields parameter is documented and tested with Python SDK attribute names ("body", "type_"), but the merge loop in update_routine operates on the camelCase REST API repr (updates_repr):

forfieldinfields:
iffieldinupdates_repr: # camelCase keys only ("definitionBody", not "body")merged_repr[field] =updates_repr[field]
else:
merged_repr.pop(field, None) # also camelCase; Python names are no-ops here

A user following the operator docstring example and passing fields=["body"] will get a silent no-op: the key "body" isn't in updates_repr (the correct key is "definitionBody"), so neither branch does anything, and the routine is unchanged. The API call succeeds with no error.

The system test uses only fields=["description"], which is identical in both namespaces and therefore masks the issue. The operator unit test uses fields=["description", "body"] — the "body" entry would silently do nothing on a real hook.

Fix: the fields parameter should use camelCase REST names throughout ("definitionBody", "routineType", etc.). Update the operator docstring, the operator unit test, and the hook docstring to reflect this. _ROUTINE_WRITABLE_PROPERTIES is an internal constant passed to client.update_routine() and correctly uses Python attribute names — that part is fine.

---

Missing changelog entry

AGENTS.md asks that new user-visible provider changes be recorded directly in providers/google/docs/changelog.rst. This PR adds 5 operators and a sensor — no changelog entry is present.

---

Minor (non-blocking)

• first, second = MagicMock(), MagicMock() in TestBigQueryListRoutinesOperator.test_execute lacks spec=. Project guidelines require spec/autospec on all mocks.

---

Drafted-by: GitHub Copilot (Claude Sonnet 4.6); reviewed by @shahar1 before posting

[alamashir]

Thanks for the review @shahar1!

Fields namespace bug — Fixed in 4fbc662. Updated the operator docstring to use ["definitionBody", "description"] instead of ["body", "description"], and updated the unit test accordingly.

Changelog entry — Added in 810e499.

Missing spec= — Fixed in 4fbc662. Added spec=Routine to the mocks in TestBigQueryListRoutinesOperator.

[shahar1]

LGTM, good to merge once CI is green

[alamashir]

Summary

Adds first-class Airflow operators for BigQuery routines (user-defined functions, stored procedures, table-valued and aggregate functions), so DAGs can own their routine definitions declaratively instead of embedding CREATE FUNCTION / CREATE PROCEDURE DDL inside BigQueryInsertJobOperator.

New operators (providers/google/src/airflow/providers/google/cloud/operators/bigquery.py):

• BigQueryCreateRoutineOperator — creates any routine type; if_exists of fail | skip | replace
• BigQueryUpdateRoutineOperator — updates selected fields on an existing routine. Because BigQuery's routines.update REST API is a full-resource PUT (not a PATCH), the hook fetches the current routine, merges the requested changes, and writes the complete resource back.
• BigQueryDeleteRoutineOperator — ignore_if_missing toggle
• BigQueryGetRoutineOperator — pushes the serialized Routine resource to XCom
• BigQueryListRoutinesOperator — dataset-scoped list via XCom

New sensor (sensors/bigquery.py):

• BigQueryRoutineExistenceSensor

New hook methods on BigQueryHook: create_routine, update_routine, delete_routine, get_routine, list_routines. They wrap google-cloud-bigquery's existing routine client APIs — no new provider dependency.

Docs: new bigquery_routines.rst guide with per-routine-type examples, cross-linked from bigquery.rst.

System test: example_bigquery_routines.py exercises the full lifecycle (create scalar UDF / procedure / TVF → sensor → update → get → list → delete).

Unit tests: 24 tests covering success, failure and edge cases (e.g. if_exists branches, missing routineReference, not_found_ok).

closes: #65467
related: #47579

Test plan

• uv run --project providers/google pytest providers/google/tests/unit/google/cloud/{hooks,operators,sensors}/test_bigquery.py -k Routine — 24 passed
• ruff format + ruff check clean on all touched files
• System test (example_bigquery_routines.py) run end-to-end against a live GCP project — passed (DAG state success, ~5 min run). The system run exercised every operator and the sensor against real BigQuery:
  • create_dataset → created a fresh dataset
  • create_scalar_routine / create_procedure / create_tvf → created SCALAR_FUNCTION, PROCEDURE, and TABLE_VALUED_FUNCTION routines via BigQueryCreateRoutineOperator
  • wait_for_routine → BigQueryRoutineExistenceSensor polled and returned True once the scalar UDF was visible
  • update_routine → mutated the scalar UDF's description via BigQueryUpdateRoutineOperator (this is what flushed out the PUT-vs-PATCH semantic and the retry-sentinel bug fixed in this PR)
  • get_routine / list_routines → fetched the updated routine and listed all three routines
  • delete_scalar_routine / delete_procedure / delete_tvf → deleted each routine via BigQueryDeleteRoutineOperator
  • delete_dataset → tore down the dataset
  • The DAG's watcher task (trigger_rule=ONE_FAILED) did not fire, confirming no upstream failures

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Opus 4.7 (Claude Code)

Generated-by: Claude Opus 4.7 (Claude Code) following the guidelines

[potiuk]

@alamashir Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.

• ❌ Unresolved review comments (3 threads from maintainers): please walk through each unresolved review thread. Even if a suggestion looks incorrect or irrelevant — and some of them will be, especially any comments left by automated reviewers like GitHub Copilot — it is still the author's responsibility to respond: apply the fix, reply in-thread with a brief explanation of why the suggestion does not apply, or resolve the thread if the feedback is no longer relevant. Leaving threads unaddressed for weeks blocks the PR from moving forward.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[potiuk]

Quick follow-up to the triage comment above — one clarification on the "Unresolved review comments" item:

Once you believe a thread has been addressed — whether by pushing a fix, or by replying in-thread with an explanation of why the suggestion doesn't apply — please mark the thread as resolved yourself by clicking the "Resolve conversation" button at the bottom of each thread. Reviewers don't auto-close their own threads, so an addressed-but-unresolved thread reads as "still waiting on the author" and keeps the PR from moving forward. The author doing the resolve-click is the expected convention on this project.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[alamashir]

Hi @potiuk — thanks for the triage and the clarification on thread-resolution convention.

Addressed:

• All 3 review threads resolved (each had an in-thread reply).
• prek run --from-ref main --stage pre-commit run locally — all hooks passed.

Marking ready for review.

[shahar1]

Overall: nearly there, two things to address before merge.

---

Bug: BigQueryUpdateRoutineOperator — fields uses the wrong namespace, causing silent no-ops

The fields parameter is documented and tested with Python SDK attribute names ("body", "type_"), but the merge loop in update_routine operates on the camelCase REST API repr (updates_repr):

forfieldinfields:
iffieldinupdates_repr: # camelCase keys only ("definitionBody", not "body")merged_repr[field] =updates_repr[field]
else:
merged_repr.pop(field, None) # also camelCase; Python names are no-ops here

A user following the operator docstring example and passing fields=["body"] will get a silent no-op: the key "body" isn't in updates_repr (the correct key is "definitionBody"), so neither branch does anything, and the routine is unchanged. The API call succeeds with no error.

The system test uses only fields=["description"], which is identical in both namespaces and therefore masks the issue. The operator unit test uses fields=["description", "body"] — the "body" entry would silently do nothing on a real hook.

Fix: the fields parameter should use camelCase REST names throughout ("definitionBody", "routineType", etc.). Update the operator docstring, the operator unit test, and the hook docstring to reflect this. _ROUTINE_WRITABLE_PROPERTIES is an internal constant passed to client.update_routine() and correctly uses Python attribute names — that part is fine.

---

Missing changelog entry

AGENTS.md asks that new user-visible provider changes be recorded directly in providers/google/docs/changelog.rst. This PR adds 5 operators and a sensor — no changelog entry is present.

---

Minor (non-blocking)

• first, second = MagicMock(), MagicMock() in TestBigQueryListRoutinesOperator.test_execute lacks spec=. Project guidelines require spec/autospec on all mocks.

---

Drafted-by: GitHub Copilot (Claude Sonnet 4.6); reviewed by @shahar1 before posting

[alamashir]

Thanks for the review @shahar1!

Fields namespace bug — Fixed in 4fbc662. Updated the operator docstring to use ["definitionBody", "description"] instead of ["body", "description"], and updated the unit test accordingly.

Changelog entry — Added in 810e499.

Missing spec= — Fixed in 4fbc662. Added spec=Routine to the mocks in TestBigQueryListRoutinesOperator.

[shahar1]

LGTM, good to merge once CI is green