Add deferrable support to `InfluxDB3Operator` by subhramit · Pull Request #71976 · apache/airflow

subhramit · 2026-08-22T10:58:13Z

Closes#67107
Follow-up to #58929

Add deferrable support to InfluxDB3Operator

When deferrable=True, the operator now defers to a new InfluxDB3QueryTrigger so the worker slot is released while the query runs. The trigger awaits InfluxDBClient3.query_async() and resumes the task with the same JSON-serializable record shape returned by the synchronous path.
Also update provider metadata and documentation to surface the trigger, add a deferrable example, and bump the influxdb3-python minimum version to >=0.12.0 (the release that introduced query_async()).

Note that this deviates from the issue's proposed API as I did not add a "poll_interval"/polling loop. InfluxDB 3's query_async streams the full result over one Arrow Flight call, which means there's no server-side job to poll a status on, unlike Snowflake/BigQuery/Redshift. The trigger awaits the query once and emits a single event.
This is more similar in shape to SQLExecuteQueryTrigger (used by GenericTransfer's deferrable path) - which also has a single await yielding one TriggerEvent with no poll_interval for the same reason:

airflow/providers/common/sql/src/airflow/providers/common/sql/triggers/sql.py

Lines 92 to 102 in 4e4d060

asyncdefrun(self) ->AsyncIterator[TriggerEvent]: 
try: 
self.log.info("Extracting data from %s", self.conn_id) 
self.log.info("Executing: \n %s", self.sql) 
self.log.info("Reading records from %s", self.conn_id) 
results=awaitself._get_records() 
self.log.info("Reading records from %s done!", self.conn_id) 
self.log.debug("results: %s", results) 
yieldTriggerEvent({"status": "success", "results": results}) 

Have added tests for the new deferrable operator path, trigger serialization/execution, and async hook behavior.

MWE

Tested on a setup of influxdb:3-core with a file object store, two rows written to a home table over the v3 line-protocol endpoint, read back through the influxdb3 CLI:

The async client on its own against the same server, to isolate it from Airflow:

c=InfluxDBClient3(host="http://localhost:8181", token=TOKEN, database="airflow_demo")
print(asyncio.run(c.query_async("SELECT * FROM home")).to_pydict())

Then through the operator - the deferrable task alongside the sync one running the same query, so the two paths could be compared:

blocking=InfluxDB3Operator(task_id="query_blocking", sql=SQL, influxdb3_conn_id=CONN)
deferred=InfluxDB3Operator(task_id="query_deferred", sql=SQL, influxdb3_conn_id=CONN, deferrable=True)
blocking>>deferred

query_deferred deferred to the triggerer, the trigger fired a success event, and the task resumed from QUEUED with the same two records the sync task returned:

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: GPT-5.4 following the guidelines
All code changes done as a result (and also this description) were manually driven, edited reviewed by me.

Signed-off-by: subhramit <subhramit.bb@live.in>

subhramit · 2026-08-22T12:38:33Z

Ah, I think I missed a lowest-direct-dependencies edge case here. The spellcheck also fails, will address both in the next commit.

Update - fixed.

Signed-off-by: Subhramit Basu <subhramit.bb@live.in>

subhramit · 2026-08-23T12:58:16Z

cc'ing @eladkal for visibility, in case any specific codeowners are to be requested for review.

eladkal · 2026-08-23T17:34:45Z

cc @arpitrathore can you take a look?

subhramit · 2026-08-23T17:53:07Z

cc @arpitrathore can you take a look?

+1 I hope... I have cc'ed them in the issue as well, but they seem to have been away from GitHub since quite some time now after their last comment expressing that they wanted to work on this.

subhramit · 2026-08-25T08:35:14Z

cc @arpitrathore can you take a look?
+1 I hope... I have cc'ed them in the issue as well, but they seem to have been away from GitHub since quite some time now after their last comment expressing that they wanted to work on this.

Adding some other recent committers in case the PR gets lost

@bbovenzi, @Lee-W, @hussein-awala,@kaxil,
@amoghrajesh, @o-nikolas,@uranusjr, @jason810496, @vincbeck,
@jedcunningham, @potiuk if they can help review

After this is done, I wish to work on #67109 as a follow-up.

eladkal

left few comments.
Can you please confirm if this was tested against a real influx environment?

eladkal · 2026-08-27T12:27:09Z

+ :param deferrable: Run the query from the triggerer instead of holding a worker slot for its
+ duration. Requires ``influxdb3-python>=0.12.0``. Note that InfluxDB 3 streams results over
+ Arrow Flight rather than exposing a job that can be polled, so the whole result set still flows back
+ through XCom -- deferring helps with long-running queries returning modest result sets
+ (aggregations, freshness probes), not with very large extracts.


Please cleanup here. Explain just what is needed.
Requires influxdb3-python>=0.12.0 is not helpful. The provider already set this min version so users don't need to make sure they hae it.
To be honest I am not sure I understand what this warning means.

Was just meaning to say that since the InfluxDB 3 client doesn’t expose a poll-based async job model here (see PR desc for details), deferrable=True releases the worker slot while the query runs in the triggerer, but the full result still comes back through XCom when the task resumes. So the main benefit is for long-running queries with relatively small result sets, not for very large extracts
Refined with better wording now

eladkal · 2026-08-27T13:28:50Z

+ InfluxDB 3 streams query results over a single Arrow Flight call rather than exposing a job that
+ can be polled, so the trigger awaits the query once instead of polling at an interval, and there


This seems to be a statement over how Influxdb operate. Can you please update the info with a link to the Influx docs? This is needed to confirm that are assumptions are right and in case that Influx change how it works we can track the change/

Added, and improved the statements a bit for clarity

eladkal · 2026-08-27T13:28:57Z

+ useful for long-running queries with small-to-moderate result sets. For very large extracts,
+ keep using :class:`~airflow.providers.influxdb.hooks.influxdb3.InfluxDB3Hook` from a Python task.
+
+ Deferrable mode requires ``influxdb3-python>=0.12.0`` and a running ``triggerer``.


can be removed

Signed-off-by: Subhramit Basu <subhramit.bb@live.in>

…db3.py Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>

Signed-off-by: subhramit <subhramit.bb@live.in>

subhramit · 2026-08-27T20:10:26Z

Can you please confirm if this was tested against a real influx environment?

Yeah.
Can give an MWE as well.

Edit - added to the PR description.

Signed-off-by: subhramit <subhramit.bb@live.in>

Signed-off-by: Subhramit Basu <subhramit.bb@live.in>

Signed-off-by: subhramit <subhramit.bb@live.in>

subhramit · 2026-08-27T22:07:45Z

@eladkal I noticed a separate dependency issue while working on this. InfluxDB3Hook.query() and query_async() import pandas at runtime, but the provider doesn't declare a it as an extra or convert missing-pandas imports into an optional-feature error, which means users would hit a ModuleNotFoundError when using those query paths if they don't have pandas installed.

I took the opportunity and covered that in 1b7294b by adding the pandas extra and raising AirflowOptionalProviderFeatureException from those paths when pandas isn’t installed, like how it’s done in

airflow/providers/apache/hive/src/airflow/providers/apache/hive/hooks/hive.py

Lines 1083 to 1087 in ee15456

importpandasaspd
exceptImportErrorase: 
fromairflow.providers.common.compat.sdkimportAirflowOptionalProviderFeatureException
raiseAirflowOptionalProviderFeatureException(e) 

and also in transfers/sql_to_s3.py

Signed-off-by: Subhramit Basu <subhramit.bb@live.in>

subhramit · 2026-08-29T18:12:43Z

@eladkal @arpitrathore gentle nudge.

ashb

This pr also needs to add async get_conn support please(as fetching a connection on a worker requires network traffic so can block)

(Non exhaustive review)

ashb · 2026-08-30T12:08:20Z

+ single async query call. The trigger therefore awaits one query call instead of polling at an
+ interval, and there is no ``poll_interval`` parameter.


Not sure we need to mention there's no poll interval in the docs. It makes sense in the pr as there issue talked about it, but just going to the docs most people will have no context on this (and its not universal on deferrable operators either )

Fair, removed

ashb · 2026-08-30T12:10:34Z

 from airflow.models import Connection


+class InfluxDB3AsyncQueryNotAvailableError(RuntimeError):


Given we depend on an updated version of theof influx client is this possible to hit?

Was being defensive but yeah ideally this should never hit. Removed.

ashb · 2026-08-30T12:13:16Z

+ f"Result type: {type(result).__module__}.{type(result).__name__}"
+ )
+
+ return await asyncio.to_thread(_convert_dataframe_to_records, result)


Why the different return type on Async? The sync path returns the pandas object directly doesn't it?

Serialization constraint, but youre right it's in the wrong place, the DataFrame cant cross a TriggerEvent, so I converted in the hook. It should be the trigger's job. Moving it there.

ashb · 2026-08-30T12:13:45Z

 from airflow.sdk.definitions.context import Context


+def _convert_dataframe_to_records(dataframe: pd.DataFrame) -> list[dict[str, Any]]:


Duplication from the hook fn

@eladkal wanted this to be private in #71976 (comment), so I wasn't sure what to do as the only other option was to create a private helper utils file for this tiny one-line method.
Maybe when I work on the sensor I'll need it anyway so I'll move this to a file now.

Since python allows it, I relaxed module encapsulation a bit, but only inside the provider's internal implementation to not create a new file and still reuse it in e6a177c. See if that is alright or we should indeed go with a new file

ashb · 2026-08-30T12:18:10Z

+ :param query: SQL query string
+ :return: List of dictionaries representing query results
+ """
+ client = await asyncio.to_thread(self.get_conn)


Those does move it off the main loop but isn't the best way of doing this.
Please look in other hooks for an awit self.aget_conn() or similar

Was unaware of this, thanks. Changed to that async-hook pattern

ashb · 2026-08-30T12:19:44Z

+ task_id="query_data_deferrable",
+ sql="SELECT * FROM \"temperature\" WHERE time > now() - INTERVAL '1 hour'",
+ influxdb3_conn_id="influxdb3_default",
+ deferrable=True,


Deferrable is true for other operators that support it isn't it?

Not by default, in providers the usual pattern is tp follow the global Airflow setting and fall back to false (see airbyte example)
Keeping it true here also matches other provider system examples for the async variant (see S3 and livy examples)

ashb · 2026-08-30T12:21:15Z

+ """Test async query with InfluxDB 3.x."""
+ pd = pytest.importorskip("pandas")
+
+ self.influxdb3_hook.client = mock.Mock()


If at all reasonable is rather we didn't mock the client, but instead the http transport/reaponse underneath it.
At the very least, this mock needs a spec parameter to enforce its "shape"

Makes sense, enforced the shape in the mocks.
P.S. I did have a look at pushing the seam lower but for query()/ query_async()
that would mean mocking upsteam client internals rather than the hook contract. IMO point of these tests are to validate our interaction with InfluxDBClient3 and the DataFrame/type handling around it, so I kept them at the client boundary

Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>

Signed-off-by: subhramit <subhramit.bb@live.in>

subhramit added 2 commits August 22, 2026 16:20

Add deferrable support to InfluxDB3Operator
ed39896
Signed-off-by: subhramit <subhramit.bb@live.in>

Bump InfluxDB3 async client version
392f6a7
Signed-off-by: subhramit <subhramit.bb@live.in>

boring-cyborgBot added area:providers kind:documentation provider:influxdb labels Aug 22, 2026

Revert unrelated change in pyproject
c0f7777
Signed-off-by: subhramit <subhramit.bb@live.in>

subhramit changed the title ~~Add deferrable mode to InfluxDB3Operator~~Add deferrable mode to InfluxDB3OperatorAug 22, 2026

Delay pandas import in InfluxDB3 async hook, reword docstring
bc859fa
Signed-off-by: Subhramit Basu <subhramit.bb@live.in>

subhramit mentioned this pull request Aug 22, 2026
InfluxDB3Operator: add deferrable variant #67107
Open
7 tasks

Merge branch 'main' into deferrable-influxdb3-operator
5ac5c4f

This was referenced Aug 25, 2026
Surface provider triggers in the operators and hooks reference #71975
Merged
InfluxDB 3: add a dedicated sensor (data freshness / windowed existence) #67109
Open

eladkal reviewed Aug 27, 2026
View reviewed changes

subhramitand others added 5 commits August 27, 2026 22:01

Merge branch 'main' into deferrable-influxdb3-operator
918c65c

Offload blocking work in InfluxDB3Hook.query_async to a thread
49d64b4
Signed-off-by: Subhramit Basu <subhramit.bb@live.in>

Merge branch 'main' into deferrable-influxdb3-operator
1af576e

Update providers/influxdb/src/airflow/providers/influxdb/hooks/influx…
558854b
…db3.py Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com>

Refine InfluxDB deferrable docs and comments
04204b8
Signed-off-by: subhramit <subhramit.bb@live.in>

subhramit requested a review from eladkal August 27, 2026 20:14

subhramit added 5 commits August 28, 2026 01:59

Fix CI - Keep InfluxDB record conversion helpers private
bb381a1
Signed-off-by: subhramit <subhramit.bb@live.in>

Add pandas extra for InfluxDB query results
1b7294b
Signed-off-by: Subhramit Basu <subhramit.bb@live.in>

Merge branch 'main' into deferrable-influxdb3-operator
9d5162c

Remove redundant version note from InfluxDB3 hook docs
fd20541
Signed-off-by: subhramit <subhramit.bb@live.in>

Enhance docstring
7488a12
Signed-off-by: subhramit <subhramit.bb@live.in>

subhramit changed the title ~~Add deferrable mode to InfluxDB3Operator~~Add deferrable support to InfluxDB3OperatorAug 27, 2026

Run update_providers_build_files.py
4d23d45
Signed-off-by: Subhramit Basu <subhramit.bb@live.in>

ashb reviewed Aug 30, 2026
View reviewed changes

subhramitand others added 7 commits August 30, 2026 18:11

Update providers/influxdb/tests/system/influxdb/example_influxdb3.py
e72d8c7
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>

Apply suggestion from@ashb
b0456f0
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>

Align InfluxDB async query handling
e6a177c
Signed-off-by: subhramit <subhramit.bb@live.in>

Trim operator note
15460da
Signed-off-by: subhramit <subhramit.bb@live.in>

Drop unreachable InfluxDB async guard
0b52bce
Signed-off-by: subhramit <subhramit.bb@live.in>

Use aget_conn in InfluxDB async hook
e19bc85
Signed-off-by: subhramit <subhramit.bb@live.in>

Tighten InfluxDB hook client mocks
bd24c24
Signed-off-by: subhramit <subhramit.bb@live.in>

subhramit requested a review from ashb August 30, 2026 13:47

subhramit added 3 commits August 30, 2026 19:19

Restore comments
566e0a7
Signed-off-by: subhramit <subhramit.bb@live.in>

Merge branch 'main' into deferrable-influxdb3-operator
35d83ca

Merge branch 'main' into deferrable-influxdb3-operator
38a19aa

	asyncdefrun(self) ->AsyncIterator[TriggerEvent]:
	try:
	self.log.info("Extracting data from %s", self.conn_id)
	self.log.info("Executing: \n %s", self.sql)
	self.log.info("Reading records from %s", self.conn_id)

	results=awaitself._get_records()

	self.log.info("Reading records from %s done!", self.conn_id)
	self.log.debug("results: %s", results)
	yieldTriggerEvent({"status": "success", "results": results})

		InfluxDB 3 streams query results over a single Arrow Flight call rather than exposing a job that
		can be polled, so the trigger awaits the query once instead of polling at an interval, and there

		single async query call. The trigger therefore awaits one query call instead of polling at an
		interval, and there is no ``poll_interval`` parameter.

		from airflow.models import Connection


		class InfluxDB3AsyncQueryNotAvailableError(RuntimeError):

		from airflow.sdk.definitions.context import Context


		def _convert_dataframe_to_records(dataframe: pd.DataFrame) -> list[dict[str, Any]]:

Uh oh!

Conversation

subhramit commented Aug 22, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MWE

Was generative AI tooling used to co-author this PR?

Uh oh!

subhramit commented Aug 22, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

subhramit commented Aug 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eladkal commented Aug 23, 2026

Uh oh!

subhramit commented Aug 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

subhramit commented Aug 25, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eladkal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

subhramitAug 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

subhramit commented Aug 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

subhramit commented Aug 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

subhramit commented Aug 29, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ashb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

subhramitAug 30, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

subhramitAug 30, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

subhramit commented Aug 22, 2026•
edited
Loading

subhramit commented Aug 22, 2026•
edited
Loading

subhramit commented Aug 23, 2026•
edited
Loading

subhramit commented Aug 23, 2026•
edited
Loading

subhramit commented Aug 25, 2026•
edited
Loading

subhramitAug 27, 2026•
edited
Loading

subhramit commented Aug 27, 2026•
edited
Loading

subhramit commented Aug 27, 2026•
edited
Loading

subhramit commented Aug 29, 2026•
edited
Loading

subhramitAug 30, 2026•
edited
Loading

subhramitAug 30, 2026•
edited
Loading

subhramitAug 30, 2026•
edited
Loading