Webserver timeout randomly

[Jeoffreybauvin]

Apache Airflow version

3.1.8

If "Other Airflow 3 version" selected, which one?

No response

What happened?

I have a dag scheduled every 10 minutes. The 3rd one always fails ....

On worker side I can see some timeouts :

2026-03-25T10:30:01.055984Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:01.056166Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
--- Supervised process Last chance exception handler ---
Traceback (most recent call last):
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1112, in run
 result = _execute_task(context=context, ti=ti, log=log)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1523, in _execute_task
 result = ctx.run(execute, context=context)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 417, in wrapper
 return func(self, *args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/decorator.py", line 252, in execute
 return_value = super().execute(context)
 ^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 417, in wrapper
 return func(self, *args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 228, in execute
 return_value = self.execute_callable()
 ^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/standard/operators/python.py", line 251, in execute_callable
 return runner.run(*self.op_args, **self.op_kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/callback_runner.py", line 82, in run
 return func(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^
 File "/opt/airflow/dags/live/dags/dag_netbox_devices_maintenance.py", line 69, in link_vms
 xcom_value = context["ti"].xcom_pull(
 ^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 366, in xcom_pull
 values = XCom.get_all(
 ^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/bases/xcom.py", line 304, in get_all
 msg = SUPERVISOR_COMMS.send(
 ^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 206, in send
 return self._get_response()
 ^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 269, in _get_response
 frame = self._read_frame()
 ^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/comms.py", line 230, in _read_frame
 len_bytes = self.socket.recv(4)
 ^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/timeout.py", line 37, in handle_timeout
 self.log.error("Process timed out", pid=os.getpid())
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/_shared/logging/structlog.py", line 95, in meth
 return self._proxy_to_logger(name, event, **kw)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_base.py", line 223, in _proxy_to_logger
 return getattr(self._logger, method_name)(*args, **kw)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_output.py", line 321, in msg
 self._write(message + b"\n")
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1186, in run
 log.exception("Task failed with exception")
 File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_native.py", line 47, in exception
 return self.error(event, *args, **kw)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/_shared/logging/structlog.py", line 95, in meth
 return self._proxy_to_logger(name, event, **kw)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_base.py", line 223, in _proxy_to_logger
 return getattr(self._logger, method_name)(*args, **kw)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_output.py", line 321, in msg
 self._write(message + b"\n")
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1697, in main
 state, _, error = run(ti, context, log)
 ^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1191, in run
 f"ti.finish.{ti.dag_id}.{ti.task_id}.{state.value}",
 ^^^^^
UnboundLocalError: cannot access local variable 'state' where it is not associated with a value

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 388, in _fork_main
 target()
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 206, in _subprocess_main
 main()
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1704, in main
 log.exception("Top level error")
 File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_native.py", line 47, in exception
 return self.error(event, *args, **kw)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/_shared/logging/structlog.py", line 95, in meth
 return self._proxy_to_logger(name, event, **kw)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_base.py", line 223, in _proxy_to_logger
 return getattr(self._logger, method_name)(*args, **kw)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/structlog/_output.py", line 321, in msg
 self._write(message + b"\n")
BrokenPipeError: [Errno 32] Broken pipe
2026-03-25T10:30:01.425867Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:02.130218Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730
2026-03-25T10:30:02.130481Z [debug ] heartbeat_tick : Prev sent/recv: 4993/1325006, now - 4993/1325174, monotonic - 420551.394657526, last_heartbeat_sent - 420531.392953123, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748
2026-03-25T10:30:02.130650Z [debug ] heartbeat_tick: sending heartbeat for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:764
2026-03-25T10:30:02.197240Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:06.425046Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:07.196062Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:10.981992Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:10.982286Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:10.982729Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:10.983217Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:30:11.057156Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:11.057421Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:11.057769Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:11.058240Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 1st time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:30:11.425105Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:12.197500Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:14.092457Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': '569d87ec-8b31-3e7b-8124-30a33342604d'} ticket:4198e42b-4207-4725-9e0d-f1eba84c01bb] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:16.425168Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:17.196067Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:20.984262Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:20.996950Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bcd4a090> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:20.997314Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0fd8d50> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:21.000576Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc65aa50> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:21.000826Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.001151Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.001307Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.001478Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.001610Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.059241Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:21.080066Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58af044f80> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:21.080298Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0faedd0> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:21.083181Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58af023d10> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:21.083411Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.083728Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.083900Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.084075Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.084207Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:21.425544Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:22.132032Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730
2026-03-25T10:30:22.132399Z [debug ] heartbeat_tick : Prev sent/recv: 4993/1325174, now - 4994/1325333, monotonic - 420571.396551106, last_heartbeat_sent - 420571.396549525, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748
2026-03-25T10:30:22.197758Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:26.427229Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:27.196171Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:27.871188Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'ec90397f-5794-3113-80d2-7121be96b7b6'} ticket:88299c2e-82b9-4e03-98fd-0323ac37d8fe] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:31.012018Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:31.012272Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:31.012521Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:31.012828Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:30:31.094629Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:31.094944Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:31.095310Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:31.095613Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 2nd time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:30:31.425628Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:32.197584Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:36.425520Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:37.196474Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:41.014457Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.018476Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc673cb0> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.018686Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0fd8d50> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.021958Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc672ff0> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.022183Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.022500Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.022699Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.022894Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.023032Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.096446Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.099030Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58aee0a090> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.099257Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0faedd0> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.102609Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58aee0ae70> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:30:41.102853Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.103176Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.103333Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.103505Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.103636Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:41.413066Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'b96a3d63-ef49-35b5-87e7-8a7c5f29634b'} ticket:9d6bfcc8-db46-42d9-94df-52b8394f82e9] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:41.425611Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:42.134000Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730
2026-03-25T10:30:42.134280Z [debug ] heartbeat_tick : Prev sent/recv: 4994/1325333, now - 4994/1325498, monotonic - 420591.398436219, last_heartbeat_sent - 420571.396549525, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748
2026-03-25T10:30:42.134482Z [debug ] heartbeat_tick: sending heartbeat for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:764
2026-03-25T10:30:42.197596Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:46.426644Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:47.197483Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:51.024749Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:51.025049Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:51.025313Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:51.025579Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:30:51.114102Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:51.114455Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:51.114779Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:30:51.115225Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 3rd time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:30:51.425463Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:52.197710Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:55.024679Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': '24170662-5cfd-39ac-8495-ffa8773da31f'} ticket:4a9d24f7-1bd0-4491-b1bc-1fea82954d40] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:56.426185Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:30:57.196374Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:01.026491Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.029477Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc6816a0> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.029706Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0fd8d50> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.033288Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc680dd0> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.033580Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.034072Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.034313Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.034606Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.034778Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.116463Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.118559Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58aee27800> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.118876Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0faedd0> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.123325Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58aee26840> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:01.123614Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.124083Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.124279Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.124484Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.124695Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:01.425581Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:02.136276Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730
2026-03-25T10:31:02.136610Z [debug ] heartbeat_tick : Prev sent/recv: 4994/1325498, now - 4995/1325657, monotonic - 420611.400776736, last_heartbeat_sent - 420611.400775552, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748
2026-03-25T10:31:02.197911Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:06.425600Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:07.196376Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:08.547138Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'ad3a4b26-88db-32b9-8b72-c6257848e8b6'} ticket:02fcf9a7-2d2e-467c-b411-9e45e8eb1413] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:11.045252Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:11.045532Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:11.045762Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:11.046066Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:31:11.135225Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:11.135532Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:11.135781Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:11.136095Z [warning ] Starting call to 'airflow.sdk.api.client.Client.request', this is the 4th time calling it. [airflow.sdk.api.client] loc=before.py:40
2026-03-25T10:31:11.425077Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:12.197613Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:16.425981Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:17.197156Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:21.046940Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.049988Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc6820c0> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.050205Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0fd8d50> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.053581Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58bc681a00> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.053835Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.054143Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.054293Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.054460Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.054587Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.137066Z [debug ] connect_tcp.started host='prodv3.inf-airflow-webserver-http.service.consul' port=8080 local_address=None timeout=10.0 socket_options=None [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.139022Z [debug ] connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58af206ed0> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.139237Z [debug ] start_tls.started ssl_context=<ssl.SSLContext object at 0x7c58c0faedd0> server_hostname='prodv3.inf-airflow-webserver-http.service.consul' timeout=10.0 [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.142489Z [debug ] start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x7c58af205b20> [httpcore.connection] loc=_trace.py:47
2026-03-25T10:31:21.142716Z [debug ] send_request_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.143039Z [debug ] send_request_headers.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.143200Z [debug ] send_request_body.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.143370Z [debug ] send_request_body.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.143501Z [debug ] receive_response_headers.started request=<Request [b'GET']> [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:21.426376Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:22.110421Z [debug ] pidbox received method ping() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'b8885937-29a4-366c-8887-e2a0601f26f9'} ticket:667cb126-3953-471d-8711-07996e013455] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:22.137485Z [debug ] heartbeat_tick : for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:730
2026-03-25T10:31:22.137742Z [debug ] heartbeat_tick : Prev sent/recv: 4995/1325657, now - 4995/1325822, monotonic - 420631.401909165, last_heartbeat_sent - 420611.400775552, heartbeat int. - 60 for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:748
2026-03-25T10:31:22.137919Z [debug ] heartbeat_tick: sending heartbeat for connection 1f470f8d16404039bc000eadbcc1e68f [amqp.connection.Connection.heartbeat_tick] loc=connection.py:764
2026-03-25T10:31:22.196920Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:26.426201Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:27.196677Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:31.065029Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:31.065299Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:31.065533Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:31.118060Z [error ] Task execute_workload[51cc03c0-8c9c-4afc-aad0-bd5527c889b1] raised unexpected: ReadTimeout('The read operation timed out') [celery.app.trace] loc=trace.py:285
Traceback (most recent call last):
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
 yield
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 250, in handle_request
 resp = self._pool.handle_request(req)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
 raise exc from None
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
 response = connection.handle_request(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 103, in handle_request
 return self._connection.handle_request(request)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 136, in handle_request
 raise exc
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 106, in handle_request
 ) = self._receive_response_headers(**kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 177, in _receive_response_headers
 event = self._receive_event(timeout=timeout)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 217, in _receive_event
 data = self._network_stream.read(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_backends/sync.py", line 126, in read
 with map_exceptions(exc_map):
 ^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
 self.gen.throw(value)
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
 raise to_exc(exc) from exc
httpcore.ReadTimeout: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/trace.py", line 479, in trace_task
 R = retval = fun(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/trace.py", line 779, in __protected_call__
 return self.run(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 202, in execute_workload
 supervise(
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1993, in supervise
 exit_code = process.wait()
 ^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 996, in wait
 self._monitor_subprocess()
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1061, in _monitor_subprocess
 alive = self._service_subprocess(max_wait_time=max_wait_time) is None
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 773, in _service_subprocess
 need_more = socket_handler(key.fileobj)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1770, in cb
 gen.send(request)
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 614, in handle_requests
 self._handle_request(msg, log, request.id)
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1247, in _handle_request
 xcoms = self.client.xcoms.get_sequence_slice(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 570, in get_sequence_slice
 resp = self.client.get(f"xcoms/{dag_id}/{run_id}/{task_id}/{key}/slice", params=params)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1053, in get
 return self.request(
 ^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 331, in wrapped_f
 return copy(f, *args, **kw)
 ^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 470, in __call__
 do = self.iter(retry_state=retry_state)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 371, in iter
 result = action(retry_state)
 ^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 413, in exc_check
 raise retry_exc.reraise()
 ^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 184, in reraise
 raise self.last_attempt.result()
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result
 return self.__get_result()
 ^^^^^^^^^^^^^^^^^^^
 File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
 raise self._exception
 File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 473, in __call__
 result = fn(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 887, in request
 return super().request(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
 return self.send(request, auth=auth, follow_redirects=follow_redirects)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
 response = self._send_handling_auth(
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
 response = self._send_handling_redirects(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
 response = self._send_single_request(request)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1014, in _send_single_request
 response = transport.handle_request(request)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 249, in handle_request
 with map_httpcore_exceptions():
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
 self.gen.throw(value)
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
 raise mapped_exc(message) from exc
httpx.ReadTimeout: The read operation timed out
2026-03-25T10:31:31.153931Z [debug ] receive_response_headers.failed exception=ReadTimeout(TimeoutError('The read operation timed out')) [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:31.154179Z [debug ] response_closed.started [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:31.154394Z [debug ] response_closed.complete [httpcore.http11] loc=_trace.py:47
2026-03-25T10:31:31.206917Z [error ] Task execute_workload[eedcb4b9-06ba-4366-8e07-cf16ad60bd25] raised unexpected: ReadTimeout('The read operation timed out') [celery.app.trace] loc=trace.py:285
Traceback (most recent call last):
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
 yield
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 250, in handle_request
 resp = self._pool.handle_request(req)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
 raise exc from None
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
 response = connection.handle_request(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 103, in handle_request
 return self._connection.handle_request(request)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 136, in handle_request
 raise exc
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 106, in handle_request
 ) = self._receive_response_headers(**kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 177, in _receive_response_headers
 event = self._receive_event(timeout=timeout)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_sync/http11.py", line 217, in _receive_event
 data = self._network_stream.read(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_backends/sync.py", line 126, in read
 with map_exceptions(exc_map):
 ^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
 self.gen.throw(value)
 File "/home/airflow/.local/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
 raise to_exc(exc) from exc
httpcore.ReadTimeout: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/trace.py", line 479, in trace_task
 R = retval = fun(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/celery/app/trace.py", line 779, in __protected_call__
 return self.run(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 202, in execute_workload
 supervise(
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1993, in supervise
 exit_code = process.wait()
 ^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 996, in wait
 self._monitor_subprocess()
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1061, in _monitor_subprocess
 alive = self._service_subprocess(max_wait_time=max_wait_time) is None
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 773, in _service_subprocess
 need_more = socket_handler(key.fileobj)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1770, in cb
 gen.send(request)
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 614, in handle_requests
 self._handle_request(msg, log, request.id)
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1247, in _handle_request
 xcoms = self.client.xcoms.get_sequence_slice(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 570, in get_sequence_slice
 resp = self.client.get(f"xcoms/{dag_id}/{run_id}/{task_id}/{key}/slice", params=params)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1053, in get
 return self.request(
 ^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 331, in wrapped_f
 return copy(f, *args, **kw)
 ^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 470, in __call__
 do = self.iter(retry_state=retry_state)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 371, in iter
 result = action(retry_state)
 ^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 413, in exc_check
 raise retry_exc.reraise()
 ^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 184, in reraise
 raise self.last_attempt.result()
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 449, in result
 return self.__get_result()
 ^^^^^^^^^^^^^^^^^^^
 File "/usr/python/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
 raise self._exception
 File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 473, in __call__
 result = fn(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 887, in request
 return super().request(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
 return self.send(request, auth=auth, follow_redirects=follow_redirects)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
 response = self._send_handling_auth(
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
 response = self._send_handling_redirects(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
 response = self._send_single_request(request)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1014, in _send_single_request
 response = transport.handle_request(request)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 249, in handle_request
 with map_httpcore_exceptions():
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
 self.gen.throw(value)
 File "/home/airflow/.local/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
 raise mapped_exc(message) from exc
httpx.ReadTimeout: The read operation timed out
2026-03-25T10:31:31.425181Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98
2026-03-25T10:31:32.197312Z [debug ] pidbox received method enable_events() [reply_to:None ticket:None] [kombu.pidbox] loc=pidbox.py:98

On webserver side, nothing at all, evenr with DEBUG log level :( . 90% on my dags works fine, but sometimes, I have some timeout on the webserver. I don't know how to debug it properly.

What you think should happen instead?

no timeout :D.

How to reproduce

No sure, the dag is pretty straight forward, the dag starts logging things, and "nothing".

Operating System

Ubuntu 24.04.4 LTS

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.22.0
apache-airflow-providers-celery==3.17.0
apache-airflow-providers-cncf-kubernetes==10.13.0
apache-airflow-providers-common-compat==1.14.0
apache-airflow-providers-common-io==1.7.1
apache-airflow-providers-common-messaging==2.0.2
apache-airflow-providers-common-sql==1.32.0
apache-airflow-providers-docker==4.5.2
apache-airflow-providers-elasticsearch==6.5.0
apache-airflow-providers-fab==3.4.0
apache-airflow-providers-ftp==3.14.1
apache-airflow-providers-git==0.2.4
apache-airflow-providers-google==20.0.0
apache-airflow-providers-grpc==3.9.2
apache-airflow-providers-hashicorp==4.5.0
apache-airflow-providers-http==6.0.0
apache-airflow-providers-microsoft-azure==13.0.0
apache-airflow-providers-mysql==6.5.0
apache-airflow-providers-odbc==4.12.0
apache-airflow-providers-openlineage==2.11.0
apache-airflow-providers-postgres==6.6.0
apache-airflow-providers-redis==4.4.2
apache-airflow-providers-sendgrid==4.2.1
apache-airflow-providers-sftp==5.7.0
apache-airflow-providers-slack==9.7.0
apache-airflow-providers-smtp==2.4.2
apache-airflow-providers-snowflake==6.10.0
apache-airflow-providers-ssh==4.3.1
apache-airflow-providers-standard==1.12.0

Deployment

Docker-Compose

Deployment details

No response

Anything else?

 AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: mysql+mysqldb://....
 AIRFLOW__CELERY__RESULT_BACKEND: db+mysql://....
 AIRFLOW__CORE__EXECUTOR: CeleryExecutor
 AIRFLOW__LOGGING__LOGGING_LEVEL: DEBUG
 AIRFLOW__CORE__DAGS_FOLDER: "/opt/airflow/dags/live/dags"
 AIRFLOW__CORE__LAZY_LOAD_PLUGINS: 'false'
 AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
 AIRFLOW__CORE__LOAD_EXAMPLES: 'False'
 AIRFLOW__CORE__DEFAULT_TASK_EXECUTION_TIMEOUT: 43200
 AIRFLOW__CORE__DEFAULT_TIMEZONE: Europe/Paris
 AIRFLOW__EMAIL__DEFAULT_EMAIL_ON_FAILURE: 'true'
 AIRFLOW__EMAIL__DEFAULT_EMAIL_ON_RETRY: 'true'
 AIRFLOW__SMTP__SMTP_HOST: prod.all.mailhub.dc01
 AIRFLOW__SMTP__SMTP_STARTTLS: 'false'
 AIRFLOW__SMTP__SMTP_SSL: 'false'
 AIRFLOW__API__INSTANCE_NAME: Airflow INFRA PROD DC01
 AIRFLOW__API__EXPOSE_CONFIG: 'True'
 AIRFLOW__API__SSL_CERT: "/opt/airflow/api.pem"
 AIRFLOW__API__SSL_KEY: "/opt/airflow/api.key"
 AIRFLOW__CELERY__BROKER_URL: amqp://app_inf_airflow01_prod:blablabla@prod.inf-airflowbroker-mq-amqp.service.consul:5672/inf-airflow3-broker01-prod
 AIRFLOW__CELERY__BROKER_USE_SSL: 'False'
 AIRFLOW__CELERY__FLOWER_URL_PREFIX: "/flower"
 AIRFLOW__CELERY__BROKER_CONNECTION_MAX_RETRIES: 10
 AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL: '20'
 AIRFLOW__DAG_PROCESSOR__MIN_FILE_PROCESS_INTERVAL: '300'
 AIRFLOW__DAG_PROCESSOR__STALE_DAG_THRESHOLD: '300'
 AIRFLOW__DAG_PROCESSOR__DAG_FILE_PROCESSOR_TIMEOUT: '300'
 AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT: '300'
 AIRFLOW__SCHEDULER__CATCHUP_BY_DEFAULT: 'False'
 TZ: Europe/Paris
 HTTP_TIMEOUT: 60
 REQUESTS_CA_BUNDLE: "/etc/ssl/certs/ca-certificates.crt"
 PYTHONPATH: "/opt/airflow/dags/live/dags"
 AIRFLOW__METRICS__STATSD_ON: 'True'

 AIRFLOW__METRICS__STATSD_PREFIX: inf_airflow02.prod
 AIRFLOW__METRICS__STATSD_HOST: 10.25.30.35
 AIRFLOW__METRICS__STATSD_PORT: '9125'
 AIRFLOW__METRICS__STATSD_INFLUXDB_ENABLED: 'True'
 AIRFLOW__KERBEROS__KINIT_PATH: "/usr/bin/kinit.heimdal"
 AIRFLOW__KERBEROS__REINIT_FREQUENCY: '3600'
 AIRFLOW__KERBEROS__KEYTABS_PATH: "/opt/airflow/security"
 AIRFLOW__KERBEROS__CCACHE_PATH: "/opt/airflow/kerberos_cache"
 AIRFLOW__FAB__ENABLE_PROXY_FIX: 'True'
 AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager
 AIRFLOW__LOGGING__REMOTE_LOGGING: 'True'
 AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: s3://prod-airflow01-logs/logs
 AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: s3_infra
 AIRFLOW__EXECUTION_API__JWT_EXPIRATION_TIME: '86400'
 AIRFLOW__WORKERS__SOCKET_CLEANUP_TIMEOUT: '120'
 AIRFLOW__WORKERS__EXECUTION_API_RETRY_WAIT_MIN: '10'
 AIRFLOW__WORKERS__MAX_FAILED_HEARTBEATS: '5'
 AIRFLOW__WORKERS__EXECUTION_API_TIMEOUT: '10'
 AIRFLOW__CORE__XCOM_BACKEND: airflow.providers.common.io.xcom.backend.XComObjectStorageBackend
 AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_PATH: s3://s3_infra@prod-airflow01-logs/xcom
 AIRFLOW__COMMON_IO__XCOM_OBJECTSTORAGE_THRESHOLD: '0'
 AIRFLOW__API__BASE_URL: https://prod-scheduler-inf.int.test.net
 AIRFLOW__CORE__EXECUTION_API_SERVER_URL: https://prodv3.inf-airflow-webserver-http.service.consul:8080/execution/

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Jeoffreybauvin]

I think I found the problem :

# grep 'memory: Killed' /var/log/syslog
Mar 22 03:31:34 inf-airflow02-p02 kernel: Out of memory: Killed process 2757791 (python3.12) total-vm:5206292kB, anon-rss:4427096kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:8996kB oom_score_adj:0
Mar 22 20:30:13 inf-airflow02-p02 kernel: Out of memory: Killed process 2757793 (python3.12) total-vm:3415244kB, anon-rss:2801000kB, file-rss:256kB, shmem-rss:0kB, UID:50000 pgtables:5764kB oom_score_adj:0
Mar 23 11:00:51 inf-airflow02-p02 kernel: Out of memory: Killed process 2757794 (python3.12) total-vm:3826816kB, anon-rss:3021792kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:6240kB oom_score_adj:0
Mar 23 13:00:53 inf-airflow02-p02 kernel: Out of memory: Killed process 10199 (airflow api_ser) total-vm:12723260kB, anon-rss:9929812kB, file-rss:128kB, shmem-rss:0kB, UID:50000 pgtables:20752kB oom_score_adj:0
Mar 23 15:02:15 inf-airflow02-p02 kernel: Out of memory: Killed process 104233 (airflow api_ser) total-vm:12557632kB, anon-rss:9984656kB, file-rss:128kB, shmem-rss:0kB, UID:50000 pgtables:20224kB oom_score_adj:0
Mar 23 17:21:12 inf-airflow02-p02 kernel: Out of memory: Killed process 233310 (airflow api_ser) total-vm:11844208kB, anon-rss:10034628kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:20628kB oom_score_adj:0
Mar 23 20:01:18 inf-airflow02-p02 kernel: Out of memory: Killed process 344516 (airflow api_ser) total-vm:11301028kB, anon-rss:10015592kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:20760kB oom_score_adj:0
Mar 23 22:00:39 inf-airflow02-p02 kernel: Out of memory: Killed process 510046 (airflow api_ser) total-vm:11363128kB, anon-rss:9997164kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:20536kB oom_score_adj:0
Mar 23 23:00:50 inf-airflow02-p02 kernel: Out of memory: Killed process 610742 (airflow api_ser) total-vm:11329820kB, anon-rss:9856988kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:20000kB oom_score_adj:0
Mar 23 23:02:06 inf-airflow02-p02 kernel: Out of memory: Killed process 661017 (airflow api_ser) total-vm:10881400kB, anon-rss:9868968kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:19696kB oom_score_adj:0
Mar 24 00:15:42 inf-airflow02-p02 kernel: Out of memory: Killed process 662196 (airflow api_ser) total-vm:11198264kB, anon-rss:10018604kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:20456kB oom_score_adj:0
Mar 24 01:01:32 inf-airflow02-p02 kernel: Out of memory: Killed process 714559 (airflow api_ser) total-vm:10803852kB, anon-rss:9976288kB, file-rss:128kB, shmem-rss:0kB, UID:50000 pgtables:20316kB oom_score_adj:0
Mar 24 02:01:59 inf-airflow02-p02 kernel: Out of memory: Killed process 753525 (airflow api_ser) total-vm:11516012kB, anon-rss:10060872kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:20324kB oom_score_adj:0
Mar 24 03:00:49 inf-airflow02-p02 kernel: Out of memory: Killed process 807378 (airflow api_ser) total-vm:12567028kB, anon-rss:9962588kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:20056kB oom_score_adj:0
Mar 24 04:01:24 inf-airflow02-p02 kernel: Out of memory: Killed process 859793 (airflow api_ser) total-vm:12408172kB, anon-rss:9843380kB, file-rss:128kB, shmem-rss:0kB, UID:50000 pgtables:20028kB oom_score_adj:0
Mar 24 06:01:47 inf-airflow02-p02 kernel: Out of memory: Killed process 930412 (airflow api_ser) total-vm:10545756kB, anon-rss:9958656kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:20148kB oom_score_adj:0
Mar 24 07:00:59 inf-airflow02-p02 kernel: Out of memory: Killed process 1029236 (airflow api_ser) total-vm:11060904kB, anon-rss:9982076kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:20344kB oom_score_adj:0
Mar 24 08:14:59 inf-airflow02-p02 kernel: Out of memory: Killed process 1081964 (airflow api_ser) total-vm:10995712kB, anon-rss:9937136kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:19968kB oom_score_adj:0
Mar 24 09:01:22 inf-airflow02-p02 kernel: Out of memory: Killed process 1150672 (airflow api_ser) total-vm:11512996kB, anon-rss:9935068kB, file-rss:128kB, shmem-rss:0kB, UID:50000 pgtables:20264kB oom_score_adj:0
Mar 24 13:01:20 inf-airflow02-p02 kernel: Out of memory: Killed process 1190816 (airflow api_ser) total-vm:11181548kB, anon-rss:9808640kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:20036kB oom_score_adj:0
Mar 24 15:17:05 inf-airflow02-p02 kernel: Out of memory: Killed process 1388333 (airflow api_ser) total-vm:11525252kB, anon-rss:9744336kB, file-rss:128kB, shmem-rss:0kB, UID:50000 pgtables:19864kB oom_score_adj:0
Mar 24 17:01:37 inf-airflow02-p02 kernel: Out of memory: Killed process 1490905 (airflow api_ser) total-vm:11975612kB, anon-rss:9957040kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:20068kB oom_score_adj:0
Mar 24 19:12:01 inf-airflow02-p02 kernel: Out of memory: Killed process 1577679 (airflow api_ser) total-vm:12720064kB, anon-rss:9784472kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:19992kB oom_score_adj:0
Mar 24 21:01:23 inf-airflow02-p02 kernel: Out of memory: Killed process 1633166 (airflow api_ser) total-vm:10683408kB, anon-rss:9879544kB, file-rss:256kB, shmem-rss:0kB, UID:50000 pgtables:19900kB oom_score_adj:0
Mar 24 22:02:16 inf-airflow02-p02 kernel: Out of memory: Killed process 1759737 (airflow api_ser) total-vm:10962096kB, anon-rss:9976652kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:20112kB oom_score_adj:0
Mar 24 23:01:32 inf-airflow02-p02 kernel: Out of memory: Killed process 1809721 (airflow api_ser) total-vm:10620192kB, anon-rss:9789980kB, file-rss:128kB, shmem-rss:0kB, UID:50000 pgtables:19628kB oom_score_adj:0
Mar 25 00:01:36 inf-airflow02-p02 kernel: Out of memory: Killed process 1876275 (airflow api_ser) total-vm:11017156kB, anon-rss:9744020kB, file-rss:128kB, shmem-rss:0kB, UID:50000 pgtables:19992kB oom_score_adj:0
Mar 25 03:00:56 inf-airflow02-p02 kernel: Out of memory: Killed process 1928548 (airflow api_ser) total-vm:10883320kB, anon-rss:9868516kB, file-rss:256kB, shmem-rss:0kB, UID:50000 pgtables:20112kB oom_score_adj:0
Mar 25 04:00:31 inf-airflow02-p02 kernel: Out of memory: Killed process 2101703 (airflow api_ser) total-vm:11629360kB, anon-rss:9973028kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:20212kB oom_score_adj:0
Mar 25 05:00:54 inf-airflow02-p02 kernel: Out of memory: Killed process 2163211 (airflow api_ser) total-vm:10819724kB, anon-rss:9885012kB, file-rss:256kB, shmem-rss:0kB, UID:50000 pgtables:19836kB oom_score_adj:0
Mar 25 06:00:50 inf-airflow02-p02 kernel: Out of memory: Killed process 2212924 (airflow api_ser) total-vm:11678620kB, anon-rss:9909712kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:19968kB oom_score_adj:0
Mar 25 07:03:17 inf-airflow02-p02 kernel: Out of memory: Killed process 2262163 (airflow api_ser) total-vm:11155260kB, anon-rss:9915388kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:19856kB oom_score_adj:0
Mar 25 09:00:55 inf-airflow02-p02 kernel: Out of memory: Killed process 2314652 (airflow api_ser) total-vm:12210460kB, anon-rss:9735012kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:19788kB oom_score_adj:0
Mar 25 11:01:36 inf-airflow02-p02 kernel: Out of memory: Killed process 2414299 (airflow api_ser) total-vm:12411924kB, anon-rss:9911144kB, file-rss:256kB, shmem-rss:0kB, UID:50000 pgtables:20552kB oom_score_adj:0
Mar 25 13:03:21 inf-airflow02-p02 kernel: Out of memory: Killed process 2513928 (airflow api_ser) total-vm:13714508kB, anon-rss:9999592kB, file-rss:0kB, shmem-rss:0kB, UID:50000 pgtables:20612kB oom_score_adj:0

[Jeoffreybauvin]

Something is wrong with the memory consumption of the webserver, but I can't figure what :

3aa05c64dee6 inf-airflow02-p02.dc01-airflow-webserver 41.68% 9.157GiB / 11.63GiB 78.71% 13.9GB / 191MB 295MB / 422kB 8

[wjddn279]

Doesn't the DAG version in Airflow keep incrementing? Like this...

[Jeoffreybauvin]

Indeed @wjddn279 , I have the same thing :

/airflow3> select * from dag_version where dag_id="vmware_inventory" order by created_at desc LIMIT 10;
+----------------------------------+----------------+------------------+-------------+----------------+----------------------------+----------------------------+
| id | version_number | dag_id | bundle_name | bundle_version | created_at | last_updated |
+----------------------------------+----------------+------------------+-------------+----------------+----------------------------+----------------------------+
| 019d290b14e7764aaf1968a889533a5f | 2405 | vmware_inventory | dags-folder | NULL | 2026-03-26 08:28:05.351932 | 2026-03-26 08:28:05.351937 |
| 019d290745fb7825824b70c70de23f05 | 2404 | vmware_inventory | dags-folder | NULL | 2026-03-26 08:23:55.771522 | 2026-03-26 08:23:55.771529 |
| 019d2906738e716aa6a36c35b6aef22d | 2403 | vmware_inventory | dags-folder | NULL | 2026-03-26 08:23:01.902452 | 2026-03-26 08:23:01.902456 |
| 019d2904456072a89963cfbb2764693c | 2402 | vmware_inventory | dags-folder | NULL | 2026-03-26 08:20:39.008974 | 2026-03-26 08:20:39.008982 |
| 019d290319107863b29221762a5366f3 | 2401 | vmware_inventory | dags-folder | NULL | 2026-03-26 08:19:22.128059 | 2026-03-26 08:19:22.128066 |
| 019d290227fa722ebd9050100966530a | 2400 | vmware_inventory | dags-folder | NULL | 2026-03-26 08:18:20.410689 | 2026-03-26 08:18:20.410697 |
| 019d28fe77c076b38c1f2531a2e1fc1e | 2399 | vmware_inventory | dags-folder | NULL | 2026-03-26 08:14:18.688073 | 2026-03-26 08:14:18.688079 |
| 019d28fd0d327c8996be026999744eee | 2398 | vmware_inventory | dags-folder | NULL | 2026-03-26 08:12:45.874338 | 2026-03-26 08:12:45.874345 |
| 019d28f8964779cf9bc3683b3af4c1f6 | 2397 | vmware_inventory | dags-folder | NULL | 2026-03-26 08:07:53.287824 | 2026-03-26 08:07:53.287828 |
| 019d28f74af37bc9a9bd005459cb068c | 2396 | vmware_inventory | dags-folder | NULL | 2026-03-26 08:06:28.467482 | 2026-03-26 08:06:28.467487 |
+----------------------------------+----------------+------------------+-------------+----------------+----------------------------+----------------------------+

[wjddn279]

@Jeoffreybauvin

The dag_version keeps increasing, causing it to consume a lot of memory on the API server.
What does the dag file for vmware_inventory look like?
(please remove your sensitive information in code)

[Jeoffreybauvin]

from datetime import datetime, timedelta
from airflow import DAG
from docker.types import Mount
from airflow.providers.standard.operators.python import PythonOperator
from airflow.models import Variable
from airflow.providers.amazon.aws.operators.s3 import S3Hook
from lib.testDockerOperator import testDockerOperator
import os
import ast

dag = DAG(dag_id="vmware_inventory", start_date=datetime(2024, 6, 10), schedule="0 * * * *", doc_md=__doc__, tags=["vmware_inventory"])

default_owner = 'test'

clusters_txt = Variable.get('VMWARE_INVENTORY')
clusters = ast.literal_eval(clusters_txt)

for cluster in clusters:

 if(cluster.startswith('fo_')):
 conn_user = Variable.get('VCENTER_BOFO_USER')
 conn_password = Variable.get('VCENTER_BOFO_PASSWORD')
 queue = Variable.get('WORKER_DEFAULT_QUEUE_BOFO')
 else:
 conn_user = Variable.get('VCENTER_' + cluster + '_USER')
 conn_password = Variable.get('VCENTER_' + cluster + '_PASSWORD')
 queue = Variable.get('WORKER_DEFAULT_QUEUE_MIXBLUE')
 
 docker_inventory = testDockerOperator(
 owner=default_owner, 
 dag=dag,
 queue=queue,
 task_id='make_vmware_inventory_' + cluster,
 image='test-f-docker/test_vmware_tooling:0.5.1',
 container_name='docker_vmware_inventory_' + cluster,
 mount_tmp_dir=False,
 command='inventory_vmware_threaded_v2.py -D ' + cluster + ' -o /' + cluster + '.json',
 execution_timeout=timedelta(minutes=30),
 retrieve_output=True,
 retrieve_output_path='/' + cluster + '.pickle',
 environment={
 'DC_USER': conn_user,
 'DC_PASSWORD': conn_password,
 'HTTP_PROXY': os.environ['HTTP_PROXY'],
 'HTTPS_PROXY': os.environ['HTTPS_PROXY'],
 },
 mounts=[
 Mount(
 source=os.environ['DAG_EXTERNAL_PATH'] + '/live/conf/vmware_inventory.yaml',
 target='/apps/config.yml',
 read_only=True,
 type='bind',
 )
 ]
 )

 def push_xcom_to_s3(**context):
 """
 Récupère l'XCom et le pousse vers S3.

 Args:
 **context: Contexte Airflow incluant `ti` et `cluster`.
 """
 # Récupérez l'XCom
 cluster = context['cluster']
 xcom_value = str(context['ti'].xcom_pull(task_ids='make_vmware_inventory_' + cluster))

 print('This is xcom for ' + cluster + ' :')
 print(xcom_value)

 # Connectez-vous à S3
 s3_hook = S3Hook(aws_conn_id='s3_infra', verify=False)

 # Poussez l'XCom vers S3
 s3_hook.load_bytes(
 bytes_data=xcom_value.encode('utf-8'), 
 bucket_name=Variable.get('S3_INFRA_BUCKET'), 
 key='vmware_inventory_' + cluster + '.json',
 replace=True,
 )

 push_xcom_to_s3_task = PythonOperator(
 owner=default_owner, 
 task_id='push_xcom_to_s3_' + cluster,
 queue=Variable.get("WORKER_DEFAULT_QUEUE_MIXBLUE", "default"),
 python_callable=push_xcom_to_s3,
 op_kwargs={'cluster': cluster},
 dag=dag,
 ) 

 docker_inventory >> push_xcom_to_s3_task

And vmware_inventory.yaml is not beeing modified at all, the dag and its conf are the same since a few months...

[wjddn279]

what is full contents of clusters_txt = Variable.get('VMWARE_INVENTORY')?
I also want to know value of clusters

[Jeoffreybauvin]

Here it is :

VMWARE_INVENTORY: "[ 'dc01', 'dc02', 'fo_audit_01_dc05', 'fo_audit_01_dc06', 'fo_datalake_01_dc05', 'fo_datalake_01_dc06', 'fo_default_01_dc05', 'fo_default_01_dc06', 'fo_hadoop_01_dc05', 'fo_hadoop_02_dc05', 'fo_hadoop_01_dc06', 'fo_hadoop_02_dc06', 'fo_observability_01_dc05', 'fo_observability_01_dc06', 'fo_observability_02_dc05', 'fo_observability_02_dc06', 'fo_pfm_01_dc05', 'fo_pfm_01_dc06', 'fo_pfm_02_dc05', 'fo_pfm_02_dc06' ]"

[anavrotski]

#61684

[wjddn279]

@Jeoffreybauvin

It's difficult to identify the cause of the dag_version continuously increasing just by looking at the dag file. Can you check the data column in serialized_dag table in the metaDB to see what values are changing?

To explain further, the data values are hashed and compared. We need to find the differences in the data values between versions to determine what is causing the hash values to differ. If it's difficult to find, please share two of those data values with us.