Render structured logs in the new UI rather than showing raw JSON

[ashb]

Closes#46657

There are multiple parts to this PR

First off: the log reader interface was a mess There was some odd+old code
do deal with reading from multiple hosts that made the message confusing. This
was added for smart sensors (which went away in v2.4 or v2.5) but this mess
remained, and reading from multiple hosts is handled differently now.

This PR keeps the current "parse+interleave" behaviour (though it's debatable
if the interleave feature is needed specifically, or if we could get away with
a simpler concat instead. Future work there if anyone wants to think about and
tackle this.) but changes the JSON resposne type from a single string (the
value of which was previoulsy a mess of double encoded JSON and repr of a
python tuple making it impossible to do anything but display at) to a list of
either strings (when it can't be parsed) or a list of
dicts/StructuredLogMessage.

I have also done some cursory rendering/displaying of these structured log
messages in the UI, but they could be greatly improved by adding colors to
various components of the log.

The current rendered HTML looks like this:

<p><spanclass="event">::group::Log message source details</span><spanclass="log-key sources">sources=["/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log","/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log.trigger.14.log"]</span></p><p><spanclass="event">::endgroup::</span></p><p>[<timedatetime="2025-02-16T12:23:30.033308">2025-02-16T12:23:30.033308</time>] <spanclass="log-level debug">DEBUG</span> - <spanclass="event">Hook impls: []</span><spanclass="log-key logger">logger="airflow.listeners.listener"</span></p>

Although not used by the UI, the non-application/json content type is now
updated to a) include the continuation token as a header, and to set the
content type as application/x-ndjson

It's not the prettiest yet, I'll leave that to people more talented at that than me

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[ashb]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

[ashb]

I have not tested this with an old/non-json Log, and I haven't fixed up the unit tests yet either.

[bbovenzi]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

It's a bunch of typescript issues. Usually easier to read inline. We we weren't setting the types of the log parsing functions so that made typescript say every variable assignment in the function was unsafe.

[Lee-W]

quick update on the current progress. core tests should already be fixed. some provider ones remaining.

• providers/amazon/tests/unit/amazon/aws/log/test_cloudwatch_task_handler.py
• providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py
• providers/google/tests/unit/google/cloud/log/test_gcs_task_handler.py
• providers/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py

[Lee-W]

just add compat code to providers. The tests should pass now... I think...

[Lee-W]

didn't notice there're some others...

• providers/microsoft/azure/tests/unit/microsoft/azure/log/test_wasb_task_handler.py::TestWasbTaskHandler::test_wasb_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns_no_match
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_missing_logs
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_none_metadata
• providers/redis/tests/unit/redis/log/test_redis_task_handler.py::TestRedisTaskHandler::test_read

[Lee-W]

As the comments are resolved, I'll merge this one. Thanks all for your help!

[ashb]

Closes#46657

There are multiple parts to this PR

First off: the log reader interface was a mess There was some odd+old code
do deal with reading from multiple hosts that made the message confusing. This
was added for smart sensors (which went away in v2.4 or v2.5) but this mess
remained, and reading from multiple hosts is handled differently now.

This PR keeps the current "parse+interleave" behaviour (though it's debatable
if the interleave feature is needed specifically, or if we could get away with
a simpler concat instead. Future work there if anyone wants to think about and
tackle this.) but changes the JSON resposne type from a single string (the
value of which was previoulsy a mess of double encoded JSON and repr of a
python tuple making it impossible to do anything but display at) to a list of
either strings (when it can't be parsed) or a list of
dicts/StructuredLogMessage.

I have also done some cursory rendering/displaying of these structured log
messages in the UI, but they could be greatly improved by adding colors to
various components of the log.

The current rendered HTML looks like this:

<p><spanclass="event">::group::Log message source details</span><spanclass="log-key sources">sources=["/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log","/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log.trigger.14.log"]</span></p><p><spanclass="event">::endgroup::</span></p><p>[<timedatetime="2025-02-16T12:23:30.033308">2025-02-16T12:23:30.033308</time>] <spanclass="log-level debug">DEBUG</span> - <spanclass="event">Hook impls: []</span><spanclass="log-key logger">logger="airflow.listeners.listener"</span></p>

Although not used by the UI, the non-application/json content type is now
updated to a) include the continuation token as a header, and to set the
content type as application/x-ndjson

It's not the prettiest yet, I'll leave that to people more talented at that than me

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[ashb]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

[ashb]

I have not tested this with an old/non-json Log, and I haven't fixed up the unit tests yet either.

[bbovenzi]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

It's a bunch of typescript issues. Usually easier to read inline. We we weren't setting the types of the log parsing functions so that made typescript say every variable assignment in the function was unsafe.

[Lee-W]

quick update on the current progress. core tests should already be fixed. some provider ones remaining.

• providers/amazon/tests/unit/amazon/aws/log/test_cloudwatch_task_handler.py
• providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py
• providers/google/tests/unit/google/cloud/log/test_gcs_task_handler.py
• providers/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py

[Lee-W]

just add compat code to providers. The tests should pass now... I think...

[Lee-W]

didn't notice there're some others...

• providers/microsoft/azure/tests/unit/microsoft/azure/log/test_wasb_task_handler.py::TestWasbTaskHandler::test_wasb_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns_no_match
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_missing_logs
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_none_metadata
• providers/redis/tests/unit/redis/log/test_redis_task_handler.py::TestRedisTaskHandler::test_read

[Lee-W]

As the comments are resolved, I'll merge this one. Thanks all for your help!

[ashb]

Closes#46657

There are multiple parts to this PR

First off: the log reader interface was a mess There was some odd+old code
do deal with reading from multiple hosts that made the message confusing. This
was added for smart sensors (which went away in v2.4 or v2.5) but this mess
remained, and reading from multiple hosts is handled differently now.

This PR keeps the current "parse+interleave" behaviour (though it's debatable
if the interleave feature is needed specifically, or if we could get away with
a simpler concat instead. Future work there if anyone wants to think about and
tackle this.) but changes the JSON resposne type from a single string (the
value of which was previoulsy a mess of double encoded JSON and repr of a
python tuple making it impossible to do anything but display at) to a list of
either strings (when it can't be parsed) or a list of
dicts/StructuredLogMessage.

I have also done some cursory rendering/displaying of these structured log
messages in the UI, but they could be greatly improved by adding colors to
various components of the log.

The current rendered HTML looks like this:

<p><spanclass="event">::group::Log message source details</span><spanclass="log-key sources">sources=["/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log","/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log.trigger.14.log"]</span></p><p><spanclass="event">::endgroup::</span></p><p>[<timedatetime="2025-02-16T12:23:30.033308">2025-02-16T12:23:30.033308</time>] <spanclass="log-level debug">DEBUG</span> - <spanclass="event">Hook impls: []</span><spanclass="log-key logger">logger="airflow.listeners.listener"</span></p>

Although not used by the UI, the non-application/json content type is now
updated to a) include the continuation token as a header, and to set the
content type as application/x-ndjson

It's not the prettiest yet, I'll leave that to people more talented at that than me

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[ashb]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

[ashb]

I have not tested this with an old/non-json Log, and I haven't fixed up the unit tests yet either.

[bbovenzi]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

It's a bunch of typescript issues. Usually easier to read inline. We we weren't setting the types of the log parsing functions so that made typescript say every variable assignment in the function was unsafe.

[Lee-W]

quick update on the current progress. core tests should already be fixed. some provider ones remaining.

• providers/amazon/tests/unit/amazon/aws/log/test_cloudwatch_task_handler.py
• providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py
• providers/google/tests/unit/google/cloud/log/test_gcs_task_handler.py
• providers/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py

[Lee-W]

just add compat code to providers. The tests should pass now... I think...

[Lee-W]

didn't notice there're some others...

• providers/microsoft/azure/tests/unit/microsoft/azure/log/test_wasb_task_handler.py::TestWasbTaskHandler::test_wasb_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns_no_match
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_missing_logs
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_none_metadata
• providers/redis/tests/unit/redis/log/test_redis_task_handler.py::TestRedisTaskHandler::test_read

[Lee-W]

As the comments are resolved, I'll merge this one. Thanks all for your help!

[ashb]

Closes#46657

There are multiple parts to this PR

First off: the log reader interface was a mess There was some odd+old code
do deal with reading from multiple hosts that made the message confusing. This
was added for smart sensors (which went away in v2.4 or v2.5) but this mess
remained, and reading from multiple hosts is handled differently now.

This PR keeps the current "parse+interleave" behaviour (though it's debatable
if the interleave feature is needed specifically, or if we could get away with
a simpler concat instead. Future work there if anyone wants to think about and
tackle this.) but changes the JSON resposne type from a single string (the
value of which was previoulsy a mess of double encoded JSON and repr of a
python tuple making it impossible to do anything but display at) to a list of
either strings (when it can't be parsed) or a list of
dicts/StructuredLogMessage.

I have also done some cursory rendering/displaying of these structured log
messages in the UI, but they could be greatly improved by adding colors to
various components of the log.

The current rendered HTML looks like this:

<p><spanclass="event">::group::Log message source details</span><spanclass="log-key sources">sources=["/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log","/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log.trigger.14.log"]</span></p><p><spanclass="event">::endgroup::</span></p><p>[<timedatetime="2025-02-16T12:23:30.033308">2025-02-16T12:23:30.033308</time>] <spanclass="log-level debug">DEBUG</span> - <spanclass="event">Hook impls: []</span><spanclass="log-key logger">logger="airflow.listeners.listener"</span></p>

Although not used by the UI, the non-application/json content type is now
updated to a) include the continuation token as a header, and to set the
content type as application/x-ndjson

It's not the prettiest yet, I'll leave that to people more talented at that than me

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[ashb]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

[ashb]

I have not tested this with an old/non-json Log, and I haven't fixed up the unit tests yet either.

[bbovenzi]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

It's a bunch of typescript issues. Usually easier to read inline. We we weren't setting the types of the log parsing functions so that made typescript say every variable assignment in the function was unsafe.

[Lee-W]

quick update on the current progress. core tests should already be fixed. some provider ones remaining.

• providers/amazon/tests/unit/amazon/aws/log/test_cloudwatch_task_handler.py
• providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py
• providers/google/tests/unit/google/cloud/log/test_gcs_task_handler.py
• providers/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py

[Lee-W]

just add compat code to providers. The tests should pass now... I think...

[Lee-W]

didn't notice there're some others...

• providers/microsoft/azure/tests/unit/microsoft/azure/log/test_wasb_task_handler.py::TestWasbTaskHandler::test_wasb_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns_no_match
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_missing_logs
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_none_metadata
• providers/redis/tests/unit/redis/log/test_redis_task_handler.py::TestRedisTaskHandler::test_read

[Lee-W]

As the comments are resolved, I'll merge this one. Thanks all for your help!

[ashb]

Closes#46657

There are multiple parts to this PR

First off: the log reader interface was a mess There was some odd+old code
do deal with reading from multiple hosts that made the message confusing. This
was added for smart sensors (which went away in v2.4 or v2.5) but this mess
remained, and reading from multiple hosts is handled differently now.

This PR keeps the current "parse+interleave" behaviour (though it's debatable
if the interleave feature is needed specifically, or if we could get away with
a simpler concat instead. Future work there if anyone wants to think about and
tackle this.) but changes the JSON resposne type from a single string (the
value of which was previoulsy a mess of double encoded JSON and repr of a
python tuple making it impossible to do anything but display at) to a list of
either strings (when it can't be parsed) or a list of
dicts/StructuredLogMessage.

I have also done some cursory rendering/displaying of these structured log
messages in the UI, but they could be greatly improved by adding colors to
various components of the log.

The current rendered HTML looks like this:

<p><spanclass="event">::group::Log message source details</span><spanclass="log-key sources">sources=["/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log","/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log.trigger.14.log"]</span></p><p><spanclass="event">::endgroup::</span></p><p>[<timedatetime="2025-02-16T12:23:30.033308">2025-02-16T12:23:30.033308</time>] <spanclass="log-level debug">DEBUG</span> - <spanclass="event">Hook impls: []</span><spanclass="log-key logger">logger="airflow.listeners.listener"</span></p>

Although not used by the UI, the non-application/json content type is now
updated to a) include the continuation token as a header, and to set the
content type as application/x-ndjson

It's not the prettiest yet, I'll leave that to people more talented at that than me

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[ashb]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

[ashb]

I have not tested this with an old/non-json Log, and I haven't fixed up the unit tests yet either.

[bbovenzi]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

It's a bunch of typescript issues. Usually easier to read inline. We we weren't setting the types of the log parsing functions so that made typescript say every variable assignment in the function was unsafe.

[Lee-W]

quick update on the current progress. core tests should already be fixed. some provider ones remaining.

• providers/amazon/tests/unit/amazon/aws/log/test_cloudwatch_task_handler.py
• providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py
• providers/google/tests/unit/google/cloud/log/test_gcs_task_handler.py
• providers/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py

[Lee-W]

just add compat code to providers. The tests should pass now... I think...

[Lee-W]

didn't notice there're some others...

• providers/microsoft/azure/tests/unit/microsoft/azure/log/test_wasb_task_handler.py::TestWasbTaskHandler::test_wasb_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns_no_match
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_missing_logs
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_none_metadata
• providers/redis/tests/unit/redis/log/test_redis_task_handler.py::TestRedisTaskHandler::test_read

[Lee-W]

As the comments are resolved, I'll merge this one. Thanks all for your help!

[ashb]

Closes#46657

There are multiple parts to this PR

First off: the log reader interface was a mess There was some odd+old code
do deal with reading from multiple hosts that made the message confusing. This
was added for smart sensors (which went away in v2.4 or v2.5) but this mess
remained, and reading from multiple hosts is handled differently now.

This PR keeps the current "parse+interleave" behaviour (though it's debatable
if the interleave feature is needed specifically, or if we could get away with
a simpler concat instead. Future work there if anyone wants to think about and
tackle this.) but changes the JSON resposne type from a single string (the
value of which was previoulsy a mess of double encoded JSON and repr of a
python tuple making it impossible to do anything but display at) to a list of
either strings (when it can't be parsed) or a list of
dicts/StructuredLogMessage.

I have also done some cursory rendering/displaying of these structured log
messages in the UI, but they could be greatly improved by adding colors to
various components of the log.

The current rendered HTML looks like this:

<p><spanclass="event">::group::Log message source details</span><spanclass="log-key sources">sources=["/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log","/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log.trigger.14.log"]</span></p><p><spanclass="event">::endgroup::</span></p><p>[<timedatetime="2025-02-16T12:23:30.033308">2025-02-16T12:23:30.033308</time>] <spanclass="log-level debug">DEBUG</span> - <spanclass="event">Hook impls: []</span><spanclass="log-key logger">logger="airflow.listeners.listener"</span></p>

Although not used by the UI, the non-application/json content type is now
updated to a) include the continuation token as a header, and to set the
content type as application/x-ndjson

It's not the prettiest yet, I'll leave that to people more talented at that than me

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[ashb]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

[ashb]

I have not tested this with an old/non-json Log, and I haven't fixed up the unit tests yet either.

[bbovenzi]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

It's a bunch of typescript issues. Usually easier to read inline. We we weren't setting the types of the log parsing functions so that made typescript say every variable assignment in the function was unsafe.

[Lee-W]

quick update on the current progress. core tests should already be fixed. some provider ones remaining.

• providers/amazon/tests/unit/amazon/aws/log/test_cloudwatch_task_handler.py
• providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py
• providers/google/tests/unit/google/cloud/log/test_gcs_task_handler.py
• providers/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py

[Lee-W]

just add compat code to providers. The tests should pass now... I think...

[Lee-W]

didn't notice there're some others...

• providers/microsoft/azure/tests/unit/microsoft/azure/log/test_wasb_task_handler.py::TestWasbTaskHandler::test_wasb_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns_no_match
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_missing_logs
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_none_metadata
• providers/redis/tests/unit/redis/log/test_redis_task_handler.py::TestRedisTaskHandler::test_read

[Lee-W]

As the comments are resolved, I'll merge this one. Thanks all for your help!

[ashb]

Closes#46657

There are multiple parts to this PR

First off: the log reader interface was a mess There was some odd+old code
do deal with reading from multiple hosts that made the message confusing. This
was added for smart sensors (which went away in v2.4 or v2.5) but this mess
remained, and reading from multiple hosts is handled differently now.

This PR keeps the current "parse+interleave" behaviour (though it's debatable
if the interleave feature is needed specifically, or if we could get away with
a simpler concat instead. Future work there if anyone wants to think about and
tackle this.) but changes the JSON resposne type from a single string (the
value of which was previoulsy a mess of double encoded JSON and repr of a
python tuple making it impossible to do anything but display at) to a list of
either strings (when it can't be parsed) or a list of
dicts/StructuredLogMessage.

I have also done some cursory rendering/displaying of these structured log
messages in the UI, but they could be greatly improved by adding colors to
various components of the log.

The current rendered HTML looks like this:

<p><spanclass="event">::group::Log message source details</span><spanclass="log-key sources">sources=["/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log","/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log.trigger.14.log"]</span></p><p><spanclass="event">::endgroup::</span></p><p>[<timedatetime="2025-02-16T12:23:30.033308">2025-02-16T12:23:30.033308</time>] <spanclass="log-level debug">DEBUG</span> - <spanclass="event">Hook impls: []</span><spanclass="log-key logger">logger="airflow.listeners.listener"</span></p>

Although not used by the UI, the non-application/json content type is now
updated to a) include the continuation token as a header, and to set the
content type as application/x-ndjson

It's not the prettiest yet, I'll leave that to people more talented at that than me

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[ashb]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

[ashb]

I have not tested this with an old/non-json Log, and I haven't fixed up the unit tests yet either.

[bbovenzi]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

It's a bunch of typescript issues. Usually easier to read inline. We we weren't setting the types of the log parsing functions so that made typescript say every variable assignment in the function was unsafe.

[Lee-W]

quick update on the current progress. core tests should already be fixed. some provider ones remaining.

• providers/amazon/tests/unit/amazon/aws/log/test_cloudwatch_task_handler.py
• providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py
• providers/google/tests/unit/google/cloud/log/test_gcs_task_handler.py
• providers/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py

[Lee-W]

just add compat code to providers. The tests should pass now... I think...

[Lee-W]

didn't notice there're some others...

• providers/microsoft/azure/tests/unit/microsoft/azure/log/test_wasb_task_handler.py::TestWasbTaskHandler::test_wasb_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns_no_match
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_missing_logs
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_none_metadata
• providers/redis/tests/unit/redis/log/test_redis_task_handler.py::TestRedisTaskHandler::test_read

[Lee-W]

As the comments are resolved, I'll merge this one. Thanks all for your help!

[ashb]

Closes#46657

There are multiple parts to this PR

First off: the log reader interface was a mess There was some odd+old code
do deal with reading from multiple hosts that made the message confusing. This
was added for smart sensors (which went away in v2.4 or v2.5) but this mess
remained, and reading from multiple hosts is handled differently now.

This PR keeps the current "parse+interleave" behaviour (though it's debatable
if the interleave feature is needed specifically, or if we could get away with
a simpler concat instead. Future work there if anyone wants to think about and
tackle this.) but changes the JSON resposne type from a single string (the
value of which was previoulsy a mess of double encoded JSON and repr of a
python tuple making it impossible to do anything but display at) to a list of
either strings (when it can't be parsed) or a list of
dicts/StructuredLogMessage.

I have also done some cursory rendering/displaying of these structured log
messages in the UI, but they could be greatly improved by adding colors to
various components of the log.

The current rendered HTML looks like this:

<p><spanclass="event">::group::Log message source details</span><spanclass="log-key sources">sources=["/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log","/root/airflow/logs/dag_id=trigger_test/run_id=manual__2025-02-16T12:23:29.118614+00:00_gOTl0Qub/task_id=waiter/attempt=1.log.trigger.14.log"]</span></p><p><spanclass="event">::endgroup::</span></p><p>[<timedatetime="2025-02-16T12:23:30.033308">2025-02-16T12:23:30.033308</time>] <spanclass="log-level debug">DEBUG</span> - <spanclass="event">Hook impls: []</span><spanclass="log-key logger">logger="airflow.listeners.listener"</span></p>

Although not used by the UI, the non-application/json content type is now
updated to a) include the continuation token as a header, and to set the
content type as application/x-ndjson

It's not the prettiest yet, I'll leave that to people more talented at that than me

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[ashb]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

[ashb]

I have not tested this with an old/non-json Log, and I haven't fixed up the unit tests yet either.

[bbovenzi]

This is failing it's static check and I don't know how to decipher the error:

/Users/ash/code/airflow/airflow/airflow/ui/src/pages/TaskInstance/Logs/Logs.tsx
97:21 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
122:27 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
/Users/ash/code/airflow/airflow/airflow/ui/src/queries/useLogs.tsx
43:59 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
50:7 error Unexpected use of continue statement no-continue
60:18 error Unsafe assignment of an `any` value @typescript-eslint/no-unsafe-assignment
75:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
75:19 error Unsafe call of an `error` type typed value @typescript-eslint/no-unsafe-call
75:24 error Unsafe member access .map on an `error` typed value @typescript-eslint/no-unsafe-member-access
78:41 error Invalid type "unknown" of template literal expression @typescript-eslint/restrict-template-expressions
81:14 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment
86:5 error Unsafe assignment of an error typed value @typescript-eslint/no-unsafe-assignment

It's a bunch of typescript issues. Usually easier to read inline. We we weren't setting the types of the log parsing functions so that made typescript say every variable assignment in the function was unsafe.

[Lee-W]

quick update on the current progress. core tests should already be fixed. some provider ones remaining.

• providers/amazon/tests/unit/amazon/aws/log/test_cloudwatch_task_handler.py
• providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py
• providers/google/tests/unit/google/cloud/log/test_gcs_task_handler.py
• providers/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py

[Lee-W]

just add compat code to providers. The tests should pass now... I think...

[Lee-W]

didn't notice there're some others...

• providers/microsoft/azure/tests/unit/microsoft/azure/log/test_wasb_task_handler.py::TestWasbTaskHandler::test_wasb_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_patterns_no_match
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_missing_logs
• providers/opensearch/tests/unit/opensearch/log/test_os_task_handler.py::TestOpensearchTaskHandler::test_read_with_none_metadata
• providers/redis/tests/unit/redis/log/test_redis_task_handler.py::TestRedisTaskHandler::test_read

[Lee-W]

As the comments are resolved, I'll merge this one. Thanks all for your help!