Logs are not integrating with elasticsearch

[peter-cheon]

Apache Airflow version

3.0.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Hello,

I tried to set up log integration using the guide at
https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/stable/logging/index.html#writing-logs-to-elasticsearch,
but the integration is not working.

What you think should happen instead?

No response

How to reproduce

In the Helm chart, I enabled the host as shown below:

elasticsearch:
 # Enable elasticsearch task logging
 enabled: true
 # A secret containing the connection
 connection:
 scheme: http
 host: es-example.local
 port: 9200

I also added the following configuration:

 elasticsearch:
 host: 'es-example.local:9200'
 write_stdout: 'True'
 write_to_es: 'True'
 json_format: 'True'
 log_id_template: "{dag_id}_{task_id}_{execution_date}_{try_number}"
 elasticsearch_configs:
 max_retries: 3
 timeout: 30
 retry_timeout: 'True'

When I check inside the container using airflow.cfg, I can see the settings as below:

[logging]
colored_console_log = False
remote_logging = True

[elasticsearch]
host = es-example.local:9200
json_format = True
log_id_template = {dag_id}_{task_id}_{execution_date}_{try_number}
write_stdout = True
write_to_es = True

[elasticsearch_configs]
max_retries = 3
retry_timeout = True
timeout = 30

Running airflow info also shows:

> airflow info

Apache Airflow 
version | 3.0.1rc1 
executor | CeleryExecutor 
task_logging_handler | airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler 
sql_alchemy_conn | postgresql://..../ 
dags_folder | /opt/airflow/dags/repo/sandbox 
plugins_folder | /opt/airflow/plugins 
base_log_folder | /opt/airflow/logs 
remote_base_log_folder | 

Everything appears to be configured correctly, but nothing is being written to the task logs.
I’m wondering if there is a problem with Elasticsearch integration in Airflow 3.0.

Thank you.

Operating System

kubernetes

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

I deployed the Helm chart from https://github.com/apache/airflow/tree/main/chart

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[peter-cheon]

If Elasticsearch is disabled, the logs are displayed correctly.

[eladkal]

does the same configuration work on 2.10?

[peter-cheon]

does the same configuration work on 2.10?

I have never used Airflow 2.1 before and I'm trying to use version 3.0 for the first time. Do I need to use version 2.1 in order to integrate with Elasticsearch?

Thank you.

[Pad71]

does the same configuration work on 2.10?

@eladkal I think, similar issue is this one -> #49863
We use elasticsearch integration since we started with Airflow (2.7.0) and it worked (we use 2.10.5 at the moment). But does not work in 3.0.

[jason810496]

Hi @peter-cheon, could you try setting elasticsearch.write_stdout to 'False'?

I just checked the condition for when the log handler actually writes logs to Elasticsearch, and it turns out that write_to_es needs to be True and write_stdout should be False.

https://github.com/apache/airflow/blob/main/providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py#L530

[peter-cheon]

write_to_es

hello, @jason810496
I added the following configuration, but it still doesn't work.

 elasticsearch:
 host: 'es-example.local:9200'
 write_stdout: 'False'
 write_to_es: 'True'
 json_format: 'True'
 log_id_template: "{dag_id}_{task_id}_{execution_date}_{try_number}"

[jason810496]

I will look into the issue a few day later, self assign.

[alex1404]

same for me; worked fine in 2.10.4