Add plain format output to cli tables

[turbaszek]

closes: #14517

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[XD-DENG]

In the original implementation that @mik-laj mentioned (#8409), header is available ("DAG ID", "Filepath", "Owner"), but they seem not available here.

May be good to have them?

[mik-laj]

Should we update docs? http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/usage-cli.html

[turbaszek]

In the original implementation that @mik-laj mentioned (#8409), header is available ("DAG ID", "Filepath", "Owner"), but they seem not available here.

May be good to have them?

But this will require using grep -v to remove the header, if the purpose is to use it in pipes I think it's better to not have it. WDYT?

[XD-DENG]

In the original implementation that @mik-laj mentioned (#8409), header is available ("DAG ID", "Filepath", "Owner"), but they seem not available here.
May be good to have them?

But this will require using grep -v to remove the header, if the purpose is to use it in pipes I think it's better to not have it. WDYT?

I agree it’s a valid concern.

The main reason for what I propose this is that as a dumb person I may need to know what’s exactly the a few columns given to me (it’s obvious what the values’ lookings of course, but may be good to be more explicit).

a possible solution may be to describe in the doc/help msg what are the columns printed if “plain” output is chosen?

[mik-laj]

We want to add this format for the best compatibility with various word processing tools. I have given AWK as an example, but other people can parse these tables with other tools. For this reason, it seems to me that we should imitate the behavior of other tools.
Docker display headers

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a6b1b5ce7137 postgres:13 "docker-entrypoint.s…" 28 hours ago Up Less than a second (health: starting) 5432/tcp airflow_postgres_1
d4dddf3b3b10 redis:latest "docker-entrypoint.s…" 28 hours ago Up Less than a second (health: starting) 0.0.0.0:6379->6379/tcp airflow_redis_1

gcloud display headers

PROJECT_ID NAME PROJECT_NUMBER
airflow-spy airflow-spy REDACTED
airflow-triage-party airflow-triage-party REDACTED
anierobi-bastion anierobi-bastion REDACTED

So I believe that we should also display the header and not create another data representation format.

[XD-DENG]

We want to add this format for the best compatibility with various word processing tools. I have given AWK as an example, but other people can parse these tables with other tools. For this reason, it seems to me that we should imitate the behavior of other tools.
Docker display headers

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a6b1b5ce7137 postgres:13 "docker-entrypoint.s…" 28 hours ago Up Less than a second (health: starting) 5432/tcp airflow_postgres_1
d4dddf3b3b10 redis:latest "docker-entrypoint.s…" 28 hours ago Up Less than a second (health: starting) 0.0.0.0:6379->6379/tcp airflow_redis_1

gcloud display headers

PROJECT_ID NAME PROJECT_NUMBER
airflow-spy airflow-spy REDACTED
airflow-triage-party airflow-triage-party REDACTED
anierobi-bastion anierobi-bastion REDACTED

So I believe that we should also display the header and not create another data representation format.

Thanks @mik-laj for the examples. Makes sense to me.

@turbaszek WDYT?

[turbaszek]

Done

root@e5dd824b21f3:/opt/airflow# airflow dags list -o plain
/opt/airflow/airflow/providers/apache/beam/hooks/beam.py:28 DeprecationWarning: This module is deprecated. Please use `airflow.hooks.base`.
[2021-03-02 08:58:39,379] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.apache.beam.hooks.beam.BeamHook'>' is missing conn_type attribute and cannot be registered
[2021-03-02 08:58:39,379] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.apache.beam.hooks.beam.BeamHook'>' is missing conn_name_attr attribute and cannot be registered
[2021-03-02 08:58:39,380] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.apache.beam.hooks.beam.BeamHook'>' is missing hook_name attribute and cannot be registered
[2021-03-02 08:58:42,256] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.apache.beam.hooks.beam.BeamHook'>' is missing conn_type attribute and cannot be registered
[2021-03-02 08:58:42,256] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.apache.beam.hooks.beam.BeamHook'>' is missing conn_name_attr attribute and cannot be registered
[2021-03-02 08:58:42,256] {providers_manager.py:249} WARNING - The '<class 'airflow.providers.apache.beam.hooks.beam.BeamHook'>' is missing hook_name attribute and cannot be registered
dag_id filepath owner paused
example_bash_operator /opt/airflow/airflow/example_dags/example_bash_operator.py airflow True
example_branch_dop_operator_v3 /opt/airflow/airflow/example_dags/example_branch_python_dop_operator_3.py airflow True
example_branch_operator /opt/airflow/airflow/example_dags/example_branch_operator.py airflow True
example_complex /opt/airflow/airflow/example_dags/example_complex.py airflow True
example_dag_decorator /opt/airflow/airflow/example_dags/example_dag_decorator.py airflow True
example_external_task_marker_child /opt/airflow/airflow/example_dags/example_external_task_marker_dag.py airflow True
example_external_task_marker_parent /opt/airflow/airflow/example_dags/example_external_task_marker_dag.py airflow True
example_kubernetes_executor /opt/airflow/airflow/example_dags/example_kubernetes_executor.py airflow True
example_kubernetes_executor_config /opt/airflow/airflow/example_dags/example_kubernetes_executor_config.py airflow True
example_nested_branch_dag /opt/airflow/airflow/example_dags/example_nested_branch_dag.py airflow True
example_passing_params_via_test_command /opt/airflow/airflow/example_dags/example_passing_params_via_test_command.py airflow True
example_python_operator /opt/airflow/airflow/example_dags/example_python_operator.py airflow True
example_short_circuit_operator /opt/airflow/airflow/example_dags/example_short_circuit_operator.py airflow True
example_skip_dag /opt/airflow/airflow/example_dags/example_skip_dag.py airflow True
example_subdag_operator /opt/airflow/airflow/example_dags/example_subdag_operator.py airflow True
example_subdag_operator.section-1 /opt/airflow/airflow/example_dags/example_subdag_operator.py airflow True
example_subdag_operator.section-2 /opt/airflow/airflow/example_dags/example_subdag_operator.py airflow True
example_task_group /opt/airflow/airflow/example_dags/example_task_group.py airflow True
example_trigger_controller_dag /opt/airflow/airflow/example_dags/example_trigger_controller_dag.py airflow True
example_trigger_target_dag /opt/airflow/airflow/example_dags/example_trigger_target_dag.py airflow True
example_weekday_branch_operator /opt/airflow/airflow/example_dags/example_branch_day_of_week_operator.py airflow True
example_xcom /opt/airflow/airflow/example_dags/example_xcom.py airflow True
example_xcom_args /opt/airflow/airflow/example_dags/example_xcomargs.py airflow True
example_xcom_args_with_operators /opt/airflow/airflow/example_dags/example_xcomargs.py airflow True
latest_only /opt/airflow/airflow/example_dags/example_latest_only.py airflow True
latest_only_with_trigger /opt/airflow/airflow/example_dags/example_latest_only_with_trigger.py airflow True
test_utils /opt/airflow/airflow/example_dags/test_utils.py airflow True
tutorial /opt/airflow/airflow/example_dags/tutorial.py airflow True
tutorial_etl_dag /opt/airflow/airflow/example_dags/tutorial_etl_dag.py airflow True
tutorial_taskflow_api_etl /opt/airflow/airflow/example_dags/tutorial_taskflow_api_etl.py airflow True

There are some nasty warning but I think that #14554 is taking care of that

[XD-DENG]

There may be other situations where WARNING may jump out. Shall we use the decorator utils.cli.suppress_logs_and_warning here?

[mik-laj]

Should we update docs? http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/usage-cli.html

WDYT?

[turbaszek]

There may be other situations where WARNING may jump out. Shall we use the decorator utils.cli.suppress_logs_and_warning here?

The decorator is already there, I'm afraid that the warning is raised before initialising airflow - but that's something for another PR I think.

[XD-DENG]

There may be other situations where WARNING may jump out. Shall we use the decorator utils.cli.suppress_logs_and_warning here?

The decorator is already there, I'm afraid that the warning is raised before initialising airflow - but that's something for another PR I think.

I see. Then all good. Thanks

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest master or amend the last commit of the PR, and push it with --force-with-lease.