Remove gunicorn daemonize for api-server by jason810496 · Pull Request #52860 · apache/airflow

jason810496 · 2025-07-04T10:21:10Z

related: #52581

Why

When I attempt to remove gunicorn dependency in #52581, I found that we are still using gunicorn for daemonization.

What

Replace gunicorn.daemonize with run_command_with_daemon_option, this also align the behavior of --daemon option for scheduler, triggerer ... and api-server
Add test for api-server with --daemon as well
- Rename _CommonCLIGunicornTestClass to _CommonCLIUvicornTestClass, since we are using uvicorn instead of gunicorn for api-server now.
Replace [api/access_logfile] config with [api/log_config], detail in Remove gunicorn daemonize for api-server #52860 (comment)

jason810496

access_log=access_logfile, # type: ignore[arg-type] this line is a strange part for me.

I add # type: ignore[arg-type] to ingore the mypy warning for now.

jason810496 · 2025-07-05T15:30:25Z

Hi @potiuk, may I might need hints for replacing config.

I thought replacing them in config.yaml and add the pair in deprecated_option in configuration will be all right.

Thanks in advance!

Update:

I found the root cause.

My step is correct but I set [api/log_config] config default as None in config.yml and I set default=conf.get("api", "log_config") in cli_config module.
There is not need to set default=conf.get("api", "log_config") or should set fallback=None for the --log-config flag in cli_config, since the it is not mandatory.

jason810496

Thanks @potiuk and @pierrejeambrun for the review!
Just resolved all comments and fixed the CI as well.

potiuk

Nice. One NIT about updating newsfragment.

- the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72)

* [v3-0-test] Remove gunicorn daemonize for api-server (#52860) * Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72) * Fix spelling wordlist * Fix configuration

jedcunningham · 2025-07-31T18:27:38Z

 ("api", "require_confirmation_dag_change"): ("webserver", "require_confirmation_dag_change", "3.1.0"),
 ("api", "instance_name"): ("webserver", "instance_name", "3.1.0"),
 ("dag_processor", "parsing_pre_import_modules"): ("scheduler", "parsing_pre_import_modules", "3.1.0"),
+ ("api", "log_config"): ("api", "access_logfile", "3.1.0"),


This is not safe to do - they are different config options. As mentioned in the comment above, this will use the old access_logfile value for log_config if it's set, which isn't what we want.

jedcunningham · 2025-07-31T19:12:11Z

+Replace API server ``access_logfile`` configuration with ``log_config``
+
+The API server configuration option ``[api] access_logfile`` has been replaced with ``[api] log_config`` to align with uvicorn's logging configuration instead of the legacy gunicorn approach.
+The new ``log_config`` option accepts a path to a logging configuration file compatible with ``logging.config.fileConfig``, providing more flexible logging configuration for the API server.


Is there a way we can continue to support a simple access logfile config? I just worry a bit about backcompat, right now this is a breaking change. Maybe we leave access_logfile, build our own log_config file dynamically, and make log_config and access_logfile mutually exclusive? And we should emit a deprecation warning when access_logfile is used.

Well, good and bad news. Good news is no backcompat issues to worry about! I believe it got broken in #45103. Bad news is I don't think access_log ever worked in AF3.

…52860) (apache#53372)" This reverts commit 7212c88.

…#53372)" (#54002) This reverts commit 7212c88.

jason810496 · 2025-07-04T10:21:10Z

related: #52581

Why

When I attempt to remove gunicorn dependency in #52581, I found that we are still using gunicorn for daemonization.

What

Replace gunicorn.daemonize with run_command_with_daemon_option, this also align the behavior of --daemon option for scheduler, triggerer ... and api-server
Add test for api-server with --daemon as well
- Rename _CommonCLIGunicornTestClass to _CommonCLIUvicornTestClass, since we are using uvicorn instead of gunicorn for api-server now.
Replace [api/access_logfile] config with [api/log_config], detail in Remove gunicorn daemonize for api-server #52860 (comment)

jason810496

access_log=access_logfile, # type: ignore[arg-type] this line is a strange part for me.

I add # type: ignore[arg-type] to ingore the mypy warning for now.

jason810496 · 2025-07-05T15:30:25Z

Hi @potiuk, may I might need hints for replacing config.

I thought replacing them in config.yaml and add the pair in deprecated_option in configuration will be all right.

Thanks in advance!

Update:

I found the root cause.

My step is correct but I set [api/log_config] config default as None in config.yml and I set default=conf.get("api", "log_config") in cli_config module.
There is not need to set default=conf.get("api", "log_config") or should set fallback=None for the --log-config flag in cli_config, since the it is not mandatory.

jason810496

Thanks @potiuk and @pierrejeambrun for the review!
Just resolved all comments and fixed the CI as well.

potiuk

Nice. One NIT about updating newsfragment.

- the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72)

* [v3-0-test] Remove gunicorn daemonize for api-server (#52860) * Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72) * Fix spelling wordlist * Fix configuration

jedcunningham · 2025-07-31T18:27:38Z

 ("api", "require_confirmation_dag_change"): ("webserver", "require_confirmation_dag_change", "3.1.0"),
 ("api", "instance_name"): ("webserver", "instance_name", "3.1.0"),
 ("dag_processor", "parsing_pre_import_modules"): ("scheduler", "parsing_pre_import_modules", "3.1.0"),
+ ("api", "log_config"): ("api", "access_logfile", "3.1.0"),


This is not safe to do - they are different config options. As mentioned in the comment above, this will use the old access_logfile value for log_config if it's set, which isn't what we want.

jedcunningham · 2025-07-31T19:12:11Z

+Replace API server ``access_logfile`` configuration with ``log_config``
+
+The API server configuration option ``[api] access_logfile`` has been replaced with ``[api] log_config`` to align with uvicorn's logging configuration instead of the legacy gunicorn approach.
+The new ``log_config`` option accepts a path to a logging configuration file compatible with ``logging.config.fileConfig``, providing more flexible logging configuration for the API server.


Is there a way we can continue to support a simple access logfile config? I just worry a bit about backcompat, right now this is a breaking change. Maybe we leave access_logfile, build our own log_config file dynamically, and make log_config and access_logfile mutually exclusive? And we should emit a deprecation warning when access_logfile is used.

Well, good and bad news. Good news is no backcompat issues to worry about! I believe it got broken in #45103. Bad news is I don't think access_log ever worked in AF3.

…52860) (apache#53372)" This reverts commit 7212c88.

…#53372)" (#54002) This reverts commit 7212c88.

jason810496 · 2025-07-04T10:21:10Z

related: #52581

Why

When I attempt to remove gunicorn dependency in #52581, I found that we are still using gunicorn for daemonization.

What

Replace gunicorn.daemonize with run_command_with_daemon_option, this also align the behavior of --daemon option for scheduler, triggerer ... and api-server
Add test for api-server with --daemon as well
- Rename _CommonCLIGunicornTestClass to _CommonCLIUvicornTestClass, since we are using uvicorn instead of gunicorn for api-server now.
Replace [api/access_logfile] config with [api/log_config], detail in Remove gunicorn daemonize for api-server #52860 (comment)

jason810496

access_log=access_logfile, # type: ignore[arg-type] this line is a strange part for me.

I add # type: ignore[arg-type] to ingore the mypy warning for now.

jason810496 · 2025-07-05T15:30:25Z

Hi @potiuk, may I might need hints for replacing config.

I thought replacing them in config.yaml and add the pair in deprecated_option in configuration will be all right.

Thanks in advance!

Update:

I found the root cause.

My step is correct but I set [api/log_config] config default as None in config.yml and I set default=conf.get("api", "log_config") in cli_config module.
There is not need to set default=conf.get("api", "log_config") or should set fallback=None for the --log-config flag in cli_config, since the it is not mandatory.

jason810496

Thanks @potiuk and @pierrejeambrun for the review!
Just resolved all comments and fixed the CI as well.

potiuk

Nice. One NIT about updating newsfragment.

- the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72)

* [v3-0-test] Remove gunicorn daemonize for api-server (#52860) * Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72) * Fix spelling wordlist * Fix configuration

jedcunningham · 2025-07-31T18:27:38Z

 ("api", "require_confirmation_dag_change"): ("webserver", "require_confirmation_dag_change", "3.1.0"),
 ("api", "instance_name"): ("webserver", "instance_name", "3.1.0"),
 ("dag_processor", "parsing_pre_import_modules"): ("scheduler", "parsing_pre_import_modules", "3.1.0"),
+ ("api", "log_config"): ("api", "access_logfile", "3.1.0"),


This is not safe to do - they are different config options. As mentioned in the comment above, this will use the old access_logfile value for log_config if it's set, which isn't what we want.

jedcunningham · 2025-07-31T19:12:11Z

+Replace API server ``access_logfile`` configuration with ``log_config``
+
+The API server configuration option ``[api] access_logfile`` has been replaced with ``[api] log_config`` to align with uvicorn's logging configuration instead of the legacy gunicorn approach.
+The new ``log_config`` option accepts a path to a logging configuration file compatible with ``logging.config.fileConfig``, providing more flexible logging configuration for the API server.


Is there a way we can continue to support a simple access logfile config? I just worry a bit about backcompat, right now this is a breaking change. Maybe we leave access_logfile, build our own log_config file dynamically, and make log_config and access_logfile mutually exclusive? And we should emit a deprecation warning when access_logfile is used.

Well, good and bad news. Good news is no backcompat issues to worry about! I believe it got broken in #45103. Bad news is I don't think access_log ever worked in AF3.

…52860) (apache#53372)" This reverts commit 7212c88.

…#53372)" (#54002) This reverts commit 7212c88.

jason810496 · 2025-07-04T10:21:10Z

related: #52581

Why

When I attempt to remove gunicorn dependency in #52581, I found that we are still using gunicorn for daemonization.

What

Replace gunicorn.daemonize with run_command_with_daemon_option, this also align the behavior of --daemon option for scheduler, triggerer ... and api-server
Add test for api-server with --daemon as well
- Rename _CommonCLIGunicornTestClass to _CommonCLIUvicornTestClass, since we are using uvicorn instead of gunicorn for api-server now.
Replace [api/access_logfile] config with [api/log_config], detail in Remove gunicorn daemonize for api-server #52860 (comment)

jason810496

access_log=access_logfile, # type: ignore[arg-type] this line is a strange part for me.

I add # type: ignore[arg-type] to ingore the mypy warning for now.

jason810496 · 2025-07-05T15:30:25Z

Hi @potiuk, may I might need hints for replacing config.

I thought replacing them in config.yaml and add the pair in deprecated_option in configuration will be all right.

Thanks in advance!

Update:

I found the root cause.

My step is correct but I set [api/log_config] config default as None in config.yml and I set default=conf.get("api", "log_config") in cli_config module.
There is not need to set default=conf.get("api", "log_config") or should set fallback=None for the --log-config flag in cli_config, since the it is not mandatory.

jason810496

Thanks @potiuk and @pierrejeambrun for the review!
Just resolved all comments and fixed the CI as well.

potiuk

Nice. One NIT about updating newsfragment.

- the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72)

* [v3-0-test] Remove gunicorn daemonize for api-server (#52860) * Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72) * Fix spelling wordlist * Fix configuration

jedcunningham · 2025-07-31T18:27:38Z

 ("api", "require_confirmation_dag_change"): ("webserver", "require_confirmation_dag_change", "3.1.0"),
 ("api", "instance_name"): ("webserver", "instance_name", "3.1.0"),
 ("dag_processor", "parsing_pre_import_modules"): ("scheduler", "parsing_pre_import_modules", "3.1.0"),
+ ("api", "log_config"): ("api", "access_logfile", "3.1.0"),


This is not safe to do - they are different config options. As mentioned in the comment above, this will use the old access_logfile value for log_config if it's set, which isn't what we want.

jedcunningham · 2025-07-31T19:12:11Z

+Replace API server ``access_logfile`` configuration with ``log_config``
+
+The API server configuration option ``[api] access_logfile`` has been replaced with ``[api] log_config`` to align with uvicorn's logging configuration instead of the legacy gunicorn approach.
+The new ``log_config`` option accepts a path to a logging configuration file compatible with ``logging.config.fileConfig``, providing more flexible logging configuration for the API server.


Is there a way we can continue to support a simple access logfile config? I just worry a bit about backcompat, right now this is a breaking change. Maybe we leave access_logfile, build our own log_config file dynamically, and make log_config and access_logfile mutually exclusive? And we should emit a deprecation warning when access_logfile is used.

Well, good and bad news. Good news is no backcompat issues to worry about! I believe it got broken in #45103. Bad news is I don't think access_log ever worked in AF3.

…52860) (apache#53372)" This reverts commit 7212c88.

…#53372)" (#54002) This reverts commit 7212c88.

jason810496 · 2025-07-04T10:21:10Z

related: #52581

Why

When I attempt to remove gunicorn dependency in #52581, I found that we are still using gunicorn for daemonization.

What

Replace gunicorn.daemonize with run_command_with_daemon_option, this also align the behavior of --daemon option for scheduler, triggerer ... and api-server
Add test for api-server with --daemon as well
- Rename _CommonCLIGunicornTestClass to _CommonCLIUvicornTestClass, since we are using uvicorn instead of gunicorn for api-server now.
Replace [api/access_logfile] config with [api/log_config], detail in Remove gunicorn daemonize for api-server #52860 (comment)

jason810496

access_log=access_logfile, # type: ignore[arg-type] this line is a strange part for me.

I add # type: ignore[arg-type] to ingore the mypy warning for now.

jason810496 · 2025-07-05T15:30:25Z

Hi @potiuk, may I might need hints for replacing config.

I thought replacing them in config.yaml and add the pair in deprecated_option in configuration will be all right.

Thanks in advance!

Update:

I found the root cause.

My step is correct but I set [api/log_config] config default as None in config.yml and I set default=conf.get("api", "log_config") in cli_config module.
There is not need to set default=conf.get("api", "log_config") or should set fallback=None for the --log-config flag in cli_config, since the it is not mandatory.

jason810496

Thanks @potiuk and @pierrejeambrun for the review!
Just resolved all comments and fixed the CI as well.

potiuk

Nice. One NIT about updating newsfragment.

- the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72)

* [v3-0-test] Remove gunicorn daemonize for api-server (#52860) * Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72) * Fix spelling wordlist * Fix configuration

jedcunningham · 2025-07-31T18:27:38Z

 ("api", "require_confirmation_dag_change"): ("webserver", "require_confirmation_dag_change", "3.1.0"),
 ("api", "instance_name"): ("webserver", "instance_name", "3.1.0"),
 ("dag_processor", "parsing_pre_import_modules"): ("scheduler", "parsing_pre_import_modules", "3.1.0"),
+ ("api", "log_config"): ("api", "access_logfile", "3.1.0"),


This is not safe to do - they are different config options. As mentioned in the comment above, this will use the old access_logfile value for log_config if it's set, which isn't what we want.

jedcunningham · 2025-07-31T19:12:11Z

+Replace API server ``access_logfile`` configuration with ``log_config``
+
+The API server configuration option ``[api] access_logfile`` has been replaced with ``[api] log_config`` to align with uvicorn's logging configuration instead of the legacy gunicorn approach.
+The new ``log_config`` option accepts a path to a logging configuration file compatible with ``logging.config.fileConfig``, providing more flexible logging configuration for the API server.


Is there a way we can continue to support a simple access logfile config? I just worry a bit about backcompat, right now this is a breaking change. Maybe we leave access_logfile, build our own log_config file dynamically, and make log_config and access_logfile mutually exclusive? And we should emit a deprecation warning when access_logfile is used.

Well, good and bad news. Good news is no backcompat issues to worry about! I believe it got broken in #45103. Bad news is I don't think access_log ever worked in AF3.

…52860) (apache#53372)" This reverts commit 7212c88.

…#53372)" (#54002) This reverts commit 7212c88.

jason810496 · 2025-07-04T10:21:10Z

related: #52581

Why

When I attempt to remove gunicorn dependency in #52581, I found that we are still using gunicorn for daemonization.

What

Replace gunicorn.daemonize with run_command_with_daemon_option, this also align the behavior of --daemon option for scheduler, triggerer ... and api-server
Add test for api-server with --daemon as well
- Rename _CommonCLIGunicornTestClass to _CommonCLIUvicornTestClass, since we are using uvicorn instead of gunicorn for api-server now.
Replace [api/access_logfile] config with [api/log_config], detail in Remove gunicorn daemonize for api-server #52860 (comment)

jason810496

access_log=access_logfile, # type: ignore[arg-type] this line is a strange part for me.

I add # type: ignore[arg-type] to ingore the mypy warning for now.

jason810496 · 2025-07-05T15:30:25Z

Hi @potiuk, may I might need hints for replacing config.

I thought replacing them in config.yaml and add the pair in deprecated_option in configuration will be all right.

Thanks in advance!

Update:

I found the root cause.

My step is correct but I set [api/log_config] config default as None in config.yml and I set default=conf.get("api", "log_config") in cli_config module.
There is not need to set default=conf.get("api", "log_config") or should set fallback=None for the --log-config flag in cli_config, since the it is not mandatory.

jason810496

Thanks @potiuk and @pierrejeambrun for the review!
Just resolved all comments and fixed the CI as well.

potiuk

Nice. One NIT about updating newsfragment.

- the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72)

* [v3-0-test] Remove gunicorn daemonize for api-server (#52860) * Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72) * Fix spelling wordlist * Fix configuration

jedcunningham · 2025-07-31T18:27:38Z

 ("api", "require_confirmation_dag_change"): ("webserver", "require_confirmation_dag_change", "3.1.0"),
 ("api", "instance_name"): ("webserver", "instance_name", "3.1.0"),
 ("dag_processor", "parsing_pre_import_modules"): ("scheduler", "parsing_pre_import_modules", "3.1.0"),
+ ("api", "log_config"): ("api", "access_logfile", "3.1.0"),


This is not safe to do - they are different config options. As mentioned in the comment above, this will use the old access_logfile value for log_config if it's set, which isn't what we want.

jedcunningham · 2025-07-31T19:12:11Z

+Replace API server ``access_logfile`` configuration with ``log_config``
+
+The API server configuration option ``[api] access_logfile`` has been replaced with ``[api] log_config`` to align with uvicorn's logging configuration instead of the legacy gunicorn approach.
+The new ``log_config`` option accepts a path to a logging configuration file compatible with ``logging.config.fileConfig``, providing more flexible logging configuration for the API server.


Is there a way we can continue to support a simple access logfile config? I just worry a bit about backcompat, right now this is a breaking change. Maybe we leave access_logfile, build our own log_config file dynamically, and make log_config and access_logfile mutually exclusive? And we should emit a deprecation warning when access_logfile is used.

Well, good and bad news. Good news is no backcompat issues to worry about! I believe it got broken in #45103. Bad news is I don't think access_log ever worked in AF3.

…52860) (apache#53372)" This reverts commit 7212c88.

…#53372)" (#54002) This reverts commit 7212c88.

jason810496 · 2025-07-04T10:21:10Z

related: #52581

Why

When I attempt to remove gunicorn dependency in #52581, I found that we are still using gunicorn for daemonization.

What

Replace gunicorn.daemonize with run_command_with_daemon_option, this also align the behavior of --daemon option for scheduler, triggerer ... and api-server
Add test for api-server with --daemon as well
- Rename _CommonCLIGunicornTestClass to _CommonCLIUvicornTestClass, since we are using uvicorn instead of gunicorn for api-server now.
Replace [api/access_logfile] config with [api/log_config], detail in Remove gunicorn daemonize for api-server #52860 (comment)

jason810496

access_log=access_logfile, # type: ignore[arg-type] this line is a strange part for me.

I add # type: ignore[arg-type] to ingore the mypy warning for now.

jason810496 · 2025-07-05T15:30:25Z

Hi @potiuk, may I might need hints for replacing config.

I thought replacing them in config.yaml and add the pair in deprecated_option in configuration will be all right.

Thanks in advance!

Update:

I found the root cause.

My step is correct but I set [api/log_config] config default as None in config.yml and I set default=conf.get("api", "log_config") in cli_config module.
There is not need to set default=conf.get("api", "log_config") or should set fallback=None for the --log-config flag in cli_config, since the it is not mandatory.

jason810496

Thanks @potiuk and @pierrejeambrun for the review!
Just resolved all comments and fixed the CI as well.

potiuk

Nice. One NIT about updating newsfragment.

- the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72)

* [v3-0-test] Remove gunicorn daemonize for api-server (#52860) * Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72) * Fix spelling wordlist * Fix configuration

jedcunningham · 2025-07-31T18:27:38Z

 ("api", "require_confirmation_dag_change"): ("webserver", "require_confirmation_dag_change", "3.1.0"),
 ("api", "instance_name"): ("webserver", "instance_name", "3.1.0"),
 ("dag_processor", "parsing_pre_import_modules"): ("scheduler", "parsing_pre_import_modules", "3.1.0"),
+ ("api", "log_config"): ("api", "access_logfile", "3.1.0"),


This is not safe to do - they are different config options. As mentioned in the comment above, this will use the old access_logfile value for log_config if it's set, which isn't what we want.

jedcunningham · 2025-07-31T19:12:11Z

+Replace API server ``access_logfile`` configuration with ``log_config``
+
+The API server configuration option ``[api] access_logfile`` has been replaced with ``[api] log_config`` to align with uvicorn's logging configuration instead of the legacy gunicorn approach.
+The new ``log_config`` option accepts a path to a logging configuration file compatible with ``logging.config.fileConfig``, providing more flexible logging configuration for the API server.


Is there a way we can continue to support a simple access logfile config? I just worry a bit about backcompat, right now this is a breaking change. Maybe we leave access_logfile, build our own log_config file dynamically, and make log_config and access_logfile mutually exclusive? And we should emit a deprecation warning when access_logfile is used.

Well, good and bad news. Good news is no backcompat issues to worry about! I believe it got broken in #45103. Bad news is I don't think access_log ever worked in AF3.

…52860) (apache#53372)" This reverts commit 7212c88.

…#53372)" (#54002) This reverts commit 7212c88.

jason810496 · 2025-07-04T10:21:10Z

related: #52581

Why

When I attempt to remove gunicorn dependency in #52581, I found that we are still using gunicorn for daemonization.

What

Replace gunicorn.daemonize with run_command_with_daemon_option, this also align the behavior of --daemon option for scheduler, triggerer ... and api-server
Add test for api-server with --daemon as well
- Rename _CommonCLIGunicornTestClass to _CommonCLIUvicornTestClass, since we are using uvicorn instead of gunicorn for api-server now.
Replace [api/access_logfile] config with [api/log_config], detail in Remove gunicorn daemonize for api-server #52860 (comment)

jason810496

access_log=access_logfile, # type: ignore[arg-type] this line is a strange part for me.

I add # type: ignore[arg-type] to ingore the mypy warning for now.

jason810496 · 2025-07-05T15:30:25Z

Hi @potiuk, may I might need hints for replacing config.

I thought replacing them in config.yaml and add the pair in deprecated_option in configuration will be all right.

Thanks in advance!

Update:

I found the root cause.

My step is correct but I set [api/log_config] config default as None in config.yml and I set default=conf.get("api", "log_config") in cli_config module.
There is not need to set default=conf.get("api", "log_config") or should set fallback=None for the --log-config flag in cli_config, since the it is not mandatory.

jason810496

Thanks @potiuk and @pierrejeambrun for the review!
Just resolved all comments and fixed the CI as well.

potiuk

Nice. One NIT about updating newsfragment.

- the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst

* Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72)

* [v3-0-test] Remove gunicorn daemonize for api-server (#52860) * Remove gunicorn daemonize for api-server * Rename _CommonCLIUvicornTestClass * Add test for api-server with daemon option * Fix setproctitle on MacOS * Replace api/access_logfile with api/log_config * Fix [api/log_config] cli_config * Remove logging and unsed option in api_server_command - the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False * Fix test_run_command_daemon * Add significant rst (cherry picked from commit ab1ee72) * Fix spelling wordlist * Fix configuration

jedcunningham · 2025-07-31T18:27:38Z

 ("api", "require_confirmation_dag_change"): ("webserver", "require_confirmation_dag_change", "3.1.0"),
 ("api", "instance_name"): ("webserver", "instance_name", "3.1.0"),
 ("dag_processor", "parsing_pre_import_modules"): ("scheduler", "parsing_pre_import_modules", "3.1.0"),
+ ("api", "log_config"): ("api", "access_logfile", "3.1.0"),


This is not safe to do - they are different config options. As mentioned in the comment above, this will use the old access_logfile value for log_config if it's set, which isn't what we want.

jedcunningham · 2025-07-31T19:12:11Z

+Replace API server ``access_logfile`` configuration with ``log_config``
+
+The API server configuration option ``[api] access_logfile`` has been replaced with ``[api] log_config`` to align with uvicorn's logging configuration instead of the legacy gunicorn approach.
+The new ``log_config`` option accepts a path to a logging configuration file compatible with ``logging.config.fileConfig``, providing more flexible logging configuration for the API server.


Is there a way we can continue to support a simple access logfile config? I just worry a bit about backcompat, right now this is a breaking change. Maybe we leave access_logfile, build our own log_config file dynamically, and make log_config and access_logfile mutually exclusive? And we should emit a deprecation warning when access_logfile is used.

Well, good and bad news. Good news is no backcompat issues to worry about! I believe it got broken in #45103. Bad news is I don't think access_log ever worked in AF3.

…52860) (apache#53372)" This reverts commit 7212c88.

…#53372)" (#54002) This reverts commit 7212c88.

boring-cyborgBot added the area:CLI label Jul 4, 2025

jason810496 force-pushed the refactor/api-server/remove-gunicorn-daemonize branch from 1dfe755 to 7a44f9aCompare July 4, 2025 10:22

jason810496 added the area:API Airflow's REST/HTTP API label Jul 4, 2025

jason810496 marked this pull request as ready for review July 4, 2025 10:22

jason810496 commented Jul 4, 2025
View reviewed changes

Comment threadairflow-core/src/airflow/cli/commands/api_server_command.py Outdated

jason810496 requested review from pierrejeambrun and potiuk July 4, 2025 10:29

potiuk reviewed Jul 4, 2025
View reviewed changes

Comment threadairflow-core/src/airflow/cli/commands/api_server_command.py

potiuk reviewed Jul 4, 2025
View reviewed changes

Comment threadairflow-core/src/airflow/cli/commands/api_server_command.py Outdated

jason810496 marked this pull request as draft July 6, 2025 02:29

jason810496 force-pushed the refactor/api-server/remove-gunicorn-daemonize branch from 89b529e to 52151feCompare July 6, 2025 03:08

jason810496 marked this pull request as ready for review July 6, 2025 04:30

jason810496 commented Jul 6, 2025
View reviewed changes

jason810496 mentioned this pull request Jul 6, 2025
Respect apps flags for api_server_command #52929
Merged

jason810496 self-assigned this Jul 6, 2025

potiuk reviewed Jul 6, 2025
View reviewed changes

Comment threadairflow-core/src/airflow/config_templates/config.yml

potiuk approved these changes Jul 6, 2025
View reviewed changes

jason810496 added 9 commits July 7, 2025 22:02

Remove gunicorn daemonize for api-server
8be9cc5

Rename _CommonCLIUvicornTestClass
bcdf81e

Add test for api-server with daemon option
215f40a

Fix setproctitle on MacOS
694ffa6

Replace api/access_logfile with api/log_config
9715977

Fix [api/log_config] cli_config
ebaada7

Remove logging and unsed option in api_server_command
0bf37d4
- the logging before daemonization is not logging the correct PID so remove it - should_setup_logging is default as False

Fix test_run_command_daemon
40c356c

Add significant rst
ef4c730

jason810496 force-pushed the refactor/api-server/remove-gunicorn-daemonize branch from 52151fe to ef4c730Compare July 7, 2025 14:02

potiuk approved these changes Jul 7, 2025
View reviewed changes

potiuk merged commit ab1ee72 into apache:mainJul 7, 2025
1 check passed

This was referenced Jul 10, 2025
Fix setproctitle usage #53122
Merged
Cleanup gunicorn_config module #53123
Merged

jason810496 mentioned this pull request Jul 15, 2025
[v3-0-test] Remove gunicorn daemonize for api-server (#52860) #53372
Merged

jedcunningham reviewed Jul 31, 2025
View reviewed changes

jedcunningham added a commit to astronomer/airflow that referenced this pull request Jul 31, 2025
Revert "[v3-0-test] Remove gunicorn daemonize for api-server (apache#…
3fdcefa
…52860) (apache#53372)" This reverts commit 7212c88.

jedcunningham mentioned this pull request Jul 31, 2025
Revert "[v3-0-test] Remove gunicorn daemonize for api-server (#52860) (#53372)" #54002
Merged

jedcunningham added a commit that referenced this pull request Aug 1, 2025
Revert "[v3-0-test] Remove gunicorn daemonize for api-server (#52860) (…
69bb85b
…#53372)" (#54002) This reverts commit 7212c88.

ashb mentioned this pull request Aug 5, 2025
Status of testing Apache Airflow 3.0.4rc2 and TaskSDK 1.0.4rc1 #54124
Closed
82 tasks

eladkal mentioned this pull request Sep 5, 2025
Status of testing Providers that were prepared on September 05, 2025 #55285
Closed