Replacing gunicornmontor with uvicorn.run() by vatsrahul1001 · Pull Request #45103 · apache/airflow

vatsrahul1001 · 2024-12-20T07:12:36Z

Related: #43035

These are the stats comparing uvicorn.run() with Gunicorn + GunicornMonitor

Comparison: Uvicorn vs. Gunicorn Performance

Request Statistics

Metric	Uvicorn	Gunicorn
Total Requests	14,714	14,726
Total Failures	0	13
Average Response Time	12.05 ms	13.46 ms
Min Response Time	7 ms	1 ms
Max Response Time	195 ms	216 ms
Average Size (bytes)	4,608	4,603.93
Requests Per Second (RPS)	49.05	49.09
Failures Per Second	0	0.04

Observations

Response Times:
- Uvicorn demonstrates slightly lower average and maximum response times compared to Gunicorn.
- Percentile analysis shows Uvicorn's response times are more consistent, with fewer extreme values at higher percentiles.
Failures:
- Uvicorn had no failures, whereas Gunicorn recorded 13 failures caused by RemoteDisconnected errors. This could indicate potential issues in connection handling under load.
Performance Consistency:
- Uvicorn offers better consistency and reliability based on the above data.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

…icorn-with-uvicorn-run

pierrejeambrun

All for it. 👍

Thanks for the metrics that encourage that this is the good choice.

Just a small question.

…omer/airflow into replace-gunicorn-with-uvicorn-run

pierrejeambrun

Approved, looks good.

Leaving some time for others to participate in the discussion.

pierrejeambrun

CI need fixing, most likely CLI tests need to be updated.

…omer/airflow into replace-gunicorn-with-uvicorn-run

vatsrahul1001 · 2025-01-24T04:28:43Z

CI need fixing, most likely CLI tests need to be updated.

Fixed tests after updating setproctitle

pierrejeambrun · 2025-01-24T09:02:47Z

Ok, I think we left enough time for people to participated in case they feel that some other options are necessary. We can always improve later, merging.

* replace gunicorm with uvicorn.run() * fixing tests * Daemonized fastapi server * fixing setproctitle format * updating setproctitle

vatsrahul1001 · 2024-12-20T07:12:36Z

Related: #43035

These are the stats comparing uvicorn.run() with Gunicorn + GunicornMonitor

Comparison: Uvicorn vs. Gunicorn Performance

Request Statistics

Metric	Uvicorn	Gunicorn
Total Requests	14,714	14,726
Total Failures	0	13
Average Response Time	12.05 ms	13.46 ms
Min Response Time	7 ms	1 ms
Max Response Time	195 ms	216 ms
Average Size (bytes)	4,608	4,603.93
Requests Per Second (RPS)	49.05	49.09
Failures Per Second	0	0.04

Observations

Response Times:
- Uvicorn demonstrates slightly lower average and maximum response times compared to Gunicorn.
- Percentile analysis shows Uvicorn's response times are more consistent, with fewer extreme values at higher percentiles.
Failures:
- Uvicorn had no failures, whereas Gunicorn recorded 13 failures caused by RemoteDisconnected errors. This could indicate potential issues in connection handling under load.
Performance Consistency:
- Uvicorn offers better consistency and reliability based on the above data.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

…icorn-with-uvicorn-run

pierrejeambrun

All for it. 👍

Thanks for the metrics that encourage that this is the good choice.

Just a small question.

…omer/airflow into replace-gunicorn-with-uvicorn-run

pierrejeambrun

Approved, looks good.

Leaving some time for others to participate in the discussion.

pierrejeambrun

CI need fixing, most likely CLI tests need to be updated.

…omer/airflow into replace-gunicorn-with-uvicorn-run

vatsrahul1001 · 2025-01-24T04:28:43Z

CI need fixing, most likely CLI tests need to be updated.

Fixed tests after updating setproctitle

pierrejeambrun · 2025-01-24T09:02:47Z

Ok, I think we left enough time for people to participated in case they feel that some other options are necessary. We can always improve later, merging.

* replace gunicorm with uvicorn.run() * fixing tests * Daemonized fastapi server * fixing setproctitle format * updating setproctitle

vatsrahul1001 · 2024-12-20T07:12:36Z

Related: #43035

These are the stats comparing uvicorn.run() with Gunicorn + GunicornMonitor

Comparison: Uvicorn vs. Gunicorn Performance

Request Statistics

Metric	Uvicorn	Gunicorn
Total Requests	14,714	14,726
Total Failures	0	13
Average Response Time	12.05 ms	13.46 ms
Min Response Time	7 ms	1 ms
Max Response Time	195 ms	216 ms
Average Size (bytes)	4,608	4,603.93
Requests Per Second (RPS)	49.05	49.09
Failures Per Second	0	0.04

Observations

Response Times:
- Uvicorn demonstrates slightly lower average and maximum response times compared to Gunicorn.
- Percentile analysis shows Uvicorn's response times are more consistent, with fewer extreme values at higher percentiles.
Failures:
- Uvicorn had no failures, whereas Gunicorn recorded 13 failures caused by RemoteDisconnected errors. This could indicate potential issues in connection handling under load.
Performance Consistency:
- Uvicorn offers better consistency and reliability based on the above data.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

…icorn-with-uvicorn-run

pierrejeambrun

All for it. 👍

Thanks for the metrics that encourage that this is the good choice.

Just a small question.

…omer/airflow into replace-gunicorn-with-uvicorn-run

pierrejeambrun

Approved, looks good.

Leaving some time for others to participate in the discussion.

pierrejeambrun

CI need fixing, most likely CLI tests need to be updated.

…omer/airflow into replace-gunicorn-with-uvicorn-run

vatsrahul1001 · 2025-01-24T04:28:43Z

CI need fixing, most likely CLI tests need to be updated.

Fixed tests after updating setproctitle

pierrejeambrun · 2025-01-24T09:02:47Z

Ok, I think we left enough time for people to participated in case they feel that some other options are necessary. We can always improve later, merging.

* replace gunicorm with uvicorn.run() * fixing tests * Daemonized fastapi server * fixing setproctitle format * updating setproctitle

vatsrahul1001 · 2024-12-20T07:12:36Z

Related: #43035

These are the stats comparing uvicorn.run() with Gunicorn + GunicornMonitor

Comparison: Uvicorn vs. Gunicorn Performance

Request Statistics

Metric	Uvicorn	Gunicorn
Total Requests	14,714	14,726
Total Failures	0	13
Average Response Time	12.05 ms	13.46 ms
Min Response Time	7 ms	1 ms
Max Response Time	195 ms	216 ms
Average Size (bytes)	4,608	4,603.93
Requests Per Second (RPS)	49.05	49.09
Failures Per Second	0	0.04

Observations

Response Times:
- Uvicorn demonstrates slightly lower average and maximum response times compared to Gunicorn.
- Percentile analysis shows Uvicorn's response times are more consistent, with fewer extreme values at higher percentiles.
Failures:
- Uvicorn had no failures, whereas Gunicorn recorded 13 failures caused by RemoteDisconnected errors. This could indicate potential issues in connection handling under load.
Performance Consistency:
- Uvicorn offers better consistency and reliability based on the above data.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

…icorn-with-uvicorn-run

pierrejeambrun

All for it. 👍

Thanks for the metrics that encourage that this is the good choice.

Just a small question.

…omer/airflow into replace-gunicorn-with-uvicorn-run

pierrejeambrun

Approved, looks good.

Leaving some time for others to participate in the discussion.

pierrejeambrun

CI need fixing, most likely CLI tests need to be updated.

…omer/airflow into replace-gunicorn-with-uvicorn-run

vatsrahul1001 · 2025-01-24T04:28:43Z

CI need fixing, most likely CLI tests need to be updated.

Fixed tests after updating setproctitle

pierrejeambrun · 2025-01-24T09:02:47Z

Ok, I think we left enough time for people to participated in case they feel that some other options are necessary. We can always improve later, merging.

* replace gunicorm with uvicorn.run() * fixing tests * Daemonized fastapi server * fixing setproctitle format * updating setproctitle

vatsrahul1001 · 2024-12-20T07:12:36Z

Related: #43035

These are the stats comparing uvicorn.run() with Gunicorn + GunicornMonitor

Comparison: Uvicorn vs. Gunicorn Performance

Request Statistics

Metric	Uvicorn	Gunicorn
Total Requests	14,714	14,726
Total Failures	0	13
Average Response Time	12.05 ms	13.46 ms
Min Response Time	7 ms	1 ms
Max Response Time	195 ms	216 ms
Average Size (bytes)	4,608	4,603.93
Requests Per Second (RPS)	49.05	49.09
Failures Per Second	0	0.04

Observations

Response Times:
- Uvicorn demonstrates slightly lower average and maximum response times compared to Gunicorn.
- Percentile analysis shows Uvicorn's response times are more consistent, with fewer extreme values at higher percentiles.
Failures:
- Uvicorn had no failures, whereas Gunicorn recorded 13 failures caused by RemoteDisconnected errors. This could indicate potential issues in connection handling under load.
Performance Consistency:
- Uvicorn offers better consistency and reliability based on the above data.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

…icorn-with-uvicorn-run

pierrejeambrun

All for it. 👍

Thanks for the metrics that encourage that this is the good choice.

Just a small question.

…omer/airflow into replace-gunicorn-with-uvicorn-run

pierrejeambrun

Approved, looks good.

Leaving some time for others to participate in the discussion.

pierrejeambrun

CI need fixing, most likely CLI tests need to be updated.

…omer/airflow into replace-gunicorn-with-uvicorn-run

vatsrahul1001 · 2025-01-24T04:28:43Z

CI need fixing, most likely CLI tests need to be updated.

Fixed tests after updating setproctitle

pierrejeambrun · 2025-01-24T09:02:47Z

Ok, I think we left enough time for people to participated in case they feel that some other options are necessary. We can always improve later, merging.

* replace gunicorm with uvicorn.run() * fixing tests * Daemonized fastapi server * fixing setproctitle format * updating setproctitle

vatsrahul1001 · 2024-12-20T07:12:36Z

Related: #43035

These are the stats comparing uvicorn.run() with Gunicorn + GunicornMonitor

Comparison: Uvicorn vs. Gunicorn Performance

Request Statistics

Metric	Uvicorn	Gunicorn
Total Requests	14,714	14,726
Total Failures	0	13
Average Response Time	12.05 ms	13.46 ms
Min Response Time	7 ms	1 ms
Max Response Time	195 ms	216 ms
Average Size (bytes)	4,608	4,603.93
Requests Per Second (RPS)	49.05	49.09
Failures Per Second	0	0.04

Observations

Response Times:
- Uvicorn demonstrates slightly lower average and maximum response times compared to Gunicorn.
- Percentile analysis shows Uvicorn's response times are more consistent, with fewer extreme values at higher percentiles.
Failures:
- Uvicorn had no failures, whereas Gunicorn recorded 13 failures caused by RemoteDisconnected errors. This could indicate potential issues in connection handling under load.
Performance Consistency:
- Uvicorn offers better consistency and reliability based on the above data.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

…icorn-with-uvicorn-run

pierrejeambrun

All for it. 👍

Thanks for the metrics that encourage that this is the good choice.

Just a small question.

…omer/airflow into replace-gunicorn-with-uvicorn-run

pierrejeambrun

Approved, looks good.

Leaving some time for others to participate in the discussion.

pierrejeambrun

CI need fixing, most likely CLI tests need to be updated.

…omer/airflow into replace-gunicorn-with-uvicorn-run

vatsrahul1001 · 2025-01-24T04:28:43Z

CI need fixing, most likely CLI tests need to be updated.

Fixed tests after updating setproctitle

pierrejeambrun · 2025-01-24T09:02:47Z

Ok, I think we left enough time for people to participated in case they feel that some other options are necessary. We can always improve later, merging.

* replace gunicorm with uvicorn.run() * fixing tests * Daemonized fastapi server * fixing setproctitle format * updating setproctitle

vatsrahul1001 · 2024-12-20T07:12:36Z

Related: #43035

These are the stats comparing uvicorn.run() with Gunicorn + GunicornMonitor

Comparison: Uvicorn vs. Gunicorn Performance

Request Statistics

Metric	Uvicorn	Gunicorn
Total Requests	14,714	14,726
Total Failures	0	13
Average Response Time	12.05 ms	13.46 ms
Min Response Time	7 ms	1 ms
Max Response Time	195 ms	216 ms
Average Size (bytes)	4,608	4,603.93
Requests Per Second (RPS)	49.05	49.09
Failures Per Second	0	0.04

Observations

Response Times:
- Uvicorn demonstrates slightly lower average and maximum response times compared to Gunicorn.
- Percentile analysis shows Uvicorn's response times are more consistent, with fewer extreme values at higher percentiles.
Failures:
- Uvicorn had no failures, whereas Gunicorn recorded 13 failures caused by RemoteDisconnected errors. This could indicate potential issues in connection handling under load.
Performance Consistency:
- Uvicorn offers better consistency and reliability based on the above data.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

…icorn-with-uvicorn-run

pierrejeambrun

All for it. 👍

Thanks for the metrics that encourage that this is the good choice.

Just a small question.

…omer/airflow into replace-gunicorn-with-uvicorn-run

pierrejeambrun

Approved, looks good.

Leaving some time for others to participate in the discussion.

pierrejeambrun

CI need fixing, most likely CLI tests need to be updated.

…omer/airflow into replace-gunicorn-with-uvicorn-run

vatsrahul1001 · 2025-01-24T04:28:43Z

CI need fixing, most likely CLI tests need to be updated.

Fixed tests after updating setproctitle

pierrejeambrun · 2025-01-24T09:02:47Z

Ok, I think we left enough time for people to participated in case they feel that some other options are necessary. We can always improve later, merging.

* replace gunicorm with uvicorn.run() * fixing tests * Daemonized fastapi server * fixing setproctitle format * updating setproctitle

vatsrahul1001 · 2024-12-20T07:12:36Z

Related: #43035

These are the stats comparing uvicorn.run() with Gunicorn + GunicornMonitor

Comparison: Uvicorn vs. Gunicorn Performance

Request Statistics

Metric	Uvicorn	Gunicorn
Total Requests	14,714	14,726
Total Failures	0	13
Average Response Time	12.05 ms	13.46 ms
Min Response Time	7 ms	1 ms
Max Response Time	195 ms	216 ms
Average Size (bytes)	4,608	4,603.93
Requests Per Second (RPS)	49.05	49.09
Failures Per Second	0	0.04

Observations

Response Times:
- Uvicorn demonstrates slightly lower average and maximum response times compared to Gunicorn.
- Percentile analysis shows Uvicorn's response times are more consistent, with fewer extreme values at higher percentiles.
Failures:
- Uvicorn had no failures, whereas Gunicorn recorded 13 failures caused by RemoteDisconnected errors. This could indicate potential issues in connection handling under load.
Performance Consistency:
- Uvicorn offers better consistency and reliability based on the above data.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

…icorn-with-uvicorn-run

pierrejeambrun

All for it. 👍

Thanks for the metrics that encourage that this is the good choice.

Just a small question.

…omer/airflow into replace-gunicorn-with-uvicorn-run

pierrejeambrun

Approved, looks good.

Leaving some time for others to participate in the discussion.

pierrejeambrun

CI need fixing, most likely CLI tests need to be updated.

…omer/airflow into replace-gunicorn-with-uvicorn-run

vatsrahul1001 · 2025-01-24T04:28:43Z

CI need fixing, most likely CLI tests need to be updated.

Fixed tests after updating setproctitle

pierrejeambrun · 2025-01-24T09:02:47Z

Ok, I think we left enough time for people to participated in case they feel that some other options are necessary. We can always improve later, merging.

* replace gunicorm with uvicorn.run() * fixing tests * Daemonized fastapi server * fixing setproctitle format * updating setproctitle

replace gunicorm with uvicorn.run()
0b6774a

boring-cyborgBot added the area:CLI label Dec 20, 2024

vatsrahul1001 marked this pull request as draft December 20, 2024 08:27

Merge branch 'main' of github.com:astronomer/airflow into replace-gun…
85305fc
…icorn-with-uvicorn-run

vatsrahul1001 force-pushed the replace-gunicorn-with-uvicorn-run branch from 1a74394 to 85305fcCompare December 31, 2024 06:19

fixing tests
9d4a1ec

vatsrahul1001 marked this pull request as ready for review December 31, 2024 07:01

Merge branch 'main' into replace-gunicorn-with-uvicorn-run
4b60ca7

phanikumv requested review from kaxil and pierrejeambrun January 6, 2025 06:51

kaxil requested review from ashb and jscheffl January 6, 2025 13:56

pierrejeambrun reviewed Jan 7, 2025
View reviewed changes

Comment threadairflow/cli/commands/local_commands/fastapi_api_command.py
Comment threadairflow/cli/commands/local_commands/fastapi_api_command.py

vatsrahul1001 requested a review from bbovenzi January 17, 2025 04:09

vatsrahul1001and others added 4 commits January 21, 2025 09:21

Merge branch 'main' into replace-gunicorn-with-uvicorn-run
a167f94

Daemonized fastapi server
7ceea79

Merge branch 'replace-gunicorn-with-uvicorn-run' of github.com:astron…
eb85115
…omer/airflow into replace-gunicorn-with-uvicorn-run

Merge branch 'main' into replace-gunicorn-with-uvicorn-run
1332d3d

vatsrahul1001 requested review from pierrejeambrun and uranusjr January 21, 2025 12:43

apache deleted a comment Jan 21, 2025

Merge branch 'main' into replace-gunicorn-with-uvicorn-run
613d466

pierrejeambrun approved these changes Jan 21, 2025
View reviewed changes

Comment threadairflow/cli/commands/local_commands/fastapi_api_command.py
Comment threadairflow/cli/commands/local_commands/fastapi_api_command.py Outdated

vatsrahul1001and others added 3 commits January 22, 2025 10:18

fixing setproctitle format
3dcf8f2

Merge branch 'main' into replace-gunicorn-with-uvicorn-run
fc3535e

Merge branch 'main' into replace-gunicorn-with-uvicorn-run
57e3cfe

vatsrahul1001 added 3 commits January 22, 2025 16:55

Merge branch 'main' into replace-gunicorn-with-uvicorn-run
d928ddb

Merge branch 'main' into replace-gunicorn-with-uvicorn-run
0ab6d7b

Merge branch 'main' into replace-gunicorn-with-uvicorn-run
49642e3

pierrejeambrun reviewed Jan 23, 2025
View reviewed changes

vatsrahul1001and others added 3 commits January 23, 2025 22:44

updating setproctitle
d667103

Merge branch 'replace-gunicorn-with-uvicorn-run' of github.com:astron…
410cf5a
…omer/airflow into replace-gunicorn-with-uvicorn-run

Merge branch 'main' into replace-gunicorn-with-uvicorn-run
0b3696e

Merge branch 'main' into replace-gunicorn-with-uvicorn-run
b381f7d

pierrejeambrun approved these changes Jan 24, 2025
View reviewed changes

pierrejeambrun merged commit 933b003 into apache:mainJan 24, 2025

pierrejeambrun deleted the replace-gunicorn-with-uvicorn-run branch January 24, 2025 09:03

vatsrahul1001 mentioned this pull request Jan 25, 2025
Investigate if we can replace gunicornmontor with uvicorn.run()#43035
Closed

utkarsharma2 added the type:improvement Changelog: Improvements label Jan 27, 2025

utkarsharma2 added this to the Airflow 3.0.0 milestone Jan 27, 2025

gpathak128 pushed a commit to gpathak128/airflow that referenced this pull request Jan 29, 2025
Replacing gunicornmontor with uvicorn.run() (apache#45103)
01437cd
* replace gunicorm with uvicorn.run() * fixing tests * Daemonized fastapi server * fixing setproctitle format * updating setproctitle

niklasr22 pushed a commit to niklasr22/airflow that referenced this pull request Feb 8, 2025
Replacing gunicornmontor with uvicorn.run() (apache#45103)
8991602
* replace gunicorm with uvicorn.run() * fixing tests * Daemonized fastapi server * fixing setproctitle format * updating setproctitle

jedcunningham mentioned this pull request Jul 31, 2025
Remove gunicorn daemonize for api-server #52860
Merged