Skip to content

Fix misleading error message when GitHook creation fails - #59084

Merged
amoghrajesh merged 6 commits into
apache:mainfrom
astronomer:dagmanager-error-handling
Dec 8, 2025
Merged

Fix misleading error message when GitHook creation fails#59084
amoghrajesh merged 6 commits into
apache:mainfrom
astronomer:dagmanager-error-handling

Conversation

@amoghrajesh

Copy link
Copy Markdown
Contributor

Previously, when GitHook creation failed (take for example a simple case of git connection not found), the exception was caught and suppressed, allowing bundle initialization to proceed in a way, which was a facade. This led to a misleading error message later: "Connection {conn_id} doesn't have a host url" instead of the actual error.

I am making this change to ensure that exceptions propagate immediately during bundle creation with clear error messages. The DAG processor handles exceptions per-bundle, so one failing bundle doesn't prevent others from loading successfully.

For testing, use a config like this where git connection doesnt exist:

export AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST='[ { "name": "test-git-bundle", "classpath": "airflow.providers.git.bundles.git.GitDagBundle", "kwargs": { "tracking_ref": "main", "git_conn_id": "non_existent_connection" } }, { "name": "local-bundle", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": { "path": "/Users/amoghdesai/Documents/OSS/repos/airflow/files/test_dags", "refresh_interval": 0 } }]'

Before it looked like this:

2025-12-05T07:18:54.567663Z [error ] Errorinitializingbundletest-git-bundle: Connectionnon_existent_connectiondoesn'thaveahosturl [airflow.dag_processing.manager.DagFileProcessorManager] loc=manager.py:528Traceback (mostrecentcalllast):
File"/opt/airflow/airflow-core/src/airflow/dag_processing/manager.py", line526, in_refresh_dag_bundlesbundle.initialize()
File"/opt/airflow/providers/git/src/airflow/providers/git/bundles/git.py", line134, ininitializeraiseAirflowException(f"Connection {self.git_conn_id} doesn't have a host url")
airflow.sdk.exceptions.AirflowException: Connectionnon_existent_connectiondoesn'thaveahosturl2025-12-05T07:18:59.647017Z [error ] Errorinitializingbundletest-git-bundle: Connectionnon_existent_connectiondoesn'thaveahosturl [airflow.dag_processing.manager.DagFileProcessorManager] loc=manager.py:528Traceback (mostrecentcalllast):
File"/opt/airflow/airflow-core/src/airflow/dag_processing/manager.py", line526, in_refresh_dag_bundlesbundle.initialize()
File"/opt/airflow/providers/git/src/airflow/providers/git/bundles/git.py", line134, ininitializeraiseAirflowException(f"Connection {self.git_conn_id} doesn't have a host url")
airflow.sdk.exceptions.AirflowException: Connectionnon_existent_connectiondoesn'thaveahosturl

Now it looks like this:

root@c435d9bb0884:/opt/airflow# airflow dag-processor2025-12-05T08:45:11.600794Z [warning ] Failedtoconvertvalue. Pleasecheckmemray_trace_componentskeyinprofilingsection. itmustbeoneofscheduler, dag_processor, api, ifnotthevalueisignored [airflow._shared.configuration.parser] loc=parser.py:1127__file__='/files/plugins/triggera.py'loaded__file__='/files/plugins/triggera_comprehensive.py'loaded2025-12-05T08:45:12.101992Z [info ] StartingtheDagProcessorJob [airflow.jobs.dag_processor_job_runner.DagProcessorJobRunner] loc=dag_processor_job_runner.py:592025-12-05T08:45:12.102262Z [info ] Processingfilesusingupto2processesatatime [airflow.dag_processing.manager.DagFileProcessorManager] loc=manager.py:2662025-12-05T08:45:12.102305Z [info ] Processeachfileatmostonceevery30seconds [airflow.dag_processing.manager.DagFileProcessorManager] loc=manager.py:2672025-12-05T08:45:12.168422Z [info ] DAGbundlesloaded: test-git-bundle, local-bundle [airflow.dag_processing.bundles.manager.DagBundlesManager] loc=manager.py:2092025-12-05T08:45:12.174930Z [error ] CouldnotcreateGitHook [airflow.providers.git.bundles.git] bare_repo_path=PosixPath('/tmp/airflow/dag_bundles/test-git-bundle/bare') bundle_name=test-git-bundleconn_id=non_existent_connectionexc=AirflowNotFoundException("The conn_id `non_existent_connection` isn't defined") git_conn_id=non_existent_connectionloc=git.py:94repo_path=PosixPath('/tmp/airflow/dag_bundles/test-git-bundle/tracking_repo') version=Noneversions_path=PosixPath('/tmp/airflow/dag_bundles/test-git-bundle/versions')
Traceback (mostrecentcalllast):
File"/opt/airflow/providers/git/src/airflow/providers/git/bundles/git.py", line92, in__init__self.hook=GitHook(git_conn_id=git_conn_idor"git_default", repo_url=self.repo_url)
File"/opt/airflow/providers/git/src/airflow/providers/git/hooks/git.py", line69, in__init__connection=self.get_connection(git_conn_id)
File"/opt/airflow/task-sdk/src/airflow/sdk/bases/hook.py", line61, inget_connectionconn=Connection.get(conn_id)
File"/opt/airflow/task-sdk/src/airflow/sdk/definitions/connection.py", line224, ingetreturn_get_connection(conn_id)
File"/opt/airflow/task-sdk/src/airflow/sdk/execution_time/context.py", line176, in_get_connectionraiseAirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.sdk.exceptions.AirflowNotFoundException: Theconn_id`non_existent_connection`isn'tdefined2025-12-05T08:45:12.175495Z [error ] Errorcreatingbundle'test-git-bundle': Theconn_id`non_existent_connection`isn'tdefined [airflow.dag_processing.bundles.manager.DagBundlesManager] loc=manager.py:249Traceback (mostrecentcalllast):
File"/opt/airflow/airflow-core/src/airflow/dag_processing/bundles/manager.py", line247, insync_bundles_to_dbnew_template, new_params=_extract_and_sign_template(name)
File"/opt/airflow/airflow-core/src/airflow/dag_processing/bundles/manager.py", line216, in_extract_and_sign_templatebundle_instance=self.get_bundle(name)
File"/opt/airflow/airflow-core/src/airflow/dag_processing/bundles/manager.py", line337, inget_bundlereturncfg_bundle.bundle_class(name=name, version=version, **cfg_bundle.kwargs)
File"/opt/airflow/providers/git/src/airflow/providers/git/bundles/git.py", line92, in__init__self.hook=GitHook(git_conn_id=git_conn_idor"git_default", repo_url=self.repo_url)
File"/opt/airflow/providers/git/src/airflow/providers/git/hooks/git.py", line69, in__init__connection=self.get_connection(git_conn_id)
File"/opt/airflow/task-sdk/src/airflow/sdk/bases/hook.py", line61, inget_connectionconn=Connection.get(conn_id)
File"/opt/airflow/task-sdk/src/airflow/sdk/definitions/connection.py", line224, ingetreturn_get_connection(conn_id)
File"/opt/airflow/task-sdk/src/airflow/sdk/execution_time/context.py", line176, in_get_connectionraiseAirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
airflow.sdk.exceptions.AirflowNotFoundException: Theconn_id`non_existent_connection`isn'tdefined2025-12-05T08:45:12.175815Z [warning ] Removingownershipofteam'None'fromDagbundle'local-bundle' [airflow.dag_processing.bundles.manager.DagBundlesManager] loc=manager.py:28

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Comment threadproviders/git/src/airflow/providers/git/bundles/git.py
@amoghrajesh
amoghrajeshforce-pushed the dagmanager-error-handling branch from ac325a8 to 14559b6CompareDecember 8, 2025 07:42
@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Sorry for the spam folks, possibly due to a bad rebase I had!

@amoghrajesh
amoghrajesh merged commit 85b65eb into apache:mainDec 8, 2025
121 checks passed
@amoghrajesh
amoghrajesh deleted the dagmanager-error-handling branch December 8, 2025 15:05
@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Ahhh missed the v3 label!

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Oh this one will need manual port over.

@amoghrajesh

Copy link
Copy Markdown
ContributorAuthor

Manual port to v3 branch here: #59236

henry3260 pushed a commit to henry3260/airflow that referenced this pull request Dec 10, 2025
@ephraimbuddyephraimbuddy removed this from the Airflow 3.1.5 milestone Jan 6, 2026
jhgoebbert pushed a commit to jhgoebbert/airflow_Owen-CH-Leung that referenced this pull request Feb 8, 2026
Subham-KRLX pushed a commit to Subham-KRLX/airflow that referenced this pull request Mar 4, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@amoghrajesh@pierrejeambrun@dstandish@ephraimbuddy