Add durable/deferrable filters to the Airflow Registry

[amoghrajesh]

---

Was generative AI tooling used to co-author this PR?

• Yes - claude sonnet 4.6

---

Important Note

What "durable" and "deferrable" mean in the PR: there are two different reasons an operator can survive a crash, and they work in totally different ways. "Durable" means the operator own code remembers where its job was and reconnects to it instead of starting over, and this operates using task state store in synchronous mode. "Deferrable" is running using a trigger, in asynchronous mode. The end result looks similar to a user (nothing gets redone), but one is a promise the operator has to keep and the other is just how the system already works for everyone.

What

Follow up to the merged supports_durable_execution badge PR (#67236). Now that we have badget support, we should also have a few filters for ease of navigation for two related-but-distinct capabilities: "durable" (anything that survives a crash without redoing work -- ResumableJobMixin/manual-marker operators, or deferrable ones) and "deferrable" (the narrower, deferrable-only subset). The registry had no deferrable detection at all before this change.

Current behaviour

The Registry only tracks supports_durable_execution (ResumableJobMixin or the manual __supports_durable_execution marker coming in PR: #70291). Deferrable operators -- a much more common capability across most cloud providers -- have no signal anywhere in the pipeline, and there's no way to filter a provider's module list by either capability.

Proposed change

• Detection: supports_deferrable(cls) added to dev/registry/extract_parameters.py -- checks for a deferrable constructor parameter, reusing the existing get_params_from_class() machinery already used for parameters.json (no new introspection needed).
• Schema: supports_deferrable: bool = False added to ModuleContract, wired into make_entry() alongside supports_durable_execution (14 fields total, kept as an independent signal, not merged into the existing one).
• Frontend: two independent badges on module cards ("Durable", "Deferrable": teal/blue) shown per their own signal, since an operator can have either, both, or neither. Two filter toggles on a provider's module list: "Durable (includes deferrable)" and "Deferrable only".

Changes of Note

• The two filters are a deliberate union/subset pair, not independent booleans. "Durable" is supports_durable_execution OR supports_deferrable -- checking it shows every deferrable operator plus every ResumableJobMixin/manual-marker operator, even ones that aren't deferrable. "Deferrable only" is supports_deferrable alone, never OR'd. This means an operator that is both durable and deferrable appears under both filters -- verified this is intentional and not confusing in practice, since the two independent badges on the card explain why.
• The checkbox labels ("Durable (includes deferrable)" / "Deferrable only") were chosen specifically to make that asymmetric containment relationship (deferrable ⊆ durable, never the reverse) visible at the point of clicking, rather than relying on a tooltip or a caption someone could miss.
• is_deferrable inheriting a parent's __init__ unmodified still counts as deferrable (unlike the supports_durable_execution manual-marker case, which deliberately does not propagate to subclasses) -- inheriting an unmodified constructor is a real functional guarantee the parameter still works, whereas inheriting a durable-execution claim says nothing about whether an overridden execute() still preserves it.

Testing

• Built breeze to extract the provider data using: breeze registry extract-data

• Built the actual Eleventy site against this real data and confirmed in the rendered HTML: both badges appear on the right operator cards (e.g. Databricks' page shows both a "Durable" and "Deferrable" badge on DatabricksRunNowOperator), both filter checkboxes render with the intended labels, and the data-durable/data-deferrable attributes are correctly populated per the OR/non-OR logic on every module checked.

For current changes, this is the behaviour:

• Databricks: DatabricksSubmitRunOperator and DatabricksRunNowOperator are both durable and deferrable. DatabricksNotebookOperator, DatabricksSQLStatementsOperator, DatabricksTaskBaseOperator, DatabricksTaskOperator, and DatabricksSQLStatementsSensor are deferrable only. The rest of the provider's operators/hooks/sensors are neither.
• Apache Spark: SparkSubmitOperator is durable but not deferrable (no deferrable param) -- this is the single durable-only case across the entire registry.
• Google/BigQuery: BigQueryInsertJobOperator is both durable and deferrable, same pattern as Databricks.
• Amazon/Redshift: RedshiftDataOperator is both durable and deferrable.
• Snowflake: SnowflakeSqlApiOperator is both durable and deferrable.

Across the full dataset: 1779 modules total, 6 durable, 217 deferrable, 5 with both (the five listed above).

Some screenshots:

In this image, "Durable (includes deferrable)" is selected, and you can see both durable + deferrable (BigQueryInsertJobOperator) and just deferrable ones selected

"Deferrable only selection":

Operator like snowflake has only one operator with both deferrable and durable property, so its here:

Spark is one of few operators with only "durable", and no deferrable option:

What's next

Interactive click-through verification (actually toggling the checkboxes in a running browser to confirm filterModules() behaves correctly) hasn't been done -- only the generated HTML/data have been checked, not runtime JS behavior. Registry-wide (cross-provider) discovery -- "which are all the durable/deferrable operators across the whole registry," via a Pagefind search filter facet -- was discussed and deliberately deferred to a separate follow-up.

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[kaxil]

The isLatest gate closes the older-version path, but there's a second route to the same dead toggle it doesn't cover, and this one hits the pages people actually land on.

The live catalogue is rebuilt incrementally. publish-docs-to-s3.yml:655 passes specific provider ids into registry-build.yml, which pulls the published modules.json down from S3, and merge_registry_data.py:83-84 keeps every non-targeted provider's module dicts verbatim. _validate (registry_contract_models.py:233-235) is model_validate(payload); return payload, so it never writes the Pydantic defaults back into the dict. So after the next provider release, every provider that wasn't in that build still has pv.isLatest true and no supports_deferrable key at all, and ticking either box blanks its grid.

It isn't only a stale-data problem either. From the current tree, 17 of 105 provider packages contain a class declaring a deferrable parameter, and the durable marker appears in exactly 2 (pod.py:288, agent.py:226), so 87 packages have neither. Amazon and Google alone account for most of the deferrable classes. Both toggles are dead controls on the large majority of provider pages even after a full rebuild.

Counting in the template and rendering each box only when its count is non-zero fixes the stale-data case and the genuinely-empty case together, and self-heals when a rebuild lands. The accumulate-in-a-loop pattern already works in this file (totalModules at :38-39), and the type tabs 28 lines down already gate themselves the same way ({% if moduleCounts[t.id] > 0 %} at :358). Putting the count in the label saves a click too. Worth pairing with the empty state from provider-filters.js:81-94, which already hides the grid and writes a #filter-status count for the providers list; the module grid has no equivalent, and the "no modules" banner at :454-465 is keyed off totalModules so client-side filtering never reaches it.

[amoghrajesh]

Good catch, the isLatest gate only covered one of the two failure modes you found. Switched to counting qualifying modules directly in the template (mirroring the existing totalModules accumulator and the moduleCounts[t.id] > 0 gate the type tabs already use) and rendering each toggle only when its count is non-zero. This covers the stale data case (a field that's missing or false everywhere computes to a zero count) and the genuinely empty case with the same check, and self-heals after every rebuild without needing to special-case either. Put the count in each label too, per your suggestion.

Verified against real data: Databricks shows both toggles at (7)/(7); Apache Spark shows only the Durable toggle at (1) with no Deferrable toggle in the DOM at all (it has zero deferrable operators); a provider with neither capability renders no toggle block whatsoever. Also confirmed the toggles still compose correctly with the existing type/category/search filters. (see image for DBX)

One piece from your comment I have not done yet: the empty state message for when a filter combination client side narrows to zero results (right now it just goes blank, same as before). Happy to add that here too if you'd rather not split it into a separate follow-up.

[amoghrajesh]

Handled in handling simple review comments

[kaxil]

"Reconnects on retry (ResumableJobMixin)" names a mechanism neither durable-marked operator actually uses. KubernetesPodOperator is (BaseOperator) (pod.py:142) and AgentOperator is (BaseOperator, HITLReviewMixin) (agent.py:120); both carry the manual marker with a comment right above it saying "supports durable execution directly, without ResumableJobMixin" (pod.py:286-288, agent.py:224-226). is_durable_capable has the two paths precisely so those operators qualify without the mixin, so the mixin is the one thing the currently-badged modules have in common only by not using it. The shipped badge copy at :418 describes the behaviour without naming an SDK class, which reads better for a site visitor anyway.

While you're in here, the Deferrable badge tooltip on :421 says "trigger state is persisted and reassigned automatically", which is a bit stronger than what happens: the row persists the classpath and the serialized kwargs, not progress, so a reassigned trigger re-runs run() from the start rather than resuming. deferring.rst:170 tells trigger authors to assume exactly that ("a trigger instance can run more than once").

[amoghrajesh]

Good catch on both. Fixed the "Durable" filter checkbox's tooltip to drop the ResumableJobMixin mention —

Also reworded the "Deferrable" badge tooltip.

[amoghrajesh]

Handled in handling simple review comments

[kaxil]

get_params_from_class unions every ancestor's __init__ params, so this answers "does some ancestor name deferrable" rather than "can this defer". Because the union ignores who actually reads the parameter, it currently publishes at least one capability claim that isn't true.

DiscordWebhookOperator is class DiscordWebhookOperator(HttpOperator) and is registered in the catalogue (discord/provider.yaml:74-77). Its own __init__ declares no deferrable and forwards **kwargs up (discord_webhook.py:57-70), so the union picks the param up from HttpOperator.__init__ (http/operators/http.py:121). But the only thing that reads it is HttpOperator.execute's if self.deferrable: self.execute_async(...) (http.py:164-168), and DiscordWebhookOperator.execute overrides that outright with a synchronous self.hook.execute() (discord_webhook.py:95-102). So it gets the Deferrable badge, matches both new checkboxes, and ships supports_deferrable: true in /api/modules.json, while passing deferrable=True frees no worker slot at all. An inherited knob whose only reader has been overridden is dead.

It misses in the other direction too, which is what makes this a limit of the proxy rather than a patchable edge case. An operator that defers unconditionally has no flag to expose, so there is no param to find. DateTimeSensorAsync is the clearest: its docstring is "Deferring itself to avoid taking up a worker slot", its execute body is a bare self.defer(...), and deferrable appears nowhere in the file (sensors/date_time.py:83-143, registered at standard/provider.yaml:102). I ran the predicate against DateTimeSensorAsync, MSGraphAsyncOperator, AwaitMessageSensor, SSHRemoteJobOperator and HITLOperator: all five return False while each one's own execute or poke calls self.defer(. So "Deferrable only" hides exactly the always-deferrable operators, and "Requires deferrable=True" on the :421 tooltip is unfollowable for them since there is no such param to set. A sweep suggests a handful of other always-deferring classes worth a spot check, though I only confirmed these five end to end.

is_durable_capable right above already takes the opposite position on inheritance, and says why: the marker is deliberately name-mangled so it is not inherited, because a subclass that overrides execute() "may not preserve the parent's task_state_store reconnect behavior, so the declaration must not be inherited". That reasoning applies verbatim here, and this predicate is the neighbour that ignores it.

I don't have a clean bound to propose. I tried the three that look obvious and each has a named counter-example in the tree, so this is probably worth deciding deliberately rather than patching:

• Grep the class's own execute for self.defer(, the way is_durable_capable greps for execute_resumable: false-negatives TriggerDagRunOperator. Its execute is trigger_dagrun.py:233-302 and delegates, so the if self.deferrable: self.defer( sits in _trigger_dag_af_2 at :396-397, one call away.
• Grep the whole class body instead: false-negatives TimeDeltaSensorAsync (sensors/time_delta.py:154-167, registered at provider.yaml:103), which inherits the param and the deferring execute from TimeDeltaSensor, so its own body contains neither self.defer( nor self.deferrable.
• Exclude classes that override execute at all: drops HttpOperator itself, which defines its own.

A per-class opt-in marker mirroring __supports_durable_execution would be consistent with the durable path and immune to all three, at the cost of provider-side annotation. Whichever way you go, the Discord shape is worth a regression test: forwards **kwargs to a deferrable parent, overrides execute synchronously, must classify as non-deferrable.

[amoghrajesh]

Very good catch. I am working a solution in my head very similar to is_durable_capable

[amoghrajesh]

Replaced the whole approach now. Instead of checking whether a deferrable parameter exists anywhere in the class's ancestry, supports_deferrable() now resolves execute via getattr (the same MRO-following resolution is_durable_capable already uses) and checks that method's actual source for self.deferrable or self.defer. That directly answers "does the code that runs for this class use deferral" instead of proxying through whether a setting merely exists somewhere in the hierarchy.

Verified against every class you named, and a couple more from the same family, using the real code (not just fixtures): HttpOperator -> True, DiscordWebhookOperator -> False (the false positive is gone), DateTimeSensorAsync/TimeDeltaSensorAsync/AwaitMessageSensor/HITLOperator/MSGraphAsyncOperator/SSHRemoteJobOperator -> all True. TimeDeltaSensorAsync in particular is now handled correctly because it resolves through the real MRO rather than only checking each class's own body and that's also why it doesn't fall into the same trap as the "grep whole class body" option you ruled out.

TriggerDagRunOperator still can't be found by source grep alone, since its self.defer call is one hop away inside _trigger_dag_af_2. Added it to a small, explicitly commented _DEFERRABLE_EXCEPTIONS set rather than trying to chase call graphs. Went with this over the full per-class opt-in marker you mentioned, since the source grep now has zero known false positives and covers everything except this one exception.

Added the Discord shape regression test you asked for, plus five more covering each shape above. Ran it against every provider in the tree (1954 modules, zero errors) as a sanity check at scale.

[amoghrajesh]

Handled in 03b18d4

[kaxil]

This is a real improvement, the light-mode durable badge was effectively unreadable at 1.62:1 before. It just doesn't quite clear the line: compositing the 15% teal-500 wash onto the card's own --color-card-bg, which resolves to #ffffff in light mode (tokens.css:250), teal-600 comes out at 3.26:1, against the 4.5:1 that 12px --font-medium text needs. teal-700 (#0f766e) gets it to 4.77:1 on the same composite. For what it's worth my last comment under-called the blue one too: blue-600 is 4.35:1, just under the line, where blue-700 (#1d4ed8) would be 5.64:1. Dark mode is comfortable either way, 7.9:1 and 6.1:1.

[amoghrajesh]

Sure, my numbers were against a plain white background, not the actual composited card background, so they were off I guess.

Bumped both to -700: --color-teal-700 & --color-blue-700 (#1d4ed8, ~5.64:1), added to tokens.css and swapped into the light-dark() calls for both badges.

[amoghrajesh]

Handled in handling simple review comments

[kaxil]

This one looks like it came back in with the main merge rather than on purpose. #70190 narrowed the selector to .module-tab[data-type], which is still there on line 33, so the branch's older copy landing above it leaves two var moduleTabs in the same scope. Nothing breaks today since the second assignment wins, but the dead line is the exact selector the comment on 31-32 exists to warn against: the natural instinct on seeing two near-identical declarations is to delete the second one, and then the More button joins moduleTabs and clicking it resets currentType to 'all'.

[amoghrajesh]

My bad, fixed it

[amoghrajesh]

Handled in handling simple review comments