UI: Add cross-Dag Time Schedule view - #71558

Open
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule
Open

UI: Add cross-Dag Time Schedule view#71558
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule

Conversation

@minyeamer

Copy link
Copy Markdown
Contributor

Add a Time Schedule tab for comparing Dag start times and durations across an Airflow environment.

Features:

  • Day and Week timelines with overlap handling and fixed headers
  • Mean, Max, and Min aggregation within zoom-dependent time buckets
  • Pointer-anchored zoom using buttons, the mouse wheel, or arrow keys
  • Existing Dag run, tag, timetable type, and optional team filters
  • Configurable limits from the latest 200 runs through all matching runs
  • Contiguous API pagination using the configured page size
  • Gray expected-run bars for scheduled Dags without matching Dag runs
  • Readable tooltips and links to the relevant Dag or Dag run
  • Saved view, aggregation, run-limit, and scheduled-only preferences
  • Light and dark theme support

This is a UI-only change. It reuses existing endpoints and adds no schema or backend API changes.

Earlier work in #68532 and #68547 informed the time-bucket design. The view placement follows #45497, while bounded loading follows the work in #47900, #60241, and #65388.

Related: #22001


Screenshots:

  • Day view:

time_schedule-day-tooltip

  • Week view:
time_schedule-week-tooltip

Videos:

  • Zoom (Day view):
time_schedule-day-zoom_in_out.mp4
  • Zoom (Week view):
time_schedule-week-zoom_in_out.mp4
  • Dag run links (Day view):
time_schedule-day-bar_link.mp4
  • Dag run limits (Week view):
time_schedule-week-display_n_dags.mp4
  • Aggregation (Day view):
time_schedule-day-aggregation.mp4

Was generative AI tooling used to co-author this PR?
  • Yes (GPT-5.6 Sol)

Generated-by: Codex (GPT-5.6 Sol) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Features:
- Add Day and Week timelines for Dag run timing patterns
- Group runs by Dag, state, time bucket, and weekday
- Support Mean, Max, and Min duration aggregation
- Show expected runs for scheduled Dags without matching runs
- Add Dag run, tag, timetable type, and team filters
Interactions:
- Add pointer-anchored timeline zoom controls
- Link timeline bars to their Dag or Dag run
- Add readable tooltips for run details
- Persist view, aggregation, limit, and filter preferences
- Support overlapping runs in Day and Week layouts
Data loading:
- Add selectable limits from 200 runs through all runs
- Paginate using the API-configured page size
- Prevent gaps between paginated Dag run results
- Limit Dag detail requests to expected-run markers
Documentation and tests:
- Document Time Schedule behavior in the UI guide
- Add light and dark screenshots
- Cover layouts, aggregation, filters, zoom, and pagination
@bbovenzi

Copy link
Copy Markdown
Contributor

Very cool! I could see a world where we could even show arrows between runs that are connected via partition keys. Or highlighting when Max active dag runs is being hit.

Love the minute detailing. That's great for dags that run every minute! Just needs a label.

  1. Let's move this as a tab inside the Dashboard home page instead of in the tabled dags/runs/tis view.
  2. Let's add the state icons to the bars so that they can be distinguished by something other than color
  3. We need labels and/or tooltips to help explain some of these filters:
    a) Latest 200 should probably be "Limit" to be consistent with other views
    b) Mean/Max/Min means nothing to me without context
  4. Day/Week modes are different types of views not a simple setting. It should probably match the tabs we have to change the calendar view on the dag details page.
  1. The filters are a bit inconsistent. We have the Filter Pill UX that is used elsewhere but separately we also have timetable filters, scheduled dags only filter, and tag filters. Let's consolidate them all together. I also recommend rebasing since @ryanahamilton's recent changes add some great examples.

@bbovenzibbovenzi added this to the Airflow 3.4.0 milestone Aug 21, 2026

@bbovenzibbovenzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a cool demo. But it definitely needs some work to "productionize" the code to be usable for deployments with any degree of scale.

Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/docs/ui.rst Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Capture changes during partial author review:
Data access and correctness:
- Avoid browser request fan-out and client-side filtering as Dag and run counts grow.
- Bound each view to 5,000 matching runs rather than allowing unbounded reads.
- Calculate only the selected Day or Week aggregation with access controls and filters applied.
Interaction quality:
- Stream timeline batches so useful data appears before the complete result is ready.
- Preserve displayed bars during zoom-driven refreshes while the final zoom level settles.
- Keep planned Dag markers visible even when no Dag runs match the current view.
Contract and verification:
- Keep the private UI OpenAPI contract and generated client aligned with the endpoint.
- Cover streaming, limits, filtering, aggregation, planned runs, and zoom behavior.
- Avoid MySQL's unsupported LIMIT subquery in an IN predicate.
- Mark NDJSON reads as sequential and fix TimeSchedule type errors.
- Confirm earlier UI import failures no longer affect asset builds.
Keep Time Schedule compatible with the shared FilterBar refactor merged from main.
- Deleted Dag list filter imports caused Vite module-resolution failures.
- The failure blocked React UI tests and CI jobs that build UI assets.
- Reuse the shared filter state for Tag and Timetable Type filtering.
@minyeamer

Copy link
Copy Markdown
ContributorAuthor

@bbovenzi
Thanks for the detailed review. I interpreted the four main concerns as scalability and
data-flow issues rather than only UI adjustments.

Since b13f993, excluding merge commits, I made the following changes:

  • Replaced client-side pagination and request fan-out with a dedicated streaming endpoint.
  • Removed the unbounded "All Dag runs" option and enforced a maximum of 5,000 runs on the backend.
  • Moved Dag and Dag run filtering into the backend query construction.
  • Aggregate only the selected Day or Week view on the server.
  • Stream results in batches so the UI can render progressively.
  • Updated the filter implementation after the shared FilterBar changes were merged.
  • Kept the private OpenAPI contract and generated client aligned with the backend route.

The current flow is:

flowchart TD
UI[Time Schedule UI] --> Hook[useTimeScheduleData]
Hook -->|One GET request| API[GET /ui/time-schedule]
API --> Filter[Apply authorization and filters in SQL]
Filter --> Limit[Select up to 5,000 Dag runs]
Limit --> Batch[Process 25 Dags per batch]
Batch --> Aggregate[Aggregate selected Day or Week view]
Aggregate --> Stream[Return NDJSON TimeScheduleBatch]
Stream --> Hook
Hook --> Render[Progressively render timeline bars]
Loading

The browser now receives only the filtered data needed for the selected view, while the backend controls the result size and aggregation cost.

I initially intended this change to remain UI-only and did not plan to add a backend API. Because the review identified the scalability limitations of the original approach, I needed to first understand Airflow's backend API structure, streaming session lifecycle, authorization filters, and OpenAPI generation flow. I also spent additional time reviewing the AI-assisted code carefully, which is why the follow-up changes took about 5 days.

The implementation is now focused on the production-scale concerns from the review while preserving the existing Airflow UI/API patterns.

@minyeamer

Copy link
Copy Markdown
ContributorAuthor

I missed these additional suggestions in my previous follow-up. I plan to address the following:

  • Move Time Schedule into the Dashboard as a tab.
  • Add state icons to timeline bars so states are not distinguished by color alone.
  • Improve labels and tooltips:
    • Rename Latest 200 to Limit 200.
    • Add context for the duration aggregation options.
  • Replace the Day/Week selector with tabs.
  • Consolidate the remaining filters through the shared Filter Pill UX.

For the partition-key arrows, I understand this may include dependencies between Dag runs, such as those created through TriggerDagRun. However, with the timeline sorted alphabetically by dag_id, the connections could easily cross and become difficult to follow.

I’m happy to implement this with more specific guidance on which relationships to show and how they should be laid out. With the current context, I’m hesitant to add a potentially confusing visualization.


Drafted-by: GPT-5

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:translationsarea:UIRelated to UI/UX. For Frontend Developers.kind:documentationtranslation:default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@minyeamer@bbovenzi
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

UI: Add cross-Dag Time Schedule view - #71558

Open
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule
Open

UI: Add cross-Dag Time Schedule view#71558
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule

Conversation

@minyeamer

Copy link
Copy Markdown
Contributor

Add a Time Schedule tab for comparing Dag start times and durations across an Airflow environment.

Features:

  • Day and Week timelines with overlap handling and fixed headers
  • Mean, Max, and Min aggregation within zoom-dependent time buckets
  • Pointer-anchored zoom using buttons, the mouse wheel, or arrow keys
  • Existing Dag run, tag, timetable type, and optional team filters
  • Configurable limits from the latest 200 runs through all matching runs
  • Contiguous API pagination using the configured page size
  • Gray expected-run bars for scheduled Dags without matching Dag runs
  • Readable tooltips and links to the relevant Dag or Dag run
  • Saved view, aggregation, run-limit, and scheduled-only preferences
  • Light and dark theme support

This is a UI-only change. It reuses existing endpoints and adds no schema or backend API changes.

Earlier work in #68532 and #68547 informed the time-bucket design. The view placement follows #45497, while bounded loading follows the work in #47900, #60241, and #65388.

Related: #22001


Screenshots:

  • Day view:

time_schedule-day-tooltip

  • Week view:
time_schedule-week-tooltip

Videos:

  • Zoom (Day view):
time_schedule-day-zoom_in_out.mp4
  • Zoom (Week view):
time_schedule-week-zoom_in_out.mp4
  • Dag run links (Day view):
time_schedule-day-bar_link.mp4
  • Dag run limits (Week view):
time_schedule-week-display_n_dags.mp4
  • Aggregation (Day view):
time_schedule-day-aggregation.mp4

Was generative AI tooling used to co-author this PR?
  • Yes (GPT-5.6 Sol)

Generated-by: Codex (GPT-5.6 Sol) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Features:
- Add Day and Week timelines for Dag run timing patterns
- Group runs by Dag, state, time bucket, and weekday
- Support Mean, Max, and Min duration aggregation
- Show expected runs for scheduled Dags without matching runs
- Add Dag run, tag, timetable type, and team filters
Interactions:
- Add pointer-anchored timeline zoom controls
- Link timeline bars to their Dag or Dag run
- Add readable tooltips for run details
- Persist view, aggregation, limit, and filter preferences
- Support overlapping runs in Day and Week layouts
Data loading:
- Add selectable limits from 200 runs through all runs
- Paginate using the API-configured page size
- Prevent gaps between paginated Dag run results
- Limit Dag detail requests to expected-run markers
Documentation and tests:
- Document Time Schedule behavior in the UI guide
- Add light and dark screenshots
- Cover layouts, aggregation, filters, zoom, and pagination
@bbovenzi

Copy link
Copy Markdown
Contributor

Very cool! I could see a world where we could even show arrows between runs that are connected via partition keys. Or highlighting when Max active dag runs is being hit.

Love the minute detailing. That's great for dags that run every minute! Just needs a label.

  1. Let's move this as a tab inside the Dashboard home page instead of in the tabled dags/runs/tis view.
  2. Let's add the state icons to the bars so that they can be distinguished by something other than color
  3. We need labels and/or tooltips to help explain some of these filters:
    a) Latest 200 should probably be "Limit" to be consistent with other views
    b) Mean/Max/Min means nothing to me without context
  4. Day/Week modes are different types of views not a simple setting. It should probably match the tabs we have to change the calendar view on the dag details page.
  1. The filters are a bit inconsistent. We have the Filter Pill UX that is used elsewhere but separately we also have timetable filters, scheduled dags only filter, and tag filters. Let's consolidate them all together. I also recommend rebasing since @ryanahamilton's recent changes add some great examples.

@bbovenzibbovenzi added this to the Airflow 3.4.0 milestone Aug 21, 2026

@bbovenzibbovenzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a cool demo. But it definitely needs some work to "productionize" the code to be usable for deployments with any degree of scale.

Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/docs/ui.rst Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Capture changes during partial author review:
Data access and correctness:
- Avoid browser request fan-out and client-side filtering as Dag and run counts grow.
- Bound each view to 5,000 matching runs rather than allowing unbounded reads.
- Calculate only the selected Day or Week aggregation with access controls and filters applied.
Interaction quality:
- Stream timeline batches so useful data appears before the complete result is ready.
- Preserve displayed bars during zoom-driven refreshes while the final zoom level settles.
- Keep planned Dag markers visible even when no Dag runs match the current view.
Contract and verification:
- Keep the private UI OpenAPI contract and generated client aligned with the endpoint.
- Cover streaming, limits, filtering, aggregation, planned runs, and zoom behavior.
- Avoid MySQL's unsupported LIMIT subquery in an IN predicate.
- Mark NDJSON reads as sequential and fix TimeSchedule type errors.
- Confirm earlier UI import failures no longer affect asset builds.
Keep Time Schedule compatible with the shared FilterBar refactor merged from main.
- Deleted Dag list filter imports caused Vite module-resolution failures.
- The failure blocked React UI tests and CI jobs that build UI assets.
- Reuse the shared filter state for Tag and Timetable Type filtering.
@minyeamer

Copy link
Copy Markdown
ContributorAuthor

@bbovenzi
Thanks for the detailed review. I interpreted the four main concerns as scalability and
data-flow issues rather than only UI adjustments.

Since b13f993, excluding merge commits, I made the following changes:

  • Replaced client-side pagination and request fan-out with a dedicated streaming endpoint.
  • Removed the unbounded "All Dag runs" option and enforced a maximum of 5,000 runs on the backend.
  • Moved Dag and Dag run filtering into the backend query construction.
  • Aggregate only the selected Day or Week view on the server.
  • Stream results in batches so the UI can render progressively.
  • Updated the filter implementation after the shared FilterBar changes were merged.
  • Kept the private OpenAPI contract and generated client aligned with the backend route.

The current flow is:

flowchart TD
UI[Time Schedule UI] --> Hook[useTimeScheduleData]
Hook -->|One GET request| API[GET /ui/time-schedule]
API --> Filter[Apply authorization and filters in SQL]
Filter --> Limit[Select up to 5,000 Dag runs]
Limit --> Batch[Process 25 Dags per batch]
Batch --> Aggregate[Aggregate selected Day or Week view]
Aggregate --> Stream[Return NDJSON TimeScheduleBatch]
Stream --> Hook
Hook --> Render[Progressively render timeline bars]
Loading

The browser now receives only the filtered data needed for the selected view, while the backend controls the result size and aggregation cost.

I initially intended this change to remain UI-only and did not plan to add a backend API. Because the review identified the scalability limitations of the original approach, I needed to first understand Airflow's backend API structure, streaming session lifecycle, authorization filters, and OpenAPI generation flow. I also spent additional time reviewing the AI-assisted code carefully, which is why the follow-up changes took about 5 days.

The implementation is now focused on the production-scale concerns from the review while preserving the existing Airflow UI/API patterns.

@minyeamer

Copy link
Copy Markdown
ContributorAuthor

I missed these additional suggestions in my previous follow-up. I plan to address the following:

  • Move Time Schedule into the Dashboard as a tab.
  • Add state icons to timeline bars so states are not distinguished by color alone.
  • Improve labels and tooltips:
    • Rename Latest 200 to Limit 200.
    • Add context for the duration aggregation options.
  • Replace the Day/Week selector with tabs.
  • Consolidate the remaining filters through the shared Filter Pill UX.

For the partition-key arrows, I understand this may include dependencies between Dag runs, such as those created through TriggerDagRun. However, with the timeline sorted alphabetically by dag_id, the connections could easily cross and become difficult to follow.

I’m happy to implement this with more specific guidance on which relationships to show and how they should be laid out. With the current context, I’m hesitant to add a potentially confusing visualization.


Drafted-by: GPT-5

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:translationsarea:UIRelated to UI/UX. For Frontend Developers.kind:documentationtranslation:default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@minyeamer@bbovenzi
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

UI: Add cross-Dag Time Schedule view - #71558

Open
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule
Open

UI: Add cross-Dag Time Schedule view#71558
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule

Conversation

@minyeamer

Copy link
Copy Markdown
Contributor

Add a Time Schedule tab for comparing Dag start times and durations across an Airflow environment.

Features:

  • Day and Week timelines with overlap handling and fixed headers
  • Mean, Max, and Min aggregation within zoom-dependent time buckets
  • Pointer-anchored zoom using buttons, the mouse wheel, or arrow keys
  • Existing Dag run, tag, timetable type, and optional team filters
  • Configurable limits from the latest 200 runs through all matching runs
  • Contiguous API pagination using the configured page size
  • Gray expected-run bars for scheduled Dags without matching Dag runs
  • Readable tooltips and links to the relevant Dag or Dag run
  • Saved view, aggregation, run-limit, and scheduled-only preferences
  • Light and dark theme support

This is a UI-only change. It reuses existing endpoints and adds no schema or backend API changes.

Earlier work in #68532 and #68547 informed the time-bucket design. The view placement follows #45497, while bounded loading follows the work in #47900, #60241, and #65388.

Related: #22001


Screenshots:

  • Day view:

time_schedule-day-tooltip

  • Week view:
time_schedule-week-tooltip

Videos:

  • Zoom (Day view):
time_schedule-day-zoom_in_out.mp4
  • Zoom (Week view):
time_schedule-week-zoom_in_out.mp4
  • Dag run links (Day view):
time_schedule-day-bar_link.mp4
  • Dag run limits (Week view):
time_schedule-week-display_n_dags.mp4
  • Aggregation (Day view):
time_schedule-day-aggregation.mp4

Was generative AI tooling used to co-author this PR?
  • Yes (GPT-5.6 Sol)

Generated-by: Codex (GPT-5.6 Sol) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Features:
- Add Day and Week timelines for Dag run timing patterns
- Group runs by Dag, state, time bucket, and weekday
- Support Mean, Max, and Min duration aggregation
- Show expected runs for scheduled Dags without matching runs
- Add Dag run, tag, timetable type, and team filters
Interactions:
- Add pointer-anchored timeline zoom controls
- Link timeline bars to their Dag or Dag run
- Add readable tooltips for run details
- Persist view, aggregation, limit, and filter preferences
- Support overlapping runs in Day and Week layouts
Data loading:
- Add selectable limits from 200 runs through all runs
- Paginate using the API-configured page size
- Prevent gaps between paginated Dag run results
- Limit Dag detail requests to expected-run markers
Documentation and tests:
- Document Time Schedule behavior in the UI guide
- Add light and dark screenshots
- Cover layouts, aggregation, filters, zoom, and pagination
@bbovenzi

Copy link
Copy Markdown
Contributor

Very cool! I could see a world where we could even show arrows between runs that are connected via partition keys. Or highlighting when Max active dag runs is being hit.

Love the minute detailing. That's great for dags that run every minute! Just needs a label.

  1. Let's move this as a tab inside the Dashboard home page instead of in the tabled dags/runs/tis view.
  2. Let's add the state icons to the bars so that they can be distinguished by something other than color
  3. We need labels and/or tooltips to help explain some of these filters:
    a) Latest 200 should probably be "Limit" to be consistent with other views
    b) Mean/Max/Min means nothing to me without context
  4. Day/Week modes are different types of views not a simple setting. It should probably match the tabs we have to change the calendar view on the dag details page.
  1. The filters are a bit inconsistent. We have the Filter Pill UX that is used elsewhere but separately we also have timetable filters, scheduled dags only filter, and tag filters. Let's consolidate them all together. I also recommend rebasing since @ryanahamilton's recent changes add some great examples.

@bbovenzibbovenzi added this to the Airflow 3.4.0 milestone Aug 21, 2026

@bbovenzibbovenzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a cool demo. But it definitely needs some work to "productionize" the code to be usable for deployments with any degree of scale.

Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/docs/ui.rst Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Capture changes during partial author review:
Data access and correctness:
- Avoid browser request fan-out and client-side filtering as Dag and run counts grow.
- Bound each view to 5,000 matching runs rather than allowing unbounded reads.
- Calculate only the selected Day or Week aggregation with access controls and filters applied.
Interaction quality:
- Stream timeline batches so useful data appears before the complete result is ready.
- Preserve displayed bars during zoom-driven refreshes while the final zoom level settles.
- Keep planned Dag markers visible even when no Dag runs match the current view.
Contract and verification:
- Keep the private UI OpenAPI contract and generated client aligned with the endpoint.
- Cover streaming, limits, filtering, aggregation, planned runs, and zoom behavior.
- Avoid MySQL's unsupported LIMIT subquery in an IN predicate.
- Mark NDJSON reads as sequential and fix TimeSchedule type errors.
- Confirm earlier UI import failures no longer affect asset builds.
Keep Time Schedule compatible with the shared FilterBar refactor merged from main.
- Deleted Dag list filter imports caused Vite module-resolution failures.
- The failure blocked React UI tests and CI jobs that build UI assets.
- Reuse the shared filter state for Tag and Timetable Type filtering.
@minyeamer

Copy link
Copy Markdown
ContributorAuthor

@bbovenzi
Thanks for the detailed review. I interpreted the four main concerns as scalability and
data-flow issues rather than only UI adjustments.

Since b13f993, excluding merge commits, I made the following changes:

  • Replaced client-side pagination and request fan-out with a dedicated streaming endpoint.
  • Removed the unbounded "All Dag runs" option and enforced a maximum of 5,000 runs on the backend.
  • Moved Dag and Dag run filtering into the backend query construction.
  • Aggregate only the selected Day or Week view on the server.
  • Stream results in batches so the UI can render progressively.
  • Updated the filter implementation after the shared FilterBar changes were merged.
  • Kept the private OpenAPI contract and generated client aligned with the backend route.

The current flow is:

flowchart TD
UI[Time Schedule UI] --> Hook[useTimeScheduleData]
Hook -->|One GET request| API[GET /ui/time-schedule]
API --> Filter[Apply authorization and filters in SQL]
Filter --> Limit[Select up to 5,000 Dag runs]
Limit --> Batch[Process 25 Dags per batch]
Batch --> Aggregate[Aggregate selected Day or Week view]
Aggregate --> Stream[Return NDJSON TimeScheduleBatch]
Stream --> Hook
Hook --> Render[Progressively render timeline bars]
Loading

The browser now receives only the filtered data needed for the selected view, while the backend controls the result size and aggregation cost.

I initially intended this change to remain UI-only and did not plan to add a backend API. Because the review identified the scalability limitations of the original approach, I needed to first understand Airflow's backend API structure, streaming session lifecycle, authorization filters, and OpenAPI generation flow. I also spent additional time reviewing the AI-assisted code carefully, which is why the follow-up changes took about 5 days.

The implementation is now focused on the production-scale concerns from the review while preserving the existing Airflow UI/API patterns.

@minyeamer

Copy link
Copy Markdown
ContributorAuthor

I missed these additional suggestions in my previous follow-up. I plan to address the following:

  • Move Time Schedule into the Dashboard as a tab.
  • Add state icons to timeline bars so states are not distinguished by color alone.
  • Improve labels and tooltips:
    • Rename Latest 200 to Limit 200.
    • Add context for the duration aggregation options.
  • Replace the Day/Week selector with tabs.
  • Consolidate the remaining filters through the shared Filter Pill UX.

For the partition-key arrows, I understand this may include dependencies between Dag runs, such as those created through TriggerDagRun. However, with the timeline sorted alphabetically by dag_id, the connections could easily cross and become difficult to follow.

I’m happy to implement this with more specific guidance on which relationships to show and how they should be laid out. With the current context, I’m hesitant to add a potentially confusing visualization.


Drafted-by: GPT-5

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:translationsarea:UIRelated to UI/UX. For Frontend Developers.kind:documentationtranslation:default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@minyeamer@bbovenzi
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

UI: Add cross-Dag Time Schedule view - #71558

Open
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule
Open

UI: Add cross-Dag Time Schedule view#71558
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule

Conversation

@minyeamer

Copy link
Copy Markdown
Contributor

Add a Time Schedule tab for comparing Dag start times and durations across an Airflow environment.

Features:

  • Day and Week timelines with overlap handling and fixed headers
  • Mean, Max, and Min aggregation within zoom-dependent time buckets
  • Pointer-anchored zoom using buttons, the mouse wheel, or arrow keys
  • Existing Dag run, tag, timetable type, and optional team filters
  • Configurable limits from the latest 200 runs through all matching runs
  • Contiguous API pagination using the configured page size
  • Gray expected-run bars for scheduled Dags without matching Dag runs
  • Readable tooltips and links to the relevant Dag or Dag run
  • Saved view, aggregation, run-limit, and scheduled-only preferences
  • Light and dark theme support

This is a UI-only change. It reuses existing endpoints and adds no schema or backend API changes.

Earlier work in #68532 and #68547 informed the time-bucket design. The view placement follows #45497, while bounded loading follows the work in #47900, #60241, and #65388.

Related: #22001


Screenshots:

  • Day view:

time_schedule-day-tooltip

  • Week view:
time_schedule-week-tooltip

Videos:

  • Zoom (Day view):
time_schedule-day-zoom_in_out.mp4
  • Zoom (Week view):
time_schedule-week-zoom_in_out.mp4
  • Dag run links (Day view):
time_schedule-day-bar_link.mp4
  • Dag run limits (Week view):
time_schedule-week-display_n_dags.mp4
  • Aggregation (Day view):
time_schedule-day-aggregation.mp4

Was generative AI tooling used to co-author this PR?
  • Yes (GPT-5.6 Sol)

Generated-by: Codex (GPT-5.6 Sol) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Features:
- Add Day and Week timelines for Dag run timing patterns
- Group runs by Dag, state, time bucket, and weekday
- Support Mean, Max, and Min duration aggregation
- Show expected runs for scheduled Dags without matching runs
- Add Dag run, tag, timetable type, and team filters
Interactions:
- Add pointer-anchored timeline zoom controls
- Link timeline bars to their Dag or Dag run
- Add readable tooltips for run details
- Persist view, aggregation, limit, and filter preferences
- Support overlapping runs in Day and Week layouts
Data loading:
- Add selectable limits from 200 runs through all runs
- Paginate using the API-configured page size
- Prevent gaps between paginated Dag run results
- Limit Dag detail requests to expected-run markers
Documentation and tests:
- Document Time Schedule behavior in the UI guide
- Add light and dark screenshots
- Cover layouts, aggregation, filters, zoom, and pagination
@bbovenzi

Copy link
Copy Markdown
Contributor

Very cool! I could see a world where we could even show arrows between runs that are connected via partition keys. Or highlighting when Max active dag runs is being hit.

Love the minute detailing. That's great for dags that run every minute! Just needs a label.

  1. Let's move this as a tab inside the Dashboard home page instead of in the tabled dags/runs/tis view.
  2. Let's add the state icons to the bars so that they can be distinguished by something other than color
  3. We need labels and/or tooltips to help explain some of these filters:
    a) Latest 200 should probably be "Limit" to be consistent with other views
    b) Mean/Max/Min means nothing to me without context
  4. Day/Week modes are different types of views not a simple setting. It should probably match the tabs we have to change the calendar view on the dag details page.
  1. The filters are a bit inconsistent. We have the Filter Pill UX that is used elsewhere but separately we also have timetable filters, scheduled dags only filter, and tag filters. Let's consolidate them all together. I also recommend rebasing since @ryanahamilton's recent changes add some great examples.

@bbovenzibbovenzi added this to the Airflow 3.4.0 milestone Aug 21, 2026

@bbovenzibbovenzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a cool demo. But it definitely needs some work to "productionize" the code to be usable for deployments with any degree of scale.

Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/docs/ui.rst Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Capture changes during partial author review:
Data access and correctness:
- Avoid browser request fan-out and client-side filtering as Dag and run counts grow.
- Bound each view to 5,000 matching runs rather than allowing unbounded reads.
- Calculate only the selected Day or Week aggregation with access controls and filters applied.
Interaction quality:
- Stream timeline batches so useful data appears before the complete result is ready.
- Preserve displayed bars during zoom-driven refreshes while the final zoom level settles.
- Keep planned Dag markers visible even when no Dag runs match the current view.
Contract and verification:
- Keep the private UI OpenAPI contract and generated client aligned with the endpoint.
- Cover streaming, limits, filtering, aggregation, planned runs, and zoom behavior.
- Avoid MySQL's unsupported LIMIT subquery in an IN predicate.
- Mark NDJSON reads as sequential and fix TimeSchedule type errors.
- Confirm earlier UI import failures no longer affect asset builds.
Keep Time Schedule compatible with the shared FilterBar refactor merged from main.
- Deleted Dag list filter imports caused Vite module-resolution failures.
- The failure blocked React UI tests and CI jobs that build UI assets.
- Reuse the shared filter state for Tag and Timetable Type filtering.
@minyeamer

Copy link
Copy Markdown
ContributorAuthor

@bbovenzi
Thanks for the detailed review. I interpreted the four main concerns as scalability and
data-flow issues rather than only UI adjustments.

Since b13f993, excluding merge commits, I made the following changes:

  • Replaced client-side pagination and request fan-out with a dedicated streaming endpoint.
  • Removed the unbounded "All Dag runs" option and enforced a maximum of 5,000 runs on the backend.
  • Moved Dag and Dag run filtering into the backend query construction.
  • Aggregate only the selected Day or Week view on the server.
  • Stream results in batches so the UI can render progressively.
  • Updated the filter implementation after the shared FilterBar changes were merged.
  • Kept the private OpenAPI contract and generated client aligned with the backend route.

The current flow is:

flowchart TD
UI[Time Schedule UI] --> Hook[useTimeScheduleData]
Hook -->|One GET request| API[GET /ui/time-schedule]
API --> Filter[Apply authorization and filters in SQL]
Filter --> Limit[Select up to 5,000 Dag runs]
Limit --> Batch[Process 25 Dags per batch]
Batch --> Aggregate[Aggregate selected Day or Week view]
Aggregate --> Stream[Return NDJSON TimeScheduleBatch]
Stream --> Hook
Hook --> Render[Progressively render timeline bars]
Loading

The browser now receives only the filtered data needed for the selected view, while the backend controls the result size and aggregation cost.

I initially intended this change to remain UI-only and did not plan to add a backend API. Because the review identified the scalability limitations of the original approach, I needed to first understand Airflow's backend API structure, streaming session lifecycle, authorization filters, and OpenAPI generation flow. I also spent additional time reviewing the AI-assisted code carefully, which is why the follow-up changes took about 5 days.

The implementation is now focused on the production-scale concerns from the review while preserving the existing Airflow UI/API patterns.

@minyeamer

Copy link
Copy Markdown
ContributorAuthor

I missed these additional suggestions in my previous follow-up. I plan to address the following:

  • Move Time Schedule into the Dashboard as a tab.
  • Add state icons to timeline bars so states are not distinguished by color alone.
  • Improve labels and tooltips:
    • Rename Latest 200 to Limit 200.
    • Add context for the duration aggregation options.
  • Replace the Day/Week selector with tabs.
  • Consolidate the remaining filters through the shared Filter Pill UX.

For the partition-key arrows, I understand this may include dependencies between Dag runs, such as those created through TriggerDagRun. However, with the timeline sorted alphabetically by dag_id, the connections could easily cross and become difficult to follow.

I’m happy to implement this with more specific guidance on which relationships to show and how they should be laid out. With the current context, I’m hesitant to add a potentially confusing visualization.


Drafted-by: GPT-5

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:translationsarea:UIRelated to UI/UX. For Frontend Developers.kind:documentationtranslation:default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@minyeamer@bbovenzi
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

UI: Add cross-Dag Time Schedule view - #71558

Open
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule
Open

UI: Add cross-Dag Time Schedule view#71558
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule

Conversation

@minyeamer

Copy link
Copy Markdown
Contributor

Add a Time Schedule tab for comparing Dag start times and durations across an Airflow environment.

Features:

  • Day and Week timelines with overlap handling and fixed headers
  • Mean, Max, and Min aggregation within zoom-dependent time buckets
  • Pointer-anchored zoom using buttons, the mouse wheel, or arrow keys
  • Existing Dag run, tag, timetable type, and optional team filters
  • Configurable limits from the latest 200 runs through all matching runs
  • Contiguous API pagination using the configured page size
  • Gray expected-run bars for scheduled Dags without matching Dag runs
  • Readable tooltips and links to the relevant Dag or Dag run
  • Saved view, aggregation, run-limit, and scheduled-only preferences
  • Light and dark theme support

This is a UI-only change. It reuses existing endpoints and adds no schema or backend API changes.

Earlier work in #68532 and #68547 informed the time-bucket design. The view placement follows #45497, while bounded loading follows the work in #47900, #60241, and #65388.

Related: #22001


Screenshots:

  • Day view:

time_schedule-day-tooltip

  • Week view:
time_schedule-week-tooltip

Videos:

  • Zoom (Day view):
time_schedule-day-zoom_in_out.mp4
  • Zoom (Week view):
time_schedule-week-zoom_in_out.mp4
  • Dag run links (Day view):
time_schedule-day-bar_link.mp4
  • Dag run limits (Week view):
time_schedule-week-display_n_dags.mp4
  • Aggregation (Day view):
time_schedule-day-aggregation.mp4

Was generative AI tooling used to co-author this PR?
  • Yes (GPT-5.6 Sol)

Generated-by: Codex (GPT-5.6 Sol) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Features:
- Add Day and Week timelines for Dag run timing patterns
- Group runs by Dag, state, time bucket, and weekday
- Support Mean, Max, and Min duration aggregation
- Show expected runs for scheduled Dags without matching runs
- Add Dag run, tag, timetable type, and team filters
Interactions:
- Add pointer-anchored timeline zoom controls
- Link timeline bars to their Dag or Dag run
- Add readable tooltips for run details
- Persist view, aggregation, limit, and filter preferences
- Support overlapping runs in Day and Week layouts
Data loading:
- Add selectable limits from 200 runs through all runs
- Paginate using the API-configured page size
- Prevent gaps between paginated Dag run results
- Limit Dag detail requests to expected-run markers
Documentation and tests:
- Document Time Schedule behavior in the UI guide
- Add light and dark screenshots
- Cover layouts, aggregation, filters, zoom, and pagination
@bbovenzi

Copy link
Copy Markdown
Contributor

Very cool! I could see a world where we could even show arrows between runs that are connected via partition keys. Or highlighting when Max active dag runs is being hit.

Love the minute detailing. That's great for dags that run every minute! Just needs a label.

  1. Let's move this as a tab inside the Dashboard home page instead of in the tabled dags/runs/tis view.
  2. Let's add the state icons to the bars so that they can be distinguished by something other than color
  3. We need labels and/or tooltips to help explain some of these filters:
    a) Latest 200 should probably be "Limit" to be consistent with other views
    b) Mean/Max/Min means nothing to me without context
  4. Day/Week modes are different types of views not a simple setting. It should probably match the tabs we have to change the calendar view on the dag details page.
  1. The filters are a bit inconsistent. We have the Filter Pill UX that is used elsewhere but separately we also have timetable filters, scheduled dags only filter, and tag filters. Let's consolidate them all together. I also recommend rebasing since @ryanahamilton's recent changes add some great examples.

@bbovenzibbovenzi added this to the Airflow 3.4.0 milestone Aug 21, 2026

@bbovenzibbovenzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a cool demo. But it definitely needs some work to "productionize" the code to be usable for deployments with any degree of scale.

Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/docs/ui.rst Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Capture changes during partial author review:
Data access and correctness:
- Avoid browser request fan-out and client-side filtering as Dag and run counts grow.
- Bound each view to 5,000 matching runs rather than allowing unbounded reads.
- Calculate only the selected Day or Week aggregation with access controls and filters applied.
Interaction quality:
- Stream timeline batches so useful data appears before the complete result is ready.
- Preserve displayed bars during zoom-driven refreshes while the final zoom level settles.
- Keep planned Dag markers visible even when no Dag runs match the current view.
Contract and verification:
- Keep the private UI OpenAPI contract and generated client aligned with the endpoint.
- Cover streaming, limits, filtering, aggregation, planned runs, and zoom behavior.
- Avoid MySQL's unsupported LIMIT subquery in an IN predicate.
- Mark NDJSON reads as sequential and fix TimeSchedule type errors.
- Confirm earlier UI import failures no longer affect asset builds.
Keep Time Schedule compatible with the shared FilterBar refactor merged from main.
- Deleted Dag list filter imports caused Vite module-resolution failures.
- The failure blocked React UI tests and CI jobs that build UI assets.
- Reuse the shared filter state for Tag and Timetable Type filtering.
@minyeamer

Copy link
Copy Markdown
ContributorAuthor

@bbovenzi
Thanks for the detailed review. I interpreted the four main concerns as scalability and
data-flow issues rather than only UI adjustments.

Since b13f993, excluding merge commits, I made the following changes:

  • Replaced client-side pagination and request fan-out with a dedicated streaming endpoint.
  • Removed the unbounded "All Dag runs" option and enforced a maximum of 5,000 runs on the backend.
  • Moved Dag and Dag run filtering into the backend query construction.
  • Aggregate only the selected Day or Week view on the server.
  • Stream results in batches so the UI can render progressively.
  • Updated the filter implementation after the shared FilterBar changes were merged.
  • Kept the private OpenAPI contract and generated client aligned with the backend route.

The current flow is:

flowchart TD
UI[Time Schedule UI] --> Hook[useTimeScheduleData]
Hook -->|One GET request| API[GET /ui/time-schedule]
API --> Filter[Apply authorization and filters in SQL]
Filter --> Limit[Select up to 5,000 Dag runs]
Limit --> Batch[Process 25 Dags per batch]
Batch --> Aggregate[Aggregate selected Day or Week view]
Aggregate --> Stream[Return NDJSON TimeScheduleBatch]
Stream --> Hook
Hook --> Render[Progressively render timeline bars]
Loading

The browser now receives only the filtered data needed for the selected view, while the backend controls the result size and aggregation cost.

I initially intended this change to remain UI-only and did not plan to add a backend API. Because the review identified the scalability limitations of the original approach, I needed to first understand Airflow's backend API structure, streaming session lifecycle, authorization filters, and OpenAPI generation flow. I also spent additional time reviewing the AI-assisted code carefully, which is why the follow-up changes took about 5 days.

The implementation is now focused on the production-scale concerns from the review while preserving the existing Airflow UI/API patterns.

@minyeamer

Copy link
Copy Markdown
ContributorAuthor

I missed these additional suggestions in my previous follow-up. I plan to address the following:

  • Move Time Schedule into the Dashboard as a tab.
  • Add state icons to timeline bars so states are not distinguished by color alone.
  • Improve labels and tooltips:
    • Rename Latest 200 to Limit 200.
    • Add context for the duration aggregation options.
  • Replace the Day/Week selector with tabs.
  • Consolidate the remaining filters through the shared Filter Pill UX.

For the partition-key arrows, I understand this may include dependencies between Dag runs, such as those created through TriggerDagRun. However, with the timeline sorted alphabetically by dag_id, the connections could easily cross and become difficult to follow.

I’m happy to implement this with more specific guidance on which relationships to show and how they should be laid out. With the current context, I’m hesitant to add a potentially confusing visualization.


Drafted-by: GPT-5

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:translationsarea:UIRelated to UI/UX. For Frontend Developers.kind:documentationtranslation:default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@minyeamer@bbovenzi
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

UI: Add cross-Dag Time Schedule view - #71558

Open
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule
Open

UI: Add cross-Dag Time Schedule view#71558
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule

Conversation

@minyeamer

Copy link
Copy Markdown
Contributor

Add a Time Schedule tab for comparing Dag start times and durations across an Airflow environment.

Features:

  • Day and Week timelines with overlap handling and fixed headers
  • Mean, Max, and Min aggregation within zoom-dependent time buckets
  • Pointer-anchored zoom using buttons, the mouse wheel, or arrow keys
  • Existing Dag run, tag, timetable type, and optional team filters
  • Configurable limits from the latest 200 runs through all matching runs
  • Contiguous API pagination using the configured page size
  • Gray expected-run bars for scheduled Dags without matching Dag runs
  • Readable tooltips and links to the relevant Dag or Dag run
  • Saved view, aggregation, run-limit, and scheduled-only preferences
  • Light and dark theme support

This is a UI-only change. It reuses existing endpoints and adds no schema or backend API changes.

Earlier work in #68532 and #68547 informed the time-bucket design. The view placement follows #45497, while bounded loading follows the work in #47900, #60241, and #65388.

Related: #22001


Screenshots:

  • Day view:

time_schedule-day-tooltip

  • Week view:
time_schedule-week-tooltip

Videos:

  • Zoom (Day view):
time_schedule-day-zoom_in_out.mp4
  • Zoom (Week view):
time_schedule-week-zoom_in_out.mp4
  • Dag run links (Day view):
time_schedule-day-bar_link.mp4
  • Dag run limits (Week view):
time_schedule-week-display_n_dags.mp4
  • Aggregation (Day view):
time_schedule-day-aggregation.mp4

Was generative AI tooling used to co-author this PR?
  • Yes (GPT-5.6 Sol)

Generated-by: Codex (GPT-5.6 Sol) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Features:
- Add Day and Week timelines for Dag run timing patterns
- Group runs by Dag, state, time bucket, and weekday
- Support Mean, Max, and Min duration aggregation
- Show expected runs for scheduled Dags without matching runs
- Add Dag run, tag, timetable type, and team filters
Interactions:
- Add pointer-anchored timeline zoom controls
- Link timeline bars to their Dag or Dag run
- Add readable tooltips for run details
- Persist view, aggregation, limit, and filter preferences
- Support overlapping runs in Day and Week layouts
Data loading:
- Add selectable limits from 200 runs through all runs
- Paginate using the API-configured page size
- Prevent gaps between paginated Dag run results
- Limit Dag detail requests to expected-run markers
Documentation and tests:
- Document Time Schedule behavior in the UI guide
- Add light and dark screenshots
- Cover layouts, aggregation, filters, zoom, and pagination
@bbovenzi

Copy link
Copy Markdown
Contributor

Very cool! I could see a world where we could even show arrows between runs that are connected via partition keys. Or highlighting when Max active dag runs is being hit.

Love the minute detailing. That's great for dags that run every minute! Just needs a label.

  1. Let's move this as a tab inside the Dashboard home page instead of in the tabled dags/runs/tis view.
  2. Let's add the state icons to the bars so that they can be distinguished by something other than color
  3. We need labels and/or tooltips to help explain some of these filters:
    a) Latest 200 should probably be "Limit" to be consistent with other views
    b) Mean/Max/Min means nothing to me without context
  4. Day/Week modes are different types of views not a simple setting. It should probably match the tabs we have to change the calendar view on the dag details page.
  1. The filters are a bit inconsistent. We have the Filter Pill UX that is used elsewhere but separately we also have timetable filters, scheduled dags only filter, and tag filters. Let's consolidate them all together. I also recommend rebasing since @ryanahamilton's recent changes add some great examples.

@bbovenzibbovenzi added this to the Airflow 3.4.0 milestone Aug 21, 2026

@bbovenzibbovenzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a cool demo. But it definitely needs some work to "productionize" the code to be usable for deployments with any degree of scale.

Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/docs/ui.rst Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Capture changes during partial author review:
Data access and correctness:
- Avoid browser request fan-out and client-side filtering as Dag and run counts grow.
- Bound each view to 5,000 matching runs rather than allowing unbounded reads.
- Calculate only the selected Day or Week aggregation with access controls and filters applied.
Interaction quality:
- Stream timeline batches so useful data appears before the complete result is ready.
- Preserve displayed bars during zoom-driven refreshes while the final zoom level settles.
- Keep planned Dag markers visible even when no Dag runs match the current view.
Contract and verification:
- Keep the private UI OpenAPI contract and generated client aligned with the endpoint.
- Cover streaming, limits, filtering, aggregation, planned runs, and zoom behavior.
- Avoid MySQL's unsupported LIMIT subquery in an IN predicate.
- Mark NDJSON reads as sequential and fix TimeSchedule type errors.
- Confirm earlier UI import failures no longer affect asset builds.
Keep Time Schedule compatible with the shared FilterBar refactor merged from main.
- Deleted Dag list filter imports caused Vite module-resolution failures.
- The failure blocked React UI tests and CI jobs that build UI assets.
- Reuse the shared filter state for Tag and Timetable Type filtering.
@minyeamer

Copy link
Copy Markdown
ContributorAuthor

@bbovenzi
Thanks for the detailed review. I interpreted the four main concerns as scalability and
data-flow issues rather than only UI adjustments.

Since b13f993, excluding merge commits, I made the following changes:

  • Replaced client-side pagination and request fan-out with a dedicated streaming endpoint.
  • Removed the unbounded "All Dag runs" option and enforced a maximum of 5,000 runs on the backend.
  • Moved Dag and Dag run filtering into the backend query construction.
  • Aggregate only the selected Day or Week view on the server.
  • Stream results in batches so the UI can render progressively.
  • Updated the filter implementation after the shared FilterBar changes were merged.
  • Kept the private OpenAPI contract and generated client aligned with the backend route.

The current flow is:

flowchart TD
UI[Time Schedule UI] --> Hook[useTimeScheduleData]
Hook -->|One GET request| API[GET /ui/time-schedule]
API --> Filter[Apply authorization and filters in SQL]
Filter --> Limit[Select up to 5,000 Dag runs]
Limit --> Batch[Process 25 Dags per batch]
Batch --> Aggregate[Aggregate selected Day or Week view]
Aggregate --> Stream[Return NDJSON TimeScheduleBatch]
Stream --> Hook
Hook --> Render[Progressively render timeline bars]
Loading

The browser now receives only the filtered data needed for the selected view, while the backend controls the result size and aggregation cost.

I initially intended this change to remain UI-only and did not plan to add a backend API. Because the review identified the scalability limitations of the original approach, I needed to first understand Airflow's backend API structure, streaming session lifecycle, authorization filters, and OpenAPI generation flow. I also spent additional time reviewing the AI-assisted code carefully, which is why the follow-up changes took about 5 days.

The implementation is now focused on the production-scale concerns from the review while preserving the existing Airflow UI/API patterns.

@minyeamer

Copy link
Copy Markdown
ContributorAuthor

I missed these additional suggestions in my previous follow-up. I plan to address the following:

  • Move Time Schedule into the Dashboard as a tab.
  • Add state icons to timeline bars so states are not distinguished by color alone.
  • Improve labels and tooltips:
    • Rename Latest 200 to Limit 200.
    • Add context for the duration aggregation options.
  • Replace the Day/Week selector with tabs.
  • Consolidate the remaining filters through the shared Filter Pill UX.

For the partition-key arrows, I understand this may include dependencies between Dag runs, such as those created through TriggerDagRun. However, with the timeline sorted alphabetically by dag_id, the connections could easily cross and become difficult to follow.

I’m happy to implement this with more specific guidance on which relationships to show and how they should be laid out. With the current context, I’m hesitant to add a potentially confusing visualization.


Drafted-by: GPT-5

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:translationsarea:UIRelated to UI/UX. For Frontend Developers.kind:documentationtranslation:default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@minyeamer@bbovenzi
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

UI: Add cross-Dag Time Schedule view - #71558

Open
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule
Open

UI: Add cross-Dag Time Schedule view#71558
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule

Conversation

@minyeamer

Copy link
Copy Markdown
Contributor

Add a Time Schedule tab for comparing Dag start times and durations across an Airflow environment.

Features:

  • Day and Week timelines with overlap handling and fixed headers
  • Mean, Max, and Min aggregation within zoom-dependent time buckets
  • Pointer-anchored zoom using buttons, the mouse wheel, or arrow keys
  • Existing Dag run, tag, timetable type, and optional team filters
  • Configurable limits from the latest 200 runs through all matching runs
  • Contiguous API pagination using the configured page size
  • Gray expected-run bars for scheduled Dags without matching Dag runs
  • Readable tooltips and links to the relevant Dag or Dag run
  • Saved view, aggregation, run-limit, and scheduled-only preferences
  • Light and dark theme support

This is a UI-only change. It reuses existing endpoints and adds no schema or backend API changes.

Earlier work in #68532 and #68547 informed the time-bucket design. The view placement follows #45497, while bounded loading follows the work in #47900, #60241, and #65388.

Related: #22001


Screenshots:

  • Day view:

time_schedule-day-tooltip

  • Week view:
time_schedule-week-tooltip

Videos:

  • Zoom (Day view):
time_schedule-day-zoom_in_out.mp4
  • Zoom (Week view):
time_schedule-week-zoom_in_out.mp4
  • Dag run links (Day view):
time_schedule-day-bar_link.mp4
  • Dag run limits (Week view):
time_schedule-week-display_n_dags.mp4
  • Aggregation (Day view):
time_schedule-day-aggregation.mp4

Was generative AI tooling used to co-author this PR?
  • Yes (GPT-5.6 Sol)

Generated-by: Codex (GPT-5.6 Sol) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Features:
- Add Day and Week timelines for Dag run timing patterns
- Group runs by Dag, state, time bucket, and weekday
- Support Mean, Max, and Min duration aggregation
- Show expected runs for scheduled Dags without matching runs
- Add Dag run, tag, timetable type, and team filters
Interactions:
- Add pointer-anchored timeline zoom controls
- Link timeline bars to their Dag or Dag run
- Add readable tooltips for run details
- Persist view, aggregation, limit, and filter preferences
- Support overlapping runs in Day and Week layouts
Data loading:
- Add selectable limits from 200 runs through all runs
- Paginate using the API-configured page size
- Prevent gaps between paginated Dag run results
- Limit Dag detail requests to expected-run markers
Documentation and tests:
- Document Time Schedule behavior in the UI guide
- Add light and dark screenshots
- Cover layouts, aggregation, filters, zoom, and pagination
@bbovenzi

Copy link
Copy Markdown
Contributor

Very cool! I could see a world where we could even show arrows between runs that are connected via partition keys. Or highlighting when Max active dag runs is being hit.

Love the minute detailing. That's great for dags that run every minute! Just needs a label.

  1. Let's move this as a tab inside the Dashboard home page instead of in the tabled dags/runs/tis view.
  2. Let's add the state icons to the bars so that they can be distinguished by something other than color
  3. We need labels and/or tooltips to help explain some of these filters:
    a) Latest 200 should probably be "Limit" to be consistent with other views
    b) Mean/Max/Min means nothing to me without context
  4. Day/Week modes are different types of views not a simple setting. It should probably match the tabs we have to change the calendar view on the dag details page.
  1. The filters are a bit inconsistent. We have the Filter Pill UX that is used elsewhere but separately we also have timetable filters, scheduled dags only filter, and tag filters. Let's consolidate them all together. I also recommend rebasing since @ryanahamilton's recent changes add some great examples.

@bbovenzibbovenzi added this to the Airflow 3.4.0 milestone Aug 21, 2026

@bbovenzibbovenzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a cool demo. But it definitely needs some work to "productionize" the code to be usable for deployments with any degree of scale.

Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/docs/ui.rst Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Capture changes during partial author review:
Data access and correctness:
- Avoid browser request fan-out and client-side filtering as Dag and run counts grow.
- Bound each view to 5,000 matching runs rather than allowing unbounded reads.
- Calculate only the selected Day or Week aggregation with access controls and filters applied.
Interaction quality:
- Stream timeline batches so useful data appears before the complete result is ready.
- Preserve displayed bars during zoom-driven refreshes while the final zoom level settles.
- Keep planned Dag markers visible even when no Dag runs match the current view.
Contract and verification:
- Keep the private UI OpenAPI contract and generated client aligned with the endpoint.
- Cover streaming, limits, filtering, aggregation, planned runs, and zoom behavior.
- Avoid MySQL's unsupported LIMIT subquery in an IN predicate.
- Mark NDJSON reads as sequential and fix TimeSchedule type errors.
- Confirm earlier UI import failures no longer affect asset builds.
Keep Time Schedule compatible with the shared FilterBar refactor merged from main.
- Deleted Dag list filter imports caused Vite module-resolution failures.
- The failure blocked React UI tests and CI jobs that build UI assets.
- Reuse the shared filter state for Tag and Timetable Type filtering.
@minyeamer

Copy link
Copy Markdown
ContributorAuthor

@bbovenzi
Thanks for the detailed review. I interpreted the four main concerns as scalability and
data-flow issues rather than only UI adjustments.

Since b13f993, excluding merge commits, I made the following changes:

  • Replaced client-side pagination and request fan-out with a dedicated streaming endpoint.
  • Removed the unbounded "All Dag runs" option and enforced a maximum of 5,000 runs on the backend.
  • Moved Dag and Dag run filtering into the backend query construction.
  • Aggregate only the selected Day or Week view on the server.
  • Stream results in batches so the UI can render progressively.
  • Updated the filter implementation after the shared FilterBar changes were merged.
  • Kept the private OpenAPI contract and generated client aligned with the backend route.

The current flow is:

flowchart TD
UI[Time Schedule UI] --> Hook[useTimeScheduleData]
Hook -->|One GET request| API[GET /ui/time-schedule]
API --> Filter[Apply authorization and filters in SQL]
Filter --> Limit[Select up to 5,000 Dag runs]
Limit --> Batch[Process 25 Dags per batch]
Batch --> Aggregate[Aggregate selected Day or Week view]
Aggregate --> Stream[Return NDJSON TimeScheduleBatch]
Stream --> Hook
Hook --> Render[Progressively render timeline bars]
Loading

The browser now receives only the filtered data needed for the selected view, while the backend controls the result size and aggregation cost.

I initially intended this change to remain UI-only and did not plan to add a backend API. Because the review identified the scalability limitations of the original approach, I needed to first understand Airflow's backend API structure, streaming session lifecycle, authorization filters, and OpenAPI generation flow. I also spent additional time reviewing the AI-assisted code carefully, which is why the follow-up changes took about 5 days.

The implementation is now focused on the production-scale concerns from the review while preserving the existing Airflow UI/API patterns.

@minyeamer

Copy link
Copy Markdown
ContributorAuthor

I missed these additional suggestions in my previous follow-up. I plan to address the following:

  • Move Time Schedule into the Dashboard as a tab.
  • Add state icons to timeline bars so states are not distinguished by color alone.
  • Improve labels and tooltips:
    • Rename Latest 200 to Limit 200.
    • Add context for the duration aggregation options.
  • Replace the Day/Week selector with tabs.
  • Consolidate the remaining filters through the shared Filter Pill UX.

For the partition-key arrows, I understand this may include dependencies between Dag runs, such as those created through TriggerDagRun. However, with the timeline sorted alphabetically by dag_id, the connections could easily cross and become difficult to follow.

I’m happy to implement this with more specific guidance on which relationships to show and how they should be laid out. With the current context, I’m hesitant to add a potentially confusing visualization.


Drafted-by: GPT-5

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:translationsarea:UIRelated to UI/UX. For Frontend Developers.kind:documentationtranslation:default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@minyeamer@bbovenzi
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

UI: Add cross-Dag Time Schedule view - #71558

Open
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule
Open

UI: Add cross-Dag Time Schedule view#71558
minyeamer wants to merge 9 commits into
apache:mainfrom
minyeamer:ui/time-schedule

Conversation

@minyeamer

Copy link
Copy Markdown
Contributor

Add a Time Schedule tab for comparing Dag start times and durations across an Airflow environment.

Features:

  • Day and Week timelines with overlap handling and fixed headers
  • Mean, Max, and Min aggregation within zoom-dependent time buckets
  • Pointer-anchored zoom using buttons, the mouse wheel, or arrow keys
  • Existing Dag run, tag, timetable type, and optional team filters
  • Configurable limits from the latest 200 runs through all matching runs
  • Contiguous API pagination using the configured page size
  • Gray expected-run bars for scheduled Dags without matching Dag runs
  • Readable tooltips and links to the relevant Dag or Dag run
  • Saved view, aggregation, run-limit, and scheduled-only preferences
  • Light and dark theme support

This is a UI-only change. It reuses existing endpoints and adds no schema or backend API changes.

Earlier work in #68532 and #68547 informed the time-bucket design. The view placement follows #45497, while bounded loading follows the work in #47900, #60241, and #65388.

Related: #22001


Screenshots:

  • Day view:

time_schedule-day-tooltip

  • Week view:
time_schedule-week-tooltip

Videos:

  • Zoom (Day view):
time_schedule-day-zoom_in_out.mp4
  • Zoom (Week view):
time_schedule-week-zoom_in_out.mp4
  • Dag run links (Day view):
time_schedule-day-bar_link.mp4
  • Dag run limits (Week view):
time_schedule-week-display_n_dags.mp4
  • Aggregation (Day view):
time_schedule-day-aggregation.mp4

Was generative AI tooling used to co-author this PR?
  • Yes (GPT-5.6 Sol)

Generated-by: Codex (GPT-5.6 Sol) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Features:
- Add Day and Week timelines for Dag run timing patterns
- Group runs by Dag, state, time bucket, and weekday
- Support Mean, Max, and Min duration aggregation
- Show expected runs for scheduled Dags without matching runs
- Add Dag run, tag, timetable type, and team filters
Interactions:
- Add pointer-anchored timeline zoom controls
- Link timeline bars to their Dag or Dag run
- Add readable tooltips for run details
- Persist view, aggregation, limit, and filter preferences
- Support overlapping runs in Day and Week layouts
Data loading:
- Add selectable limits from 200 runs through all runs
- Paginate using the API-configured page size
- Prevent gaps between paginated Dag run results
- Limit Dag detail requests to expected-run markers
Documentation and tests:
- Document Time Schedule behavior in the UI guide
- Add light and dark screenshots
- Cover layouts, aggregation, filters, zoom, and pagination
@bbovenzi

Copy link
Copy Markdown
Contributor

Very cool! I could see a world where we could even show arrows between runs that are connected via partition keys. Or highlighting when Max active dag runs is being hit.

Love the minute detailing. That's great for dags that run every minute! Just needs a label.

  1. Let's move this as a tab inside the Dashboard home page instead of in the tabled dags/runs/tis view.
  2. Let's add the state icons to the bars so that they can be distinguished by something other than color
  3. We need labels and/or tooltips to help explain some of these filters:
    a) Latest 200 should probably be "Limit" to be consistent with other views
    b) Mean/Max/Min means nothing to me without context
  4. Day/Week modes are different types of views not a simple setting. It should probably match the tabs we have to change the calendar view on the dag details page.
  1. The filters are a bit inconsistent. We have the Filter Pill UX that is used elsewhere but separately we also have timetable filters, scheduled dags only filter, and tag filters. Let's consolidate them all together. I also recommend rebasing since @ryanahamilton's recent changes add some great examples.

@bbovenzibbovenzi added this to the Airflow 3.4.0 milestone Aug 21, 2026

@bbovenzibbovenzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a cool demo. But it definitely needs some work to "productionize" the code to be usable for deployments with any degree of scale.

Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/docs/ui.rst Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Comment threadairflow-core/src/airflow/ui/src/pages/TimeSchedule/useTimeScheduleData.ts Outdated
Capture changes during partial author review:
Data access and correctness:
- Avoid browser request fan-out and client-side filtering as Dag and run counts grow.
- Bound each view to 5,000 matching runs rather than allowing unbounded reads.
- Calculate only the selected Day or Week aggregation with access controls and filters applied.
Interaction quality:
- Stream timeline batches so useful data appears before the complete result is ready.
- Preserve displayed bars during zoom-driven refreshes while the final zoom level settles.
- Keep planned Dag markers visible even when no Dag runs match the current view.
Contract and verification:
- Keep the private UI OpenAPI contract and generated client aligned with the endpoint.
- Cover streaming, limits, filtering, aggregation, planned runs, and zoom behavior.
- Avoid MySQL's unsupported LIMIT subquery in an IN predicate.
- Mark NDJSON reads as sequential and fix TimeSchedule type errors.
- Confirm earlier UI import failures no longer affect asset builds.
Keep Time Schedule compatible with the shared FilterBar refactor merged from main.
- Deleted Dag list filter imports caused Vite module-resolution failures.
- The failure blocked React UI tests and CI jobs that build UI assets.
- Reuse the shared filter state for Tag and Timetable Type filtering.
@minyeamer

Copy link
Copy Markdown
ContributorAuthor

@bbovenzi
Thanks for the detailed review. I interpreted the four main concerns as scalability and
data-flow issues rather than only UI adjustments.

Since b13f993, excluding merge commits, I made the following changes:

  • Replaced client-side pagination and request fan-out with a dedicated streaming endpoint.
  • Removed the unbounded "All Dag runs" option and enforced a maximum of 5,000 runs on the backend.
  • Moved Dag and Dag run filtering into the backend query construction.
  • Aggregate only the selected Day or Week view on the server.
  • Stream results in batches so the UI can render progressively.
  • Updated the filter implementation after the shared FilterBar changes were merged.
  • Kept the private OpenAPI contract and generated client aligned with the backend route.

The current flow is:

flowchart TD
UI[Time Schedule UI] --> Hook[useTimeScheduleData]
Hook -->|One GET request| API[GET /ui/time-schedule]
API --> Filter[Apply authorization and filters in SQL]
Filter --> Limit[Select up to 5,000 Dag runs]
Limit --> Batch[Process 25 Dags per batch]
Batch --> Aggregate[Aggregate selected Day or Week view]
Aggregate --> Stream[Return NDJSON TimeScheduleBatch]
Stream --> Hook
Hook --> Render[Progressively render timeline bars]
Loading

The browser now receives only the filtered data needed for the selected view, while the backend controls the result size and aggregation cost.

I initially intended this change to remain UI-only and did not plan to add a backend API. Because the review identified the scalability limitations of the original approach, I needed to first understand Airflow's backend API structure, streaming session lifecycle, authorization filters, and OpenAPI generation flow. I also spent additional time reviewing the AI-assisted code carefully, which is why the follow-up changes took about 5 days.

The implementation is now focused on the production-scale concerns from the review while preserving the existing Airflow UI/API patterns.

@minyeamer

Copy link
Copy Markdown
ContributorAuthor

I missed these additional suggestions in my previous follow-up. I plan to address the following:

  • Move Time Schedule into the Dashboard as a tab.
  • Add state icons to timeline bars so states are not distinguished by color alone.
  • Improve labels and tooltips:
    • Rename Latest 200 to Limit 200.
    • Add context for the duration aggregation options.
  • Replace the Day/Week selector with tabs.
  • Consolidate the remaining filters through the shared Filter Pill UX.

For the partition-key arrows, I understand this may include dependencies between Dag runs, such as those created through TriggerDagRun. However, with the timeline sorted alphabetically by dag_id, the connections could easily cross and become difficult to follow.

I’m happy to implement this with more specific guidance on which relationships to show and how they should be laid out. With the current context, I’m hesitant to add a potentially confusing visualization.


Drafted-by: GPT-5

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:translationsarea:UIRelated to UI/UX. For Frontend Developers.kind:documentationtranslation:default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@minyeamer@bbovenzi