feat: add pgjson format support for EXPLAIN ANALYZE by adriangb · Pull Request #21767 · apache/datafusion

adriangb · 2026-04-21T19:41:17Z

Which issue does this PR close?

Closes #.

Rationale for this change

DataFusion already emits PostgreSQL JSON (pgjson) for logical plans via EXPLAIN (FORMAT pgjson) .... This PR extends that support to EXPLAIN ANALYZE so the physical plan, along with live execution metrics, can be fed into pgjson visualizers such as Dalibo and PEV2.

Today, EXPLAIN ANALYZE FORMAT pgjson is explicitly rejected in the planner with "EXPLAIN ANALYZE with FORMAT is not supported". With this PR the restriction is lifted for pgjson.

What changes are included in this PR?

Add a format: ExplainFormat field to the logical Analyze node and the physical AnalyzeExec operator, threaded through SQL parsing, logical planning, and physical planning.
Accept EXPLAIN ANALYZE FORMAT pgjson <stmt>. Tree and Graphviz with ANALYZE still error with a clear message (out of scope for this PR).
Add DisplayableExecutionPlan::pgjson() and a new PgJsonExecutionPlanVisitor that mirror the logical-plan PgJsonVisitor. Per-node output includes:
- Node Type — ExecutionPlan::name()
- Details — the one-line DisplayAs::Default rendering
- Actual Rows / Actual Total Time — PG-canonical metric keys populated from output_rows / elapsed_compute (emitted as float milliseconds; note DataFusion records compute time, not wall time)
- Extras — remaining DataFusion metrics keyed by their native name
- Plans — child nodes
Add an optional set_summary() builder so AnalyzeExec can attach Total Rows and Duration at the root in verbose mode.
Honor existing analyze_level / analyze_categories config exactly as indent() does.
Update the EXPLAIN user-guide docs (docs/source/user-guide/sql/explain.md and explain-usage.md) to document pgjson support under ANALYZE and lead with the Postgres-style option-list spelling.

Composes with the `EXPLAIN (...)` option list (#21768)

This builds on the now-merged Postgres-style option list (#21768). Because both the keyword form and the parenthesized option list parse into a single ExplainStatementOptions that is threaded through explain_to_plan, pgjson works with both spellings, and the METRICS / LEVEL knobs from #21768 compose with it in one statement:

EXPLAIN (ANALYZE, FORMAT pgjson) SELECTcount(*) FROM t;
EXPLAIN (ANALYZE, FORMAT pgjson, METRICS 'rows', LEVEL summary) SELECTcount(*) FROM t;

The parenthesized form is the idiomatic spelling for pgjson workflows since it mirrors Postgres's EXPLAIN (ANALYZE, FORMAT json) — exactly what visualizers like Dalibo / PEV2 document. (Note: ANALYZE must go inside the parens; a bare EXPLAIN ANALYZE (FORMAT pgjson) is invalid, as it is in Postgres.)

Are these changes tested?

Unit tests in datafusion/physical-plan/src/display.rs:
- pgjson_renders_plan_without_metrics
- pgjson_includes_summary_when_set
- pgjson_snapshot_of_sample_plan (insta snapshot)
sqllogictest coverage in datafusion/sqllogictest/test_files/explain_analyze.slt:
- Structural golden for EXPLAIN (ANALYZE, FORMAT PGJSON, METRICS 'none') (option-list form)
- EXPLAIN (ANALYZE, FORMAT PGJSON, METRICS 'rows') showing Actual Rows surfacing
- Keyword form EXPLAIN ANALYZE FORMAT pgjson still works
- Negative tests for EXPLAIN ANALYZE FORMAT tree and EXPLAIN ANALYZE FORMAT graphviz
cargo clippy --all-targets --all-features -- -D warnings clean on the touched crates; cargo fmt --all clean.

Are there any user-facing changes?

Yes — EXPLAIN ANALYZE now accepts the pgjson format, in either spelling:

-- Postgres-style option list (idiomatic; composes with METRICS / LEVEL)
EXPLAIN (ANALYZE, FORMAT pgjson) SELECTcount(*) FROM t;
-- legacy keyword form
EXPLAIN ANALYZE FORMAT pgjson SELECTcount(*) FROM t;

No existing behavior changes: the default (EXPLAIN ANALYZE ... with no FORMAT) still emits the indent-format plan with metrics, and EXPLAIN (FORMAT pgjson) ... on the logical plan is unchanged.

🤖 Generated with Claude Code

adriangb · 2026-05-28T21:16:21Z

check: do we need to allow the new option to serialize to protobuf?

Copilot

Pull request overview

Extends DataFusion’s EXPLAIN ANALYZE to support PostgreSQL JSON (pgjson) output so physical plans with live metrics can be consumed by existing Postgres plan visualizers.

Changes:

Thread ExplainFormat through SQL planning (Analyze logical node) and physical execution (AnalyzeExec) so EXPLAIN ANALYZE FORMAT pgjson is accepted.
Add DisplayableExecutionPlan::pgjson() plus a physical-plan pgjson visitor that emits Postgres-style keys and maps key metrics into Actual Rows / Actual Total Time.
Add sqllogictest + unit tests for pgjson output and for rejecting unsupported tree / graphviz formats under ANALYZE.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
datafusion/sqllogictest/test_files/explain_analyze.slt	Adds golden/negative sqllogictest coverage for `EXPLAIN ANALYZE FORMAT pgjson`.
datafusion/sql/src/statement.rs	Lifts FORMAT restriction for `EXPLAIN ANALYZE` (pgjson only), keeps other formats rejected.
datafusion/physical-plan/src/display.rs	Implements physical-plan pgjson rendering + unit tests and optional root summary attachment.
datafusion/physical-plan/src/analyze.rs	Threads `ExplainFormat` into `AnalyzeExec` and switches output formatting based on the format.
datafusion/physical-plan/Cargo.toml	Adds `serde_json` dependency (and `preserve_order` for dev/snapshot tests).
datafusion/expr/src/logical_plan/tree_node.rs	Propagates new `Analyze.format` field through tree node rewrites.
datafusion/expr/src/logical_plan/plan.rs	Adds `format: ExplainFormat` to logical `Analyze`.
datafusion/expr/src/logical_plan/builder.rs	Populates `Analyze.format` when building plans via `ExplainOption`.
datafusion/core/src/physical_planner.rs	Passes `Analyze.format` into `AnalyzeExec` during physical planning.
Cargo.lock	Updates lockfile for new dependency usage.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-05-28T21:24:36Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

 Cloning apache/main
Building datafusion v53.1.0 (current)
Built [ 77.402s] (current)
Parsing datafusion v53.1.0 (current)
Parsed [ 0.030s] (current)
Building datafusion v53.1.0 (baseline)
Built [ 77.507s] (baseline)
Parsing datafusion v53.1.0 (baseline)
Parsed [ 0.030s] (baseline)
Checking datafusion v53.1.0 -> v53.1.0 (no change; assume patch)
Checked [ 0.714s] 222 checks: 222 pass, 30 skip
Summary no semver update required
Finished [ 158.023s] datafusion
Building datafusion-expr v53.1.0 (current)
Built [ 20.655s] (current)
Parsing datafusion-expr v53.1.0 (current)
Parsed [ 0.061s] (current)
Building datafusion-expr v53.1.0 (baseline)
Built [ 20.826s] (baseline)
Parsing datafusion-expr v53.1.0 (baseline)
Parsed [ 0.062s] (baseline)
Checking datafusion-expr v53.1.0 -> v53.1.0 (no change; assume patch)
Checked [ 1.390s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip
--- failure constructible_struct_adds_field: externally-constructible struct adds field ---
Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/constructible_struct_adds_field.ron
Failed in:
field Analyze.format in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:3474
field Analyze.format in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:3474
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 44.365s] datafusion-expr
Building datafusion-physical-plan v53.1.0 (current)
Built [ 28.062s] (current)
Parsing datafusion-physical-plan v53.1.0 (current)
Parsed [ 0.104s] (current)
Building datafusion-physical-plan v53.1.0 (baseline)
Built [ 28.407s] (baseline)
Parsing datafusion-physical-plan v53.1.0 (baseline)
Parsed [ 0.105s] (baseline)
Checking datafusion-physical-plan v53.1.0 -> v53.1.0 (no change; assume patch)
Checked [ 0.653s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip
--- failure inherent_method_missing: pub method removed or renamed ---
Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/inherent_method_missing.ron
Failed in:
AnalyzeExec::new, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/49c480626e9865884f9e36bc429bd058e3cd2217/datafusion/physical-plan/src/analyze.rs:60
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 58.977s] datafusion-physical-plan
Building datafusion-proto v53.1.0 (current)
Built [ 45.394s] (current)
Parsing datafusion-proto v53.1.0 (current)
Parsed [ 0.015s] (current)
Building datafusion-proto v53.1.0 (baseline)
Built [ 45.723s] (baseline)
Parsing datafusion-proto v53.1.0 (baseline)
Parsed [ 0.016s] (baseline)
Checking datafusion-proto v53.1.0 -> v53.1.0 (no change; assume patch)
Checked [ 0.271s] 222 checks: 222 pass, 30 skip
Summary no semver update required
Finished [ 93.208s] datafusion-proto
Building datafusion-proto-models v53.1.0 (current)
Built [ 19.158s] (current)
Parsing datafusion-proto-models v53.1.0 (current)
Parsed [ 0.099s] (current)
Building datafusion-proto-models v53.1.0 (baseline)
Built [ 18.972s] (baseline)
Parsing datafusion-proto-models v53.1.0 (baseline)
Parsed [ 0.105s] (baseline)
Checking datafusion-proto-models v53.1.0 -> v53.1.0 (no change; assume patch)
Checked [ 1.736s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip
--- failure constructible_struct_adds_field: externally-constructible struct adds field ---
Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/constructible_struct_adds_field.ron
Failed in:
field AnalyzeExecNode.format in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1858
field AnalyzeExecNode.format in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1858
field AnalyzeNode.format in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:358
field AnalyzeNode.format in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:358
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 41.679s] datafusion-proto-models
Building datafusion-sql v53.1.0 (current)
Built [ 32.054s] (current)
Parsing datafusion-sql v53.1.0 (current)
Parsed [ 0.026s] (current)
Building datafusion-sql v53.1.0 (baseline)
Built [ 32.071s] (baseline)
Parsing datafusion-sql v53.1.0 (baseline)
Parsed [ 0.026s] (baseline)
Checking datafusion-sql v53.1.0 -> v53.1.0 (no change; assume patch)
Checked [ 0.249s] 222 checks: 222 pass, 30 skip
Summary no semver update required
Finished [ 66.400s] datafusion-sql
Building datafusion-sqllogictest v53.1.0 (current)
Built [ 133.321s] (current)
Parsing datafusion-sqllogictest v53.1.0 (current)
Parsed [ 0.019s] (current)
Building datafusion-sqllogictest v53.1.0 (baseline)
Built [ 132.437s] (baseline)
Parsing datafusion-sqllogictest v53.1.0 (baseline)
Parsed [ 0.021s] (baseline)
Checking datafusion-sqllogictest v53.1.0 -> v53.1.0 (no change; assume patch)
Checked [ 0.100s] 222 checks: 222 pass, 30 skip
Summary no semver update required
Finished [ 270.133s] datafusion-sqllogictest

adriangb · 2026-05-28T21:53:06Z

@geoffreyclaude curious if you're interested in this since you also work on datafusion execution observability in https://github.com/datafusion-contrib/datafusion-tracing.

`AnalyzeExec::new` took 6 positional arguments; callers were forced to repeat the hardcoded `vec![MetricType::Summary, MetricType::Dev]` default and pass `None` for metric_categories even when not needed. Introduce `AnalyzeExecBuilder` with required params in the constructor and optional params (`metric_types`, `metric_categories`) as builder methods with sensible defaults. Add `AnalyzeExec::builder(...)` as a convenience entry-point so callers don't need a separate import. Remove `AnalyzeExec::new` (breaking change). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add `EXPLAIN ANALYZE FORMAT PGJSON` / `EXPLAIN (ANALYZE, FORMAT PGJSON)` support. The new format emits a PostgreSQL-compatible JSON plan tree with per-operator metrics attached as Actual Rows / Actual Total Time / Extras keys, matching the shape consumed by tools like pganalyze. Implementation: - `Analyze` logical plan node gains a `format: ExplainFormat` field - `AnalyzeExecBuilder` gains `with_format`; planner wires it through - New `PgJsonExecutionPlanVisitor` in display.rs walks the physical plan and builds the JSON tree bottom-up - `serde_json` promoted to `preserve_order` so key insertion order is stable in production builds Only `Indent` and `PostgresJSON` are valid with `ANALYZE`; `Tree` and `Graphviz` are rejected at planning time with a clear error. slt tests use the inline `EXPLAIN (ANALYZE, FORMAT PGJSON, METRICS ...)` form introduced in apache#21768. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add ExplainFormat to both the logical-plan AnalyzeNode (field 5) and physical-plan AnalyzeExecNode (field 7) so the pgjson format survives proto roundtrips. Wire up encode/decode in both codecs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

adriangb · 2026-05-28T23:37:13Z

@milenkovicm another EXPLAIN ANALYZE improvement in case you're interested

milenkovicm · 2026-05-29T05:34:56Z

Thanks @adriangb will have a look. Also I believe @martin-g may like this PR

geoffreyclaude · 2026-05-29T08:51:20Z

+ displayable = displayable
+ .set_metric_types(metric_types.to_vec())
+ .set_metric_categories(metric_categories.map(|c| c.to_vec()))
+ .set_show_statistics(show_statistics);


Looks like show_statistics is ignored in the pgjson path: it’s set here, but pgjson() never uses it. Should this either render stats in the JSON or reject show_statistics for pgjson for now?

geoffreyclaude · 2026-05-29T08:52:42Z

@geoffreyclaude curious if you're interested in this since you also work on datafusion execution observability in https://github.com/datafusion-contrib/datafusion-tracing.

@adriangb So this is what you meant in datafusion-contrib/datafusion-tracing#25 then? I think it makes way more sense to do it natively in DataFusion directly through this PR.

adriangb · 2026-05-29T13:13:12Z

@geoffreyclaude curious if you're interested in this since you also work on datafusion execution observability in https://github.com/datafusion-contrib/datafusion-tracing.
@adriangb So this is what you meant in datafusion-contrib/datafusion-tracing#25 then? I think it makes way more sense to do it natively in DataFusion directly through this PR.

Ah sorry I never responded there! Yes we basically have an implementation of this internally and run it on the completed plan and then attach the attribute to the root of execution span.

…d imports Address review feedback: show_statistics was forwarded into the pgjson output path but pgjson() never consumes it, so the option was silently ignored. Reject the show_statistics + pgjson combination at planning time (consistent with the existing tree/graphviz rejection) and stop forwarding the flag in the pgjson branch of create_output_batch. Also remove unused `MetricType` imports that were breaking the `clippy --all-targets --all-features -- -D warnings` CI job in datafusion-core, datafusion-proto, and the proto roundtrip tests. https://claude.ai/code/session_01L3a6UFPrLSjX9sFtj1Us2i

The planner now includes the rejected format name in the error ("EXPLAIN ANALYZE with FORMAT tree is not supported") since ANALYZE supports indent and pgjson. Update the expected error and comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ling Note that EXPLAIN ANALYZE now supports pgjson (not just indent), add a worked EXPLAIN (ANALYZE, FORMAT pgjson, METRICS 'rows') example, and cross-link the Postgres-style option list (apache#21768) which is the idiomatic spelling for pgjson visualizer workflows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

milenkovicm · 2026-05-29T18:51:43Z

will have a look tomorrow if not too late @adriangb

milenkovicm

Thanks @adriangb
really neat PR, great addition

adriangb · 2026-05-30T16:35:46Z

Thanks for the review!

github-actionsBot added sql SQL Planner logical-expr Logical plan and expressions core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Apr 21, 2026

adriangb force-pushed the explain-analyze-pgjson branch 2 times, most recently from b4c739f to 5fc8a84Compare April 21, 2026 21:01

adriangb force-pushed the explain-analyze-pgjson branch from 5fc8a84 to 74b6f5cCompare May 28, 2026 21:16

adriangb requested a review from Copilot May 28, 2026 21:16

Copilotstarted reviewing on behalf of adriangb May 28, 2026 21:16 View session

CopilotAI reviewed May 28, 2026
View reviewed changes

Comment threaddatafusion/sql/src/statement.rs
Comment threaddatafusion/physical-plan/src/analyze.rs Outdated
Comment threaddatafusion/physical-plan/src/display.rs Outdated
Comment threaddatafusion/physical-plan/src/display.rs
Comment threaddatafusion/expr/src/logical_plan/plan.rs

github-actionsBot added auto detected api change Auto detected API change proto Related to proto crate labels May 28, 2026

adriangb force-pushed the explain-analyze-pgjson branch from 7aa80cf to 7e72c0cCompare May 28, 2026 21:48

adriangb marked this pull request as ready for review May 28, 2026 21:52

adriangb force-pushed the explain-analyze-pgjson branch 3 times, most recently from 7c06439 to e4ea7ffCompare May 28, 2026 23:21

adriangband others added 3 commits May 28, 2026 18:35

adriangb force-pushed the explain-analyze-pgjson branch from e4ea7ff to 1449722Compare May 28, 2026 23:35

geoffreyclaude reviewed May 29, 2026
View reviewed changes

geoffreyclaude mentioned this pull request May 29, 2026
Emit a Postgres style JSON plan via db.query.plandatafusion-contrib/datafusion-tracing#25
Closed

adriangband others added 2 commits May 29, 2026 10:22

github-actionsBot added the documentation Improvements or additions to documentation label May 29, 2026

adriangb requested a review from milenkovicm May 29, 2026 15:42

milenkovicm approved these changes May 30, 2026
View reviewed changes

Comment threaddatafusion/physical-plan/src/analyze.rs Outdated

Update datafusion/physical-plan/src/analyze.rs
aaa5c81

adriangb enabled auto-merge May 30, 2026 14:47

adriangb added this pull request to the merge queue May 30, 2026

Merged via the queue into apache:main with commit 496f2c2May 30, 2026
40 checks passed

adriangb deleted the explain-analyze-pgjson branch May 30, 2026 15:27

Uh oh!

Conversation

adriangb commented Apr 21, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Composes with the EXPLAIN (...) option list (#21768)

Are these changes tested?

Are there any user-facing changes?

Uh oh!

adriangb commented May 28, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actionsBot commented May 28, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adriangb commented May 28, 2026

Uh oh!

adriangb commented May 28, 2026

Uh oh!

milenkovicm commented May 29, 2026

Uh oh!

geoffreyclaudeMay 29, 2026

Choose a reason for hiding this comment

Uh oh!

geoffreyclaude commented May 29, 2026

Uh oh!

adriangb commented May 29, 2026

Uh oh!

milenkovicm commented May 29, 2026

Uh oh!

milenkovicm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

adriangb commented May 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

adriangb commented Apr 21, 2026•
edited
Loading

Composes with the `EXPLAIN (...)` option list (#21768)

github-actionsBot commented May 28, 2026•
edited
Loading