Skip to content

Add metrics for FilterExec - #960

Merged
alamb merged 1 commit into
apache:masterfrom
alamb:alamb/filter_stats
Sep 10, 2021
Merged

Add metrics for FilterExec#960
alamb merged 1 commit into
apache:masterfrom
alamb:alamb/filter_stats

Conversation

@alamb

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Next part #866

Rationale for this change

We want basic understanding of where a plan's time is spent and in what operators. See #866 for more details

What changes are included in this PR?

  1. Instrument FilterExec using the API from Add BaselineMetrics, Timestamp metrics, add for CoalescePartitionsExec, rename output_time -> elapsed_compute #909
  2. Tests for same

Are there any user-facing changes?

More fields in EXPLAIN ANALYZE are now filled out

Example of how explain analayze is looking:

EXPLAIN ANALYZE selectcount(*) from (SELECTcount(*), c1 FROM aggregate_test_100 WHERE c13 !='C2GT5KVyOPZpgKVl110TyZO0NcJ434'GROUP BY c1 ORDER BY c1)
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type | plan |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | ProjectionExec: expr=[COUNT(UInt8(1))@0asCOUNT(UInt8(1))], metrics=[] |
| | HashAggregateExec: mode=Final, gby=[], aggr=[COUNT(UInt8(1))], metrics=[output_rows=1, elapsed_compute=69.362µs] |
| | CoalescePartitionsExec, metrics=[output_rows=3, elapsed_compute=NOT RECORDED] |
| | HashAggregateExec: mode=Partial, gby=[], aggr=[COUNT(UInt8(1))], metrics=[output_rows=3, elapsed_compute=119.041µs] |
| | RepartitionExec: partitioning=RoundRobinBatch(3), metrics=[send_time{inputPartition=0}=4.775µs, fetch_time{inputPartition=0}=12.77346ms, repart_time{inputPartition=0}=NOT RECORDED] |
| | SortExec: [c1@0ASC], metrics=[output_rows=5, elapsed_compute=232.203µs] |
| | CoalescePartitionsExec, metrics=[output_rows=5, elapsed_compute=NOT RECORDED] |
| | HashAggregateExec: mode=FinalPartitioned, gby=[c1@0as c1], aggr=[COUNT(UInt8(1))], metrics=[output_rows=5, elapsed_compute=373.876µs] |
| | CoalesceBatchesExec: target_batch_size=4096, metrics=[] |
| | RepartitionExec: partitioning=Hash([Column { name: "c1", index: 0 }], 3), metrics=[fetch_time{inputPartition=0}=34.072102ms, repart_time{inputPartition=0}=252.872µs, send_time{inputPartition=0}=NOT RECORDED] |
| | HashAggregateExec: mode=Partial, gby=[c1@0as c1], aggr=[COUNT(UInt8(1))], metrics=[output_rows=5, elapsed_compute=581.405µs] |
| | CoalesceBatchesExec: target_batch_size=4096, metrics=[] |
| | FilterExec: c13@1!= C2GT5KVyOPZpgKVl110TyZO0NcJ434, metrics=[output_rows=99, elapsed_compute=309.095µs] |
| | RepartitionExec: partitioning=RoundRobinBatch(3), metrics=[fetch_time{inputPartition=0}=8.239528ms, send_time{inputPartition=0}=9.864µs, repart_time{inputPartition=0}=NOT RECORDED] |
| | CsvExec: source=Path(ARROW_TEST_DATA/csv/aggregate_test_100.csv: [ARROW_TEST_DATA/csv/aggregate_test_100.csv]), has_header=true, metrics=[] |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

@alamb

alamb commented Sep 9, 2021

Copy link
Copy Markdown
ContributorAuthor

clippy failure is due to #986 I think

@alamb
alamb merged commit 4c0d430 into apache:masterSep 10, 2021
@alamb
alamb deleted the alamb/filter_stats branch September 10, 2021 17:07
@houqphouqp added the enhancement New feature or request label Nov 6, 2021
unkloud pushed a commit to unkloud/datafusion that referenced this pull request Mar 23, 2025
## Which issue does this PR close?
Closesapache#521 and apache#665.
## What changes are included in this PR?
This PR normalize the right hand of div and rem
## How are these changes tested?
Test updated
HairstonE pushed a commit to HairstonE/datafusion that referenced this pull request Oct 7, 2025
* Generate changelog
* cargo update
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancementNew feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@alamb@houqp