Skip to content

GH-3273: Add scoped chunk level statistics to avoid unbounded output - #3274

Merged
wgtmac merged 5 commits into
apache:masterfrom
ArnavBalyan:arnavb/cli-scope
Aug 31, 2025
Merged

GH-3273: Add scoped chunk level statistics to avoid unbounded output#3274
wgtmac merged 5 commits into
apache:masterfrom
ArnavBalyan:arnavb/cli-scope

Conversation

@ArnavBalyan

@ArnavBalyanArnavBalyan commented Aug 21, 2025•
edited
Loading

Copy link
Copy Markdown
Member
  • Size stats currently prints chunk-level size statistics for every row group and every column in the file. On wide schemas and/or large files, this can generate massive, output, making the command hard to use.
  • Add filtering flags to size-stats aligning with column-index:
    -c, --column: filter by column dot paths (e.g., a.b.c)
    -r, --row-group: filter by row-group indexes (0-based)
  • Fixes: Size stats produces unbounded output allow user to scope output #3273

@ArnavBalyan

ArnavBalyan commented Aug 22, 2025

Copy link
Copy Markdown
MemberAuthor

cc @wgtmac@shangxinli could you please review thanks!

1 similar comment
@ArnavBalyan

ArnavBalyan commented Aug 22, 2025

Copy link
Copy Markdown
MemberAuthor

cc @wgtmac@shangxinli could you please review thanks!

@ArnavBalyan

ArnavBalyan commented Aug 27, 2025

Copy link
Copy Markdown
MemberAuthor

cc @gszadovszky could you please help take a look thanks!

gszadovszky
gszadovszky reviewed Aug 27, 2025
View reviewed changes
Comment threadparquet-cli/src/main/java/org/apache/parquet/cli/commands/ShowSizeStatisticsCommand.java Outdated
for (int index = 0, n = rowGroups.size(); index < n; index++) {
printRowGroupSizeStats(console, index, rowGroups.get(index), schema);
List<BlockMetaData> blocks = footer.getBlocks();
Set<Integer> allowedRowGroups = rowGroups == null ? null : new HashSet<>(rowGroups);

@gszadovszkygszadovszkyAug 27, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: If you would create an empty Set instead of initializing with a null, you wouldn't have to check for null in the loop.

@ArnavBalyanArnavBalyanAug 28, 2025

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previous implementation was using map since we were tracking depth, updated to set thanks!

Comment threadparquet-cli/src/main/java/org/apache/parquet/cli/commands/ShowSizeStatisticsCommand.java Outdated
Comment on lines +103 to +106
Set<String> allowedColumns = null;
if (columns != null && !columns.isEmpty()) {
allowedColumns = new HashSet<>(columns);
}

@gszadovszkygszadovszkyAug 27, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous comment

@ArnavBalyanArnavBalyanAug 28, 2025

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

@wgtmac

wgtmac commented Aug 27, 2025

Copy link
Copy Markdown
Member

Could you please provide the cli output after your improvement? Thanks!

@ArnavBalyan

ArnavBalyan commented Aug 28, 2025•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

Could you please provide the cli output after your improvement? Thanks!

Here is the CLI output, it's the same as the previous CLI command's output just truncated based on the user's preference:

Row group 0
--------------------------------------------------------------------------------
column unencoded bytes rep level histogram def level histogram [int32_field] - - - [int64_field] - - - 
Row group 0
--------------------------------------------------------------------------------
column unencoded bytes rep level histogram def level histogram [int32_field] - - - [int64_field] - - - [float_field] - - - [double_field] - - - [binary_field] 46 B - - [flba_field] - - - [date_field] - - -

Thanks!

wgtmac
wgtmac approved these changes Aug 31, 2025
View reviewed changes

@wgtmacwgtmac left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ArnavBalyan!

@wgtmac
wgtmac merged commit 4b6fbc1 into apache:masterAug 31, 2025
7 checks passed
rahulketch pushed a commit to rahulketch/parquet-java that referenced this pull request Sep 9, 2025
@ArnavBalyan@rahulketch
apacheGH-3273: Add scoped chunk level statistics to avoid unbounded o…
b47811d
…utput (apache#3274)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wgtmacwgtmacwgtmac approved these changes
@gszadovszkygszadovszkygszadovszky approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Size stats produces unbounded output allow user to scope output

3 participants

@ArnavBalyan@wgtmac@gszadovszky