Uh oh!
There was an error while loading. Please reload this page.
Add heap memory estimation for statistics - #19599
Conversation
| pub fn heap_size(&self) -> usize { | ||
| // column_statistics + num_rows + total_byte_size | ||
| self.column_statistics.capacity() * size_of::<ColumnStatistics>() | ||
| + size_of::<Precision<usize>>() * 2 |
There was a problem hiding this comment.
Here Precision<usize> is an enum and does not have a heap allocated fields, so it is allocated in the stack.
There was a problem hiding this comment.
I think these things are usually Arc'ed - so everything should be moved to the heap, right?
There was a problem hiding this comment.
So size_of::<Precision<usize>>() * 2 should be removed?
There was a problem hiding this comment.
I think so, if we want to follow the trait in arrow, which I think according to #19599 (comment) was the conclusion of the next step? Do you plan on push a commit to do so?
There was a problem hiding this comment.
yes i am on it. pr coming up soon.
There was a problem hiding this comment.
We need to be able to get the heap size of arrays to implement it for Statistics? What's the chain of fields that takes us there?
There was a problem hiding this comment.
https://github.com/apache/datafusion/blob/main/datafusion/common/src/scalar/mod.rs#L376 as part of ScalarValue.
There was a problem hiding this comment.
And these are part of ColumnStatistics https://github.com/apache/datafusion/blob/main/datafusion/common/src/scalar/mod.rs#L376
There was a problem hiding this comment.
Bummer. Isn't there ways to get the size of an array in memory? E.g. Array::get_array_memory_size?
There was a problem hiding this comment.
This may actually work. Thanks, I will try that.
Uh oh!
There was an error while loading. Please reload this page.
ebef154 to
107caccCompareThis adds a heap_size method returning the amount of memory a statistics struct allocates on the heap.
17a4cd1 to
153d1adCompare| fn heap_size(&self) -> usize { | ||
| self.num_rows.heap_size() | ||
| + self.total_byte_size.heap_size() | ||
| + self |
There was a problem hiding this comment.
num_rows and total_byte_size will result in 0, so this is included for consistency, but could also be omitted.
There was a problem hiding this comment.
I think as defined in arrow-rs we're supposed to not include these and the caller should include them using size_off::<Statistics>(). IMO it would be nice if HeapSize had a memory_size() method that did self.heap_size() + size_of::<Self> or something like that.
adriangb
commented
Jan 12, 2026
@mkleen correct me if I'm missing something but the arrow-rs trait is currently under the That shouldn't block this PR though. I plan on reviewing it again this week (although it's a busy customer site visit week for me). I do think we'll want to get input from @alamb on this since it's turned into a bit bigger of a piece than I think any of us originally thought it would be (mostly in a good way though 😄). |
Yes, this is correct and this all makes sense to me.
No worries, thanks a lot!
Perfect! |
alamb
commented
Jan 12, 2026
Here is another suggestion: |
| fn heap_size(&self) -> usize; | ||
| } | ||
| impl HeapSize for Statistics { |
There was a problem hiding this comment.
Well, it does makes sense for the primitives of the arrow to have the implementation here, I think we can move the HeapSize for Statistics to the cache folder? Wdyt @mkleen
There was a problem hiding this comment.
Yes, this makes sense. This now became part of #20047
mkleen
commented
Jan 31, 2026
@adriangb Thanks for the feedback. What are your thoughts on @alchemist51’s suggestion to move this into the cache folder, since it will only be used there? |
adriangb
commented
Jan 31, 2026
If that keeps it out of the public API at least for now I think that’s an excellent suggestion. |
mkleen
commented
Feb 10, 2026
I will close this one, because it became part of #20047 |
Which issue does this PR close?
Relates to #19052 (comment)
Rationale for this change
This adds heap memory estimation to statistics.
What changes are included in this PR?
NA.
Are these changes tested?
Yes
Are there any user-facing changes?
Adds a new
HeapSizetrait and implementations for all relevant types used in memory estimation. The trait is taken from arrow-rs, where it is currently private, and is intended as a temporary solution until arrow-rs is updated.