feat: Expose used MemoryPool details in ResourcesExhausted error messages

[erenavsarogullari]

Which issue does this PR close?

• ClosesExpose used MemoryPool details in ResourcesExhausted error messages #20386.

Rationale for this change

memory_limit (RuntimeEnvBuilder::new().with_memory_limit()) configuration uses greedy memory pool as default. However, if memory_pool (RuntimeEnvBuilder::new().with_memory_pool()) is set, it overrides by expected memory_pool config such as fair. Also, if both memory_limit and memory_pool configs are not set, unbounded memory pool will be used so it can be useful to expose ultimately used/selected pool as part of ResourcesExhausted error message for the end user awareness and the user may need to switch used memory pool (greedy, fair, unbounded),

• Also, this comparison table is an example use-case for both greedy and fair memory pools runtime behaviors and this addition can help for this kind of comparison table by exposing used memory pool info as part of native logs.

Please find following example use-cases by datafusion-cli:
Case1: datafusion-cli result when memory-limit and top-memory-consumers > 0 are set:

eren.avsarogullari@AWGNPWVK961 debug % ./datafusion-cli --memory-limit 10M --command 'select * from generate_series(1,500000) as t1(v1) order by v1;' --top-memory-consumers 3
DataFusion CLI v53.0.0
Error: Not enough memory to continue external sort. Consider increasing the memory limit config: 'datafusion.runtime.memory_limit', or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'.
caused by
Resources exhausted: Additional allocation failed for ExternalSorter[0] with top memory consumers (across reservations) as:
ExternalSorterMerge[0]#2(can spill: false) consumed 10.0 MB, peak 10.0 MB,
DataFusion-Cli#0(can spill: false) consumed 0.0 B, peak 0.0 B,
ExternalSorter[0]#1(can spill: true) consumed 0.0 B, peak 0.0 B.
Error: Failed to allocate additional 128.0 KB for ExternalSorter[0] with 0.0 B already allocated for this reservation - 0.0 B remain available for the total memory pool: greedy(used: 10.0 MB, pool_size: 10.0 MB)

Case2: datafusion-cli result when memory-limit and top-memory-consumers = 0 (disabling top memory consumers logging) are set:

eren.avsarogullari@AWGNPWVK961 debug % ./datafusion-cli --memory-limit 10M --command 'select * from generate_series(1,500000) as t1(v1) order by v1;' --top-memory-consumers 0
DataFusion CLI v53.0.0
Error: Not enough memory to continue external sort. Consider increasing the memory limit config: 'datafusion.runtime.memory_limit', or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'.
caused by
Resources exhausted: Failed to allocate additional 128.0 KB for ExternalSorter[0] with 0.0 B already allocated for this reservation - 0.0 B remain available for the total memory pool: greedy(used: 10.0 MB, pool_size: 10.0 MB)

Case3: datafusion-cli result when only memory-limit, memory-pool and top-memory-consumers > 0 are set:

eren.avsarogullari@AWGNPWVK961 debug % ./datafusion-cli --memory-limit 10M --mem-pool-type fair --top-memory-consumers 3 --command 'select * from generate_series(1,500000) as t1(v1) order by v1;'
DataFusion CLI v53.0.0
Error: Not enough memory to continue external sort. Consider increasing the memory limit config: 'datafusion.runtime.memory_limit', or decreasing the config: 'datafusion.execution.sort_spill_reservation_bytes'.
caused by
Resources exhausted: Additional allocation failed for ExternalSorter[0] with top memory consumers (across reservations) as:
ExternalSorterMerge[0]#2(can spill: false) consumed 10.0 MB, peak 10.0 MB,
ExternalSorter[0]#1(can spill: true) consumed 0.0 B, peak 0.0 B,
DataFusion-Cli#0(can spill: false) consumed 0.0 B, peak 0.0 B.
Error: Failed to allocate additional 128.0 KB for ExternalSorter[0] with 0.0 B already allocated for this reservation - 0.0 B remain available for the total memory pool: fair(pool_size: 10.0 MB)

What changes are included in this PR?

• Adding name property to MemoryPool instances,
• Expose used MemoryPool info to Resources Exhausted error messages

Are these changes tested?

Yes and updating existing test cases.

Are there any user-facing changes?

Yes, being updated Resources Exhausted error messages.

[nathanb9]

Nice. This makes sense to me

[xudong963]

do we need this format_args!?

[erenavsarogullari]

Fixed by the last commit. Thanks for the review.

[xudong963]

how about adding used in FairSpillPool Display

[erenavsarogullari]

Currently, both GreedyMemoryPool and UnboundedMemoryPool track used memory size. FairSpillPool can also be extended to track it. I think it can be useful extension. Does it make sense to address by a follow-up issue? If so, i can create and work on it.

[xudong963]

yes, good to me

[erenavsarogullari]

#21753 has been created for follow-up effort. Thanks for quick feedback.