Uh oh!
There was an error while loading. Please reload this page.
Avoid replicated 3D boundary-flux recovery - #593
Open
gthyagi wants to merge 1 commit into
Open
Conversation
Gather coordinate-keyed 3D reactions and trace elements on rank zero, assemble and solve the global P1/P2 boundary-mass system once, and scatter only each rank's requested recovered values. Preserve existing recovery semantics while preallocating sparse COO arrays and propagating root-side failures collectively. This reduces the Zhong 1/64 geoid postprocessing peak from more than 576 GB to 301.77 GB on 192 ranks without changing the 1/32 coefficients.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix the MPI memory scaling of 3D boundary-flux recovery without changing its reaction, trace-mass, gauge-removal, or local-return mathematics.
Problem
The existing 3D path uses
MPI.allgatherfor nodal reactions and P1/P2 trace elements. Every rank therefore retains the complete global boundary mesh, assembles the same sparse surface-mass matrix, and runs the same consistent-mass solve. Memory grows with both global boundary size and rank count.This became the limiting post-processing stage for high-resolution spherical rotated-free-slip models: the Stokes solve completed, but consistent P2 normal-traction recovery exhausted aggregate job memory.
Change
xsorder;The behavior of
mass={"auto","lumped","consistent","p1"}, partial-reaction sum/overwrite semantics, mean removal, and P2 midpoint interpolation is unchanged.Validation
tests/test_1019_boundary_flux.py -k 3d: 3 passed.tests/parallel/test_1065_boundary_flux_parallel.py -k 3d: 2 passed on every rank.cellsize=1/64, 192-rank post-processing path in 7:29 with 301.77 GB peak memory; the replicated path had exhausted 576 GB.Scope
This PR contains only the general
boundary_fluxgather/solve/scatter correction and itsrotated_bcdocumentation update. Zhong geoid mathematics and benchmark-facing APIs remain isolated in #591.