Attempt to fix inter-volume trace pair functions by majosm · Pull Request #243 · inducer/grudge

majosm · 2022-03-25T00:27:51Z

I abandoned the previous one-way communication idea (I didn't know how to make this work in lazy, since there wouldn't be anything to staple the sends to). This PR is an attempt to implement a two-way version. Still untested.

Since the trace data is possibly heterogeneous in the multi-volume case, the sends in _RankBoundaryCommunicationLazy can't be stapled to the receives anymore (since the sent and received data on a given rank will in general have different numbers of subarrays). As a workaround, I staple the sends to arrays of zeros with the same shape as the data being sent. Then I reduce the resulting array container of zeros to a scalar zero value, and add it to the trace pair data being returned so that something (hopefully) will depend on it.

Depends on #239 (and inducer/meshmode#308).

inducer · 2022-03-28T21:17:50Z

+ return 0
+ else:
+ ary_tag = (comm_tag, key)
+ return staple_distributed_send(


Make the send separately from stapling it.

majosm · 2022-07-04T19:21:16Z

@inducer This is probably ready for a look too (though it still contains all of #239; I'll leave it as a draft until that gets merged.)

test failure was caused by bug in test (that has since been fixed)

majosm · 2022-09-26T15:02:08Z

Superseded by #280 and #285.

majosm · 2022-03-25T00:27:51Z

I abandoned the previous one-way communication idea (I didn't know how to make this work in lazy, since there wouldn't be anything to staple the sends to). This PR is an attempt to implement a two-way version. Still untested.

Since the trace data is possibly heterogeneous in the multi-volume case, the sends in _RankBoundaryCommunicationLazy can't be stapled to the receives anymore (since the sent and received data on a given rank will in general have different numbers of subarrays). As a workaround, I staple the sends to arrays of zeros with the same shape as the data being sent. Then I reduce the resulting array container of zeros to a scalar zero value, and add it to the trace pair data being returned so that something (hopefully) will depend on it.

Depends on #239 (and inducer/meshmode#308).

inducer · 2022-03-28T21:17:50Z

+ return 0
+ else:
+ ary_tag = (comm_tag, key)
+ return staple_distributed_send(


Make the send separately from stapling it.

majosm · 2022-07-04T19:21:16Z

@inducer This is probably ready for a look too (though it still contains all of #239; I'll leave it as a draft until that gets merged.)

test failure was caused by bug in test (that has since been fixed)

majosm · 2022-09-26T15:02:08Z

Superseded by #280 and #285.

majosm · 2022-03-25T00:27:51Z

I abandoned the previous one-way communication idea (I didn't know how to make this work in lazy, since there wouldn't be anything to staple the sends to). This PR is an attempt to implement a two-way version. Still untested.

Since the trace data is possibly heterogeneous in the multi-volume case, the sends in _RankBoundaryCommunicationLazy can't be stapled to the receives anymore (since the sent and received data on a given rank will in general have different numbers of subarrays). As a workaround, I staple the sends to arrays of zeros with the same shape as the data being sent. Then I reduce the resulting array container of zeros to a scalar zero value, and add it to the trace pair data being returned so that something (hopefully) will depend on it.

Depends on #239 (and inducer/meshmode#308).

inducer · 2022-03-28T21:17:50Z

+ return 0
+ else:
+ ary_tag = (comm_tag, key)
+ return staple_distributed_send(


Make the send separately from stapling it.

majosm · 2022-07-04T19:21:16Z

@inducer This is probably ready for a look too (though it still contains all of #239; I'll leave it as a draft until that gets merged.)

test failure was caused by bug in test (that has since been fixed)

majosm · 2022-09-26T15:02:08Z

Superseded by #280 and #285.

majosm · 2022-03-25T00:27:51Z

I abandoned the previous one-way communication idea (I didn't know how to make this work in lazy, since there wouldn't be anything to staple the sends to). This PR is an attempt to implement a two-way version. Still untested.

Since the trace data is possibly heterogeneous in the multi-volume case, the sends in _RankBoundaryCommunicationLazy can't be stapled to the receives anymore (since the sent and received data on a given rank will in general have different numbers of subarrays). As a workaround, I staple the sends to arrays of zeros with the same shape as the data being sent. Then I reduce the resulting array container of zeros to a scalar zero value, and add it to the trace pair data being returned so that something (hopefully) will depend on it.

Depends on #239 (and inducer/meshmode#308).

inducer · 2022-03-28T21:17:50Z

+ return 0
+ else:
+ ary_tag = (comm_tag, key)
+ return staple_distributed_send(


Make the send separately from stapling it.

majosm · 2022-07-04T19:21:16Z

@inducer This is probably ready for a look too (though it still contains all of #239; I'll leave it as a draft until that gets merged.)

test failure was caused by bug in test (that has since been fixed)

majosm · 2022-09-26T15:02:08Z

Superseded by #280 and #285.

majosm · 2022-03-25T00:27:51Z

I abandoned the previous one-way communication idea (I didn't know how to make this work in lazy, since there wouldn't be anything to staple the sends to). This PR is an attempt to implement a two-way version. Still untested.

Since the trace data is possibly heterogeneous in the multi-volume case, the sends in _RankBoundaryCommunicationLazy can't be stapled to the receives anymore (since the sent and received data on a given rank will in general have different numbers of subarrays). As a workaround, I staple the sends to arrays of zeros with the same shape as the data being sent. Then I reduce the resulting array container of zeros to a scalar zero value, and add it to the trace pair data being returned so that something (hopefully) will depend on it.

Depends on #239 (and inducer/meshmode#308).

inducer · 2022-03-28T21:17:50Z

+ return 0
+ else:
+ ary_tag = (comm_tag, key)
+ return staple_distributed_send(


Make the send separately from stapling it.

majosm · 2022-07-04T19:21:16Z

@inducer This is probably ready for a look too (though it still contains all of #239; I'll leave it as a draft until that gets merged.)

test failure was caused by bug in test (that has since been fixed)

majosm · 2022-09-26T15:02:08Z

Superseded by #280 and #285.

majosm · 2022-03-25T00:27:51Z

I abandoned the previous one-way communication idea (I didn't know how to make this work in lazy, since there wouldn't be anything to staple the sends to). This PR is an attempt to implement a two-way version. Still untested.

Since the trace data is possibly heterogeneous in the multi-volume case, the sends in _RankBoundaryCommunicationLazy can't be stapled to the receives anymore (since the sent and received data on a given rank will in general have different numbers of subarrays). As a workaround, I staple the sends to arrays of zeros with the same shape as the data being sent. Then I reduce the resulting array container of zeros to a scalar zero value, and add it to the trace pair data being returned so that something (hopefully) will depend on it.

Depends on #239 (and inducer/meshmode#308).

inducer · 2022-03-28T21:17:50Z

+ return 0
+ else:
+ ary_tag = (comm_tag, key)
+ return staple_distributed_send(


Make the send separately from stapling it.

majosm · 2022-07-04T19:21:16Z

@inducer This is probably ready for a look too (though it still contains all of #239; I'll leave it as a draft until that gets merged.)

test failure was caused by bug in test (that has since been fixed)

majosm · 2022-09-26T15:02:08Z

Superseded by #280 and #285.

majosm · 2022-03-25T00:27:51Z

I abandoned the previous one-way communication idea (I didn't know how to make this work in lazy, since there wouldn't be anything to staple the sends to). This PR is an attempt to implement a two-way version. Still untested.

Since the trace data is possibly heterogeneous in the multi-volume case, the sends in _RankBoundaryCommunicationLazy can't be stapled to the receives anymore (since the sent and received data on a given rank will in general have different numbers of subarrays). As a workaround, I staple the sends to arrays of zeros with the same shape as the data being sent. Then I reduce the resulting array container of zeros to a scalar zero value, and add it to the trace pair data being returned so that something (hopefully) will depend on it.

Depends on #239 (and inducer/meshmode#308).

inducer · 2022-03-28T21:17:50Z

+ return 0
+ else:
+ ary_tag = (comm_tag, key)
+ return staple_distributed_send(


Make the send separately from stapling it.

majosm · 2022-07-04T19:21:16Z

@inducer This is probably ready for a look too (though it still contains all of #239; I'll leave it as a draft until that gets merged.)

test failure was caused by bug in test (that has since been fixed)

majosm · 2022-09-26T15:02:08Z

Superseded by #280 and #285.

majosm · 2022-03-25T00:27:51Z

I abandoned the previous one-way communication idea (I didn't know how to make this work in lazy, since there wouldn't be anything to staple the sends to). This PR is an attempt to implement a two-way version. Still untested.

Since the trace data is possibly heterogeneous in the multi-volume case, the sends in _RankBoundaryCommunicationLazy can't be stapled to the receives anymore (since the sent and received data on a given rank will in general have different numbers of subarrays). As a workaround, I staple the sends to arrays of zeros with the same shape as the data being sent. Then I reduce the resulting array container of zeros to a scalar zero value, and add it to the trace pair data being returned so that something (hopefully) will depend on it.

Depends on #239 (and inducer/meshmode#308).

inducer · 2022-03-28T21:17:50Z

+ return 0
+ else:
+ ary_tag = (comm_tag, key)
+ return staple_distributed_send(


Make the send separately from stapling it.

majosm · 2022-07-04T19:21:16Z

@inducer This is probably ready for a look too (though it still contains all of #239; I'll leave it as a draft until that gets merged.)

test failure was caused by bug in test (that has since been fixed)

majosm · 2022-09-26T15:02:08Z

Superseded by #280 and #285.

majosm mentioned this pull request Mar 25, 2022
No map_distributed_send_ref_holder method in NUserCollectorinducer/pytato#288
Closed

majosm added 3 commits March 25, 2022 13:48

use partition-ID-enabled meshmode
b3de25e

temporarily change meshmode branch
ea570ee

fix docs
7cc98f6

majosm force-pushed the cross-rank-inter-volume-trace-pairs branch from 8c86c98 to 012341dCompare March 25, 2022 18:52

inducer reviewed Mar 28, 2022
View reviewed changes

majosm mentioned this pull request Apr 4, 2022
Multi-volume extras #246
Closed

Merge remote-tracking branch 'origin/multi-volume' into multi-volume
7ea9d73

majosm force-pushed the cross-rank-inter-volume-trace-pairs branch from 81b7b7e to 4c7292eCompare April 12, 2022 14:42

majosm force-pushed the cross-rank-inter-volume-trace-pairs branch from 97915bd to 0603005Compare May 2, 2022 18:23

majosm force-pushed the multi-volume branch from 161c110 to ec17d80Compare May 2, 2022 18:32

majosm force-pushed the cross-rank-inter-volume-trace-pairs branch from 0603005 to 730e0b4Compare May 2, 2022 18:34

majosm force-pushed the multi-volume branch from ec17d80 to 5ee00b4Compare May 2, 2022 19:00

Merge remote-tracking branch 'origin/multi-volume' into multi-volume
e705304

majosm force-pushed the cross-rank-inter-volume-trace-pairs branch 2 times, most recently from 16fe9e8 to f25fe19Compare May 4, 2022 20:40

Merge remote-tracking branch 'origin/multi-volume' into multi-volume
c1b68c7

majosm force-pushed the cross-rank-inter-volume-trace-pairs branch from f25fe19 to 92d0a36Compare May 23, 2022 15:41

Merge remote-tracking branch 'origin/multi-volume' into multi-volume
d072923

majosm force-pushed the cross-rank-inter-volume-trace-pairs branch from 92d0a36 to fb2cac5Compare June 21, 2022 19:38

majosm added 5 commits June 30, 2022 12:16

Merge remote-tracking branch 'origin/multi-volume' into multi-volume
fe54464

promote to part ID instead of using part ID helper
09bac09

clarify part vs. partition terminology
c46a6b5

account for explicit part_id attribute in InterPartAdjacencyGroup
06ff8f6

Merge remote-tracking branch 'origin/multi-volume' into multi-volume
d7cd9db

majosm force-pushed the cross-rank-inter-volume-trace-pairs branch from fb2cac5 to 0f94e3dCompare July 1, 2022 18:23

Merge remote-tracking branch 'origin/multi-volume' into multi-volume
4743538

majosm force-pushed the cross-rank-inter-volume-trace-pairs branch 2 times, most recently from 6ae2cca to 1c76f6eCompare July 4, 2022 15:15

use dataclass instead of tuple for PartID
49d627c

majosm added 3 commits July 4, 2022 12:31

move PartID conversion setup stuff into _normalize_mesh_part_ids
b2219d8

reset requirements.txt
a05c626

accept general integral types as MPI ranks
fb8a34b

majosm force-pushed the cross-rank-inter-volume-trace-pairs branch from 1c76f6e to fd644cfCompare July 4, 2022 18:27

majosm added 15 commits August 9, 2022 10:47

Merge remote-tracking branch 'origin/multi-volume' into multi-volume
6af8108

simplify partitioning in test
dce83fb

redesign inter-volume trace pair functions
9590cdc

reapply zero addition to local_bdry_data
9e0a95f
test failure was caused by bug in test (that has since been fixed)

handle all-Number cases
ef4a17c

cosmetic change
4cf60fa

fix bug
041cb92

tweak as_dofdesc normalization
cb3ebe1

fix bugs
fa17cd1

don't try to create trace pair if there's no adjacency
5aa6c1d

forget about heterogeneous inter-volume trace pairs for now
e34b30d

add FIXME
fb44700

fix bug
c3bb93c

ignore flake8-bugbear error
7ef21ba

fix bug
12a6443

majosm force-pushed the cross-rank-inter-volume-trace-pairs branch from fd644cf to 12a6443Compare August 9, 2022 21:14

majosm mentioned this pull request Aug 29, 2022
Multiple independent volumes #280
Merged

majosm mentioned this pull request Sep 23, 2022
Support coupled volumes #285
Draft
2 tasks

majosm closed this Sep 26, 2022