DM-44319: Refactor deblending in DetectAndMeasureTask by fred3m · Pull Request #316 · lsst/ip_diffim

fred3m · 2024-05-17T15:14:28Z

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable.

This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

parejkoj

One concern is that downstream processing will have to filter on something to remove the "multiple parent copies". I don't know if we should also do that on this ticket, but we maybe should. I think that would be in FilterDiaSourceCatalogTask in ap_assocation. I don't know if we need to do it elsewhere, too; having both parents and children in the diaSource catalogs is something we're not fully familiar with yet.

parejkoj · 2024-05-17T18:51:48Z

+ merged_footprints.makeSources(sources)
+
+ # Sky sources must be added before deblending, otherwise the
+ # sources with parent == 0 will be out of order and SFM cannot run.


We're not really running "SFM" here; maybe this instead sources with parent == 0 will be out of order and measurement will fail.?

I'll change to "measurement tasks." The point is that measurement plugins require all of the children with the same parent ID to be contiguous, but the old version of this task added the sky sources after deblending, so if deblending had worked as expected then measurement would fail.

I am a bit surprised by that, because I think that deblending was working on "isolated" positive blends before (it was just messing up other things), and we didn't see failures.

Yeah, it's actually because before it wasn't doing any deblending, so all of the sources had parent==0. The measurement framework has a call to SourceCatalog.getChildren(0), where the argument is the parent ID. This is done to run all of the measurements on the parents, but in the new catalog if the sky objects were appended to it then the records with parent == 0 was discontinuous.

parejkoj · 2024-05-17T19:04:58Z

+
+ # Update the deblended parents and add their children
+ # to the sources catalog.
+ sources.extend(temp_cat[first_child_index:], deep=False)


I think you want to do a deep copy here to ensure the catalog is contiguous, but maybe that's not necessary for the measurement tasks that are run in after this? And then the catalog is written to disk. I guess we can see if we get failures when running on larger data?

No, we actually don't want a deep copy (see Jim's comments in slack). The temporary catalog actually created a view to the original records, that way the source IDs stay in order and the original records are updated in place.

I checked and verified that using this procedure the sources are always contiguous.

parejkoj · 2024-05-17T19:07:09Z

+ # Set detection and primary flags
+ for sid in skipped_ids:
+ src = sources.find(sid)
+ self.deblend.skipParent(src, difference.mask)
+ self.setPrimaryFlags.run(sources)


I understand what setPrimaryFlags is doing here, but I'm not familiar with skipParent. Could you please expand the comment above, since I don't think we're setting any "flags" here, but rather the "not deblended" mask plane (I think?)?

Sure. The basic idea is that the deblender already has a method to set all of the appropriate flags for a parent source when deblending is skipped, so we just call that method to set those flags here.

I think calling it "detection" as the comment does isn't quite correct, though, since we've already done detection. Maybe Set blending-related mask planes and flags on the non-deblended parents. would be more clear?

Yes, see my latest commit. But for the setPrimary section I still use this comment because technically all of the flags set by isPrimary are detect_<flagName>, since they are about detection.

parejkoj · 2024-05-17T20:57:33Z

Couple more comments about comments, otherwise this looks good, thank you

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable. This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

This reverts commit 9621bc4, reversing changes made to db0af17.

Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m · 2024-05-17T15:14:28Z

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable.

This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

parejkoj

One concern is that downstream processing will have to filter on something to remove the "multiple parent copies". I don't know if we should also do that on this ticket, but we maybe should. I think that would be in FilterDiaSourceCatalogTask in ap_assocation. I don't know if we need to do it elsewhere, too; having both parents and children in the diaSource catalogs is something we're not fully familiar with yet.

parejkoj · 2024-05-17T18:51:48Z

+ merged_footprints.makeSources(sources)
+
+ # Sky sources must be added before deblending, otherwise the
+ # sources with parent == 0 will be out of order and SFM cannot run.


We're not really running "SFM" here; maybe this instead sources with parent == 0 will be out of order and measurement will fail.?

I'll change to "measurement tasks." The point is that measurement plugins require all of the children with the same parent ID to be contiguous, but the old version of this task added the sky sources after deblending, so if deblending had worked as expected then measurement would fail.

I am a bit surprised by that, because I think that deblending was working on "isolated" positive blends before (it was just messing up other things), and we didn't see failures.

Yeah, it's actually because before it wasn't doing any deblending, so all of the sources had parent==0. The measurement framework has a call to SourceCatalog.getChildren(0), where the argument is the parent ID. This is done to run all of the measurements on the parents, but in the new catalog if the sky objects were appended to it then the records with parent == 0 was discontinuous.

parejkoj · 2024-05-17T19:04:58Z

+
+ # Update the deblended parents and add their children
+ # to the sources catalog.
+ sources.extend(temp_cat[first_child_index:], deep=False)


I think you want to do a deep copy here to ensure the catalog is contiguous, but maybe that's not necessary for the measurement tasks that are run in after this? And then the catalog is written to disk. I guess we can see if we get failures when running on larger data?

No, we actually don't want a deep copy (see Jim's comments in slack). The temporary catalog actually created a view to the original records, that way the source IDs stay in order and the original records are updated in place.

I checked and verified that using this procedure the sources are always contiguous.

parejkoj · 2024-05-17T19:07:09Z

+ # Set detection and primary flags
+ for sid in skipped_ids:
+ src = sources.find(sid)
+ self.deblend.skipParent(src, difference.mask)
+ self.setPrimaryFlags.run(sources)


I understand what setPrimaryFlags is doing here, but I'm not familiar with skipParent. Could you please expand the comment above, since I don't think we're setting any "flags" here, but rather the "not deblended" mask plane (I think?)?

Sure. The basic idea is that the deblender already has a method to set all of the appropriate flags for a parent source when deblending is skipped, so we just call that method to set those flags here.

I think calling it "detection" as the comment does isn't quite correct, though, since we've already done detection. Maybe Set blending-related mask planes and flags on the non-deblended parents. would be more clear?

Yes, see my latest commit. But for the setPrimary section I still use this comment because technically all of the flags set by isPrimary are detect_<flagName>, since they are about detection.

parejkoj · 2024-05-17T20:57:33Z

Couple more comments about comments, otherwise this looks good, thank you

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable. This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

This reverts commit 9621bc4, reversing changes made to db0af17.

Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m · 2024-05-17T15:14:28Z

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable.

This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

parejkoj

One concern is that downstream processing will have to filter on something to remove the "multiple parent copies". I don't know if we should also do that on this ticket, but we maybe should. I think that would be in FilterDiaSourceCatalogTask in ap_assocation. I don't know if we need to do it elsewhere, too; having both parents and children in the diaSource catalogs is something we're not fully familiar with yet.

parejkoj · 2024-05-17T18:51:48Z

+ merged_footprints.makeSources(sources)
+
+ # Sky sources must be added before deblending, otherwise the
+ # sources with parent == 0 will be out of order and SFM cannot run.


We're not really running "SFM" here; maybe this instead sources with parent == 0 will be out of order and measurement will fail.?

I'll change to "measurement tasks." The point is that measurement plugins require all of the children with the same parent ID to be contiguous, but the old version of this task added the sky sources after deblending, so if deblending had worked as expected then measurement would fail.

I am a bit surprised by that, because I think that deblending was working on "isolated" positive blends before (it was just messing up other things), and we didn't see failures.

Yeah, it's actually because before it wasn't doing any deblending, so all of the sources had parent==0. The measurement framework has a call to SourceCatalog.getChildren(0), where the argument is the parent ID. This is done to run all of the measurements on the parents, but in the new catalog if the sky objects were appended to it then the records with parent == 0 was discontinuous.

parejkoj · 2024-05-17T19:04:58Z

+
+ # Update the deblended parents and add their children
+ # to the sources catalog.
+ sources.extend(temp_cat[first_child_index:], deep=False)


I think you want to do a deep copy here to ensure the catalog is contiguous, but maybe that's not necessary for the measurement tasks that are run in after this? And then the catalog is written to disk. I guess we can see if we get failures when running on larger data?

No, we actually don't want a deep copy (see Jim's comments in slack). The temporary catalog actually created a view to the original records, that way the source IDs stay in order and the original records are updated in place.

I checked and verified that using this procedure the sources are always contiguous.

parejkoj · 2024-05-17T19:07:09Z

+ # Set detection and primary flags
+ for sid in skipped_ids:
+ src = sources.find(sid)
+ self.deblend.skipParent(src, difference.mask)
+ self.setPrimaryFlags.run(sources)


I understand what setPrimaryFlags is doing here, but I'm not familiar with skipParent. Could you please expand the comment above, since I don't think we're setting any "flags" here, but rather the "not deblended" mask plane (I think?)?

Sure. The basic idea is that the deblender already has a method to set all of the appropriate flags for a parent source when deblending is skipped, so we just call that method to set those flags here.

I think calling it "detection" as the comment does isn't quite correct, though, since we've already done detection. Maybe Set blending-related mask planes and flags on the non-deblended parents. would be more clear?

Yes, see my latest commit. But for the setPrimary section I still use this comment because technically all of the flags set by isPrimary are detect_<flagName>, since they are about detection.

parejkoj · 2024-05-17T20:57:33Z

Couple more comments about comments, otherwise this looks good, thank you

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable. This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

This reverts commit 9621bc4, reversing changes made to db0af17.

Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m · 2024-05-17T15:14:28Z

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable.

This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

parejkoj

One concern is that downstream processing will have to filter on something to remove the "multiple parent copies". I don't know if we should also do that on this ticket, but we maybe should. I think that would be in FilterDiaSourceCatalogTask in ap_assocation. I don't know if we need to do it elsewhere, too; having both parents and children in the diaSource catalogs is something we're not fully familiar with yet.

parejkoj · 2024-05-17T18:51:48Z

+ merged_footprints.makeSources(sources)
+
+ # Sky sources must be added before deblending, otherwise the
+ # sources with parent == 0 will be out of order and SFM cannot run.


We're not really running "SFM" here; maybe this instead sources with parent == 0 will be out of order and measurement will fail.?

I'll change to "measurement tasks." The point is that measurement plugins require all of the children with the same parent ID to be contiguous, but the old version of this task added the sky sources after deblending, so if deblending had worked as expected then measurement would fail.

I am a bit surprised by that, because I think that deblending was working on "isolated" positive blends before (it was just messing up other things), and we didn't see failures.

Yeah, it's actually because before it wasn't doing any deblending, so all of the sources had parent==0. The measurement framework has a call to SourceCatalog.getChildren(0), where the argument is the parent ID. This is done to run all of the measurements on the parents, but in the new catalog if the sky objects were appended to it then the records with parent == 0 was discontinuous.

parejkoj · 2024-05-17T19:04:58Z

+
+ # Update the deblended parents and add their children
+ # to the sources catalog.
+ sources.extend(temp_cat[first_child_index:], deep=False)


I think you want to do a deep copy here to ensure the catalog is contiguous, but maybe that's not necessary for the measurement tasks that are run in after this? And then the catalog is written to disk. I guess we can see if we get failures when running on larger data?

No, we actually don't want a deep copy (see Jim's comments in slack). The temporary catalog actually created a view to the original records, that way the source IDs stay in order and the original records are updated in place.

I checked and verified that using this procedure the sources are always contiguous.

parejkoj · 2024-05-17T19:07:09Z

+ # Set detection and primary flags
+ for sid in skipped_ids:
+ src = sources.find(sid)
+ self.deblend.skipParent(src, difference.mask)
+ self.setPrimaryFlags.run(sources)


I understand what setPrimaryFlags is doing here, but I'm not familiar with skipParent. Could you please expand the comment above, since I don't think we're setting any "flags" here, but rather the "not deblended" mask plane (I think?)?

Sure. The basic idea is that the deblender already has a method to set all of the appropriate flags for a parent source when deblending is skipped, so we just call that method to set those flags here.

I think calling it "detection" as the comment does isn't quite correct, though, since we've already done detection. Maybe Set blending-related mask planes and flags on the non-deblended parents. would be more clear?

Yes, see my latest commit. But for the setPrimary section I still use this comment because technically all of the flags set by isPrimary are detect_<flagName>, since they are about detection.

parejkoj · 2024-05-17T20:57:33Z

Couple more comments about comments, otherwise this looks good, thank you

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable. This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

This reverts commit 9621bc4, reversing changes made to db0af17.

Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m · 2024-05-17T15:14:28Z

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable.

This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

parejkoj

One concern is that downstream processing will have to filter on something to remove the "multiple parent copies". I don't know if we should also do that on this ticket, but we maybe should. I think that would be in FilterDiaSourceCatalogTask in ap_assocation. I don't know if we need to do it elsewhere, too; having both parents and children in the diaSource catalogs is something we're not fully familiar with yet.

parejkoj · 2024-05-17T18:51:48Z

+ merged_footprints.makeSources(sources)
+
+ # Sky sources must be added before deblending, otherwise the
+ # sources with parent == 0 will be out of order and SFM cannot run.


We're not really running "SFM" here; maybe this instead sources with parent == 0 will be out of order and measurement will fail.?

I'll change to "measurement tasks." The point is that measurement plugins require all of the children with the same parent ID to be contiguous, but the old version of this task added the sky sources after deblending, so if deblending had worked as expected then measurement would fail.

I am a bit surprised by that, because I think that deblending was working on "isolated" positive blends before (it was just messing up other things), and we didn't see failures.

Yeah, it's actually because before it wasn't doing any deblending, so all of the sources had parent==0. The measurement framework has a call to SourceCatalog.getChildren(0), where the argument is the parent ID. This is done to run all of the measurements on the parents, but in the new catalog if the sky objects were appended to it then the records with parent == 0 was discontinuous.

parejkoj · 2024-05-17T19:04:58Z

+
+ # Update the deblended parents and add their children
+ # to the sources catalog.
+ sources.extend(temp_cat[first_child_index:], deep=False)


I think you want to do a deep copy here to ensure the catalog is contiguous, but maybe that's not necessary for the measurement tasks that are run in after this? And then the catalog is written to disk. I guess we can see if we get failures when running on larger data?

No, we actually don't want a deep copy (see Jim's comments in slack). The temporary catalog actually created a view to the original records, that way the source IDs stay in order and the original records are updated in place.

I checked and verified that using this procedure the sources are always contiguous.

parejkoj · 2024-05-17T19:07:09Z

+ # Set detection and primary flags
+ for sid in skipped_ids:
+ src = sources.find(sid)
+ self.deblend.skipParent(src, difference.mask)
+ self.setPrimaryFlags.run(sources)


I understand what setPrimaryFlags is doing here, but I'm not familiar with skipParent. Could you please expand the comment above, since I don't think we're setting any "flags" here, but rather the "not deblended" mask plane (I think?)?

Sure. The basic idea is that the deblender already has a method to set all of the appropriate flags for a parent source when deblending is skipped, so we just call that method to set those flags here.

I think calling it "detection" as the comment does isn't quite correct, though, since we've already done detection. Maybe Set blending-related mask planes and flags on the non-deblended parents. would be more clear?

Yes, see my latest commit. But for the setPrimary section I still use this comment because technically all of the flags set by isPrimary are detect_<flagName>, since they are about detection.

parejkoj · 2024-05-17T20:57:33Z

Couple more comments about comments, otherwise this looks good, thank you

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable. This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

This reverts commit 9621bc4, reversing changes made to db0af17.

Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m · 2024-05-17T15:14:28Z

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable.

This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

parejkoj

One concern is that downstream processing will have to filter on something to remove the "multiple parent copies". I don't know if we should also do that on this ticket, but we maybe should. I think that would be in FilterDiaSourceCatalogTask in ap_assocation. I don't know if we need to do it elsewhere, too; having both parents and children in the diaSource catalogs is something we're not fully familiar with yet.

parejkoj · 2024-05-17T18:51:48Z

+ merged_footprints.makeSources(sources)
+
+ # Sky sources must be added before deblending, otherwise the
+ # sources with parent == 0 will be out of order and SFM cannot run.


We're not really running "SFM" here; maybe this instead sources with parent == 0 will be out of order and measurement will fail.?

I'll change to "measurement tasks." The point is that measurement plugins require all of the children with the same parent ID to be contiguous, but the old version of this task added the sky sources after deblending, so if deblending had worked as expected then measurement would fail.

I am a bit surprised by that, because I think that deblending was working on "isolated" positive blends before (it was just messing up other things), and we didn't see failures.

Yeah, it's actually because before it wasn't doing any deblending, so all of the sources had parent==0. The measurement framework has a call to SourceCatalog.getChildren(0), where the argument is the parent ID. This is done to run all of the measurements on the parents, but in the new catalog if the sky objects were appended to it then the records with parent == 0 was discontinuous.

parejkoj · 2024-05-17T19:04:58Z

+
+ # Update the deblended parents and add their children
+ # to the sources catalog.
+ sources.extend(temp_cat[first_child_index:], deep=False)


I think you want to do a deep copy here to ensure the catalog is contiguous, but maybe that's not necessary for the measurement tasks that are run in after this? And then the catalog is written to disk. I guess we can see if we get failures when running on larger data?

No, we actually don't want a deep copy (see Jim's comments in slack). The temporary catalog actually created a view to the original records, that way the source IDs stay in order and the original records are updated in place.

I checked and verified that using this procedure the sources are always contiguous.

parejkoj · 2024-05-17T19:07:09Z

+ # Set detection and primary flags
+ for sid in skipped_ids:
+ src = sources.find(sid)
+ self.deblend.skipParent(src, difference.mask)
+ self.setPrimaryFlags.run(sources)


I understand what setPrimaryFlags is doing here, but I'm not familiar with skipParent. Could you please expand the comment above, since I don't think we're setting any "flags" here, but rather the "not deblended" mask plane (I think?)?

Sure. The basic idea is that the deblender already has a method to set all of the appropriate flags for a parent source when deblending is skipped, so we just call that method to set those flags here.

I think calling it "detection" as the comment does isn't quite correct, though, since we've already done detection. Maybe Set blending-related mask planes and flags on the non-deblended parents. would be more clear?

Yes, see my latest commit. But for the setPrimary section I still use this comment because technically all of the flags set by isPrimary are detect_<flagName>, since they are about detection.

parejkoj · 2024-05-17T20:57:33Z

Couple more comments about comments, otherwise this looks good, thank you

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable. This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

This reverts commit 9621bc4, reversing changes made to db0af17.

Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m · 2024-05-17T15:14:28Z

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable.

This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

parejkoj

One concern is that downstream processing will have to filter on something to remove the "multiple parent copies". I don't know if we should also do that on this ticket, but we maybe should. I think that would be in FilterDiaSourceCatalogTask in ap_assocation. I don't know if we need to do it elsewhere, too; having both parents and children in the diaSource catalogs is something we're not fully familiar with yet.

parejkoj · 2024-05-17T18:51:48Z

+ merged_footprints.makeSources(sources)
+
+ # Sky sources must be added before deblending, otherwise the
+ # sources with parent == 0 will be out of order and SFM cannot run.


We're not really running "SFM" here; maybe this instead sources with parent == 0 will be out of order and measurement will fail.?

I'll change to "measurement tasks." The point is that measurement plugins require all of the children with the same parent ID to be contiguous, but the old version of this task added the sky sources after deblending, so if deblending had worked as expected then measurement would fail.

I am a bit surprised by that, because I think that deblending was working on "isolated" positive blends before (it was just messing up other things), and we didn't see failures.

Yeah, it's actually because before it wasn't doing any deblending, so all of the sources had parent==0. The measurement framework has a call to SourceCatalog.getChildren(0), where the argument is the parent ID. This is done to run all of the measurements on the parents, but in the new catalog if the sky objects were appended to it then the records with parent == 0 was discontinuous.

parejkoj · 2024-05-17T19:04:58Z

+
+ # Update the deblended parents and add their children
+ # to the sources catalog.
+ sources.extend(temp_cat[first_child_index:], deep=False)


I think you want to do a deep copy here to ensure the catalog is contiguous, but maybe that's not necessary for the measurement tasks that are run in after this? And then the catalog is written to disk. I guess we can see if we get failures when running on larger data?

No, we actually don't want a deep copy (see Jim's comments in slack). The temporary catalog actually created a view to the original records, that way the source IDs stay in order and the original records are updated in place.

I checked and verified that using this procedure the sources are always contiguous.

parejkoj · 2024-05-17T19:07:09Z

+ # Set detection and primary flags
+ for sid in skipped_ids:
+ src = sources.find(sid)
+ self.deblend.skipParent(src, difference.mask)
+ self.setPrimaryFlags.run(sources)


I understand what setPrimaryFlags is doing here, but I'm not familiar with skipParent. Could you please expand the comment above, since I don't think we're setting any "flags" here, but rather the "not deblended" mask plane (I think?)?

Sure. The basic idea is that the deblender already has a method to set all of the appropriate flags for a parent source when deblending is skipped, so we just call that method to set those flags here.

I think calling it "detection" as the comment does isn't quite correct, though, since we've already done detection. Maybe Set blending-related mask planes and flags on the non-deblended parents. would be more clear?

Yes, see my latest commit. But for the setPrimary section I still use this comment because technically all of the flags set by isPrimary are detect_<flagName>, since they are about detection.

parejkoj · 2024-05-17T20:57:33Z

Couple more comments about comments, otherwise this looks good, thank you

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable. This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

This reverts commit 9621bc4, reversing changes made to db0af17.

Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m · 2024-05-17T15:14:28Z

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable.

This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

parejkoj

One concern is that downstream processing will have to filter on something to remove the "multiple parent copies". I don't know if we should also do that on this ticket, but we maybe should. I think that would be in FilterDiaSourceCatalogTask in ap_assocation. I don't know if we need to do it elsewhere, too; having both parents and children in the diaSource catalogs is something we're not fully familiar with yet.

parejkoj · 2024-05-17T18:51:48Z

+ merged_footprints.makeSources(sources)
+
+ # Sky sources must be added before deblending, otherwise the
+ # sources with parent == 0 will be out of order and SFM cannot run.


We're not really running "SFM" here; maybe this instead sources with parent == 0 will be out of order and measurement will fail.?

I'll change to "measurement tasks." The point is that measurement plugins require all of the children with the same parent ID to be contiguous, but the old version of this task added the sky sources after deblending, so if deblending had worked as expected then measurement would fail.

I am a bit surprised by that, because I think that deblending was working on "isolated" positive blends before (it was just messing up other things), and we didn't see failures.

Yeah, it's actually because before it wasn't doing any deblending, so all of the sources had parent==0. The measurement framework has a call to SourceCatalog.getChildren(0), where the argument is the parent ID. This is done to run all of the measurements on the parents, but in the new catalog if the sky objects were appended to it then the records with parent == 0 was discontinuous.

parejkoj · 2024-05-17T19:04:58Z

+
+ # Update the deblended parents and add their children
+ # to the sources catalog.
+ sources.extend(temp_cat[first_child_index:], deep=False)


I think you want to do a deep copy here to ensure the catalog is contiguous, but maybe that's not necessary for the measurement tasks that are run in after this? And then the catalog is written to disk. I guess we can see if we get failures when running on larger data?

No, we actually don't want a deep copy (see Jim's comments in slack). The temporary catalog actually created a view to the original records, that way the source IDs stay in order and the original records are updated in place.

I checked and verified that using this procedure the sources are always contiguous.

parejkoj · 2024-05-17T19:07:09Z

+ # Set detection and primary flags
+ for sid in skipped_ids:
+ src = sources.find(sid)
+ self.deblend.skipParent(src, difference.mask)
+ self.setPrimaryFlags.run(sources)


I understand what setPrimaryFlags is doing here, but I'm not familiar with skipParent. Could you please expand the comment above, since I don't think we're setting any "flags" here, but rather the "not deblended" mask plane (I think?)?

Sure. The basic idea is that the deblender already has a method to set all of the appropriate flags for a parent source when deblending is skipped, so we just call that method to set those flags here.

I think calling it "detection" as the comment does isn't quite correct, though, since we've already done detection. Maybe Set blending-related mask planes and flags on the non-deblended parents. would be more clear?

Yes, see my latest commit. But for the setPrimary section I still use this comment because technically all of the flags set by isPrimary are detect_<flagName>, since they are about detection.

parejkoj · 2024-05-17T20:57:33Z

Couple more comments about comments, otherwise this looks good, thank you

The previous deblending algorith was not properly deblending, as it was attempting to use meas_deblender for use cases that were not algorithmically suitable. This commit only runs the deblender on blends that have all positive peaks, the regime that it is designed for.

This reverts commit 9621bc4, reversing changes made to db0af17.

Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m force-pushed the tickets/DM-44319 branch 2 times, most recently from e481cd7 to 8af121fCompare May 17, 2024 15:26

parejkoj reviewed May 17, 2024
View reviewed changes

fred3m force-pushed the tickets/DM-44319 branch from 3c211c0 to c1145ecCompare May 17, 2024 23:40

fred3m force-pushed the tickets/DM-44319 branch from c1145ec to 145f463Compare May 20, 2024 13:53

fred3m merged commit 9621bc4 into mainMay 20, 2024

fred3m deleted the tickets/DM-44319 branch May 20, 2024 16:09

fred3m added a commit that referenced this pull request May 21, 2024
Revert "Merge pull request#316from lsst/tickets/DM-44319"
9638b5c
This reverts commit 9621bc4, reversing changes made to db0af17.

fred3m added a commit that referenced this pull request May 21, 2024
Merge pull request#317from lsst/tickets/DM-44319-revert
ab192c5
Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m force-pushed the tickets/DM-44319 branch 2 times, most recently from e481cd7 to 8af121fCompare May 17, 2024 15:26

parejkoj reviewed May 17, 2024
View reviewed changes

fred3m force-pushed the tickets/DM-44319 branch from 3c211c0 to c1145ecCompare May 17, 2024 23:40

fred3m force-pushed the tickets/DM-44319 branch from c1145ec to 145f463Compare May 20, 2024 13:53

fred3m merged commit 9621bc4 into mainMay 20, 2024

fred3m deleted the tickets/DM-44319 branch May 20, 2024 16:09

fred3m added a commit that referenced this pull request May 21, 2024
Revert "Merge pull request#316from lsst/tickets/DM-44319"
9638b5c
This reverts commit 9621bc4, reversing changes made to db0af17.

fred3m added a commit that referenced this pull request May 21, 2024
Merge pull request#317from lsst/tickets/DM-44319-revert
ab192c5
Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m force-pushed the tickets/DM-44319 branch 2 times, most recently from e481cd7 to 8af121fCompare May 17, 2024 15:26

parejkoj reviewed May 17, 2024
View reviewed changes

fred3m force-pushed the tickets/DM-44319 branch from 3c211c0 to c1145ecCompare May 17, 2024 23:40

fred3m force-pushed the tickets/DM-44319 branch from c1145ec to 145f463Compare May 20, 2024 13:53

fred3m merged commit 9621bc4 into mainMay 20, 2024

fred3m deleted the tickets/DM-44319 branch May 20, 2024 16:09

fred3m added a commit that referenced this pull request May 21, 2024
Revert "Merge pull request#316from lsst/tickets/DM-44319"
9638b5c
This reverts commit 9621bc4, reversing changes made to db0af17.

fred3m added a commit that referenced this pull request May 21, 2024
Merge pull request#317from lsst/tickets/DM-44319-revert
ab192c5
Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m force-pushed the tickets/DM-44319 branch 2 times, most recently from e481cd7 to 8af121fCompare May 17, 2024 15:26

parejkoj reviewed May 17, 2024
View reviewed changes

fred3m force-pushed the tickets/DM-44319 branch from 3c211c0 to c1145ecCompare May 17, 2024 23:40

fred3m force-pushed the tickets/DM-44319 branch from c1145ec to 145f463Compare May 20, 2024 13:53

fred3m merged commit 9621bc4 into mainMay 20, 2024

fred3m deleted the tickets/DM-44319 branch May 20, 2024 16:09

fred3m added a commit that referenced this pull request May 21, 2024
Revert "Merge pull request#316from lsst/tickets/DM-44319"
9638b5c
This reverts commit 9621bc4, reversing changes made to db0af17.

fred3m added a commit that referenced this pull request May 21, 2024
Merge pull request#317from lsst/tickets/DM-44319-revert
ab192c5
Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m force-pushed the tickets/DM-44319 branch 2 times, most recently from e481cd7 to 8af121fCompare May 17, 2024 15:26

parejkoj reviewed May 17, 2024
View reviewed changes

fred3m force-pushed the tickets/DM-44319 branch from 3c211c0 to c1145ecCompare May 17, 2024 23:40

fred3m force-pushed the tickets/DM-44319 branch from c1145ec to 145f463Compare May 20, 2024 13:53

fred3m merged commit 9621bc4 into mainMay 20, 2024

fred3m deleted the tickets/DM-44319 branch May 20, 2024 16:09

fred3m added a commit that referenced this pull request May 21, 2024
Revert "Merge pull request#316from lsst/tickets/DM-44319"
9638b5c
This reverts commit 9621bc4, reversing changes made to db0af17.

fred3m added a commit that referenced this pull request May 21, 2024
Merge pull request#317from lsst/tickets/DM-44319-revert
ab192c5
Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m force-pushed the tickets/DM-44319 branch 2 times, most recently from e481cd7 to 8af121fCompare May 17, 2024 15:26

parejkoj reviewed May 17, 2024
View reviewed changes

fred3m force-pushed the tickets/DM-44319 branch from 3c211c0 to c1145ecCompare May 17, 2024 23:40

fred3m force-pushed the tickets/DM-44319 branch from c1145ec to 145f463Compare May 20, 2024 13:53

fred3m merged commit 9621bc4 into mainMay 20, 2024

fred3m deleted the tickets/DM-44319 branch May 20, 2024 16:09

fred3m added a commit that referenced this pull request May 21, 2024
Revert "Merge pull request#316from lsst/tickets/DM-44319"
9638b5c
This reverts commit 9621bc4, reversing changes made to db0af17.

fred3m added a commit that referenced this pull request May 21, 2024
Merge pull request#317from lsst/tickets/DM-44319-revert
ab192c5
Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m force-pushed the tickets/DM-44319 branch 2 times, most recently from e481cd7 to 8af121fCompare May 17, 2024 15:26

parejkoj reviewed May 17, 2024
View reviewed changes

fred3m force-pushed the tickets/DM-44319 branch from 3c211c0 to c1145ecCompare May 17, 2024 23:40

fred3m force-pushed the tickets/DM-44319 branch from c1145ec to 145f463Compare May 20, 2024 13:53

fred3m merged commit 9621bc4 into mainMay 20, 2024

fred3m deleted the tickets/DM-44319 branch May 20, 2024 16:09

fred3m added a commit that referenced this pull request May 21, 2024
Revert "Merge pull request#316from lsst/tickets/DM-44319"
9638b5c
This reverts commit 9621bc4, reversing changes made to db0af17.

fred3m added a commit that referenced this pull request May 21, 2024
Merge pull request#317from lsst/tickets/DM-44319-revert
ab192c5
Revert "Merge pull request #316 from lsst/tickets/DM-44319"

fred3m force-pushed the tickets/DM-44319 branch 2 times, most recently from e481cd7 to 8af121fCompare May 17, 2024 15:26

parejkoj reviewed May 17, 2024
View reviewed changes

fred3m force-pushed the tickets/DM-44319 branch from 3c211c0 to c1145ecCompare May 17, 2024 23:40

fred3m force-pushed the tickets/DM-44319 branch from c1145ec to 145f463Compare May 20, 2024 13:53

fred3m merged commit 9621bc4 into mainMay 20, 2024

fred3m deleted the tickets/DM-44319 branch May 20, 2024 16:09

fred3m added a commit that referenced this pull request May 21, 2024
Revert "Merge pull request#316from lsst/tickets/DM-44319"
9638b5c
This reverts commit 9621bc4, reversing changes made to db0af17.

fred3m added a commit that referenced this pull request May 21, 2024
Merge pull request#317from lsst/tickets/DM-44319-revert
ab192c5
Revert "Merge pull request #316 from lsst/tickets/DM-44319"

Uh oh!

Conversation

fred3m commented May 17, 2024

Uh oh!

parejkoj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

parejkoj commented May 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

fred3m commented May 17, 2024

Uh oh!

parejkoj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!