Expand SFDQuery to include mask and other data products by andrew-saydjari · Pull Request #51 · gregreen/dustmaps

andrew-saydjari · 2024-05-05T01:51:14Z

This PR allows users to query the SFD bitmask as well as all other SFD style data products stored on the Harvard Dataverse under the same DOI. The bitmask is forced to use order 0 interpolation to avoid violating the bit quantization. It appears that the padding done to the data to try to handle distortions at the Galactic Plane was not as successfully applied to the mask data product, leading to some artifacts near b = 0. However, it appears that this is a faithful reproduction of the values in the FITS file and the WCS header and so I am not really in favor of changing it at this level. A lower level change to better handle the distortions etc. would be preferable. In total, I think making these data products, especially the bitmask, is extremely valuable.

andrew-saydjari · 2024-05-05T01:52:12Z

I should say, I did not make any effort to enable this functionality from SFDWebQuery.

andrew-saydjari · 2024-05-05T01:53:01Z

Examples of the issues with the lack of mask padding near b = 0.

gregreen · 2024-07-10T14:03:45Z

+ whichparent='SFD',
+ which='dust'


Can we come up with more intuitive names for these keyword arguments? Maybe map_variant instead of whichparent, and component instead of which? Additionally, is it possible to infer whichparent from which? If so, then one option will suffice.

Happy to make the variable name change. It is true that which is unique, and specifies whichparent, but I hadn't wanted to complicate things with the logic. The logic is fully specified by the docstring "Should be either dust (the default), i100, i60, mask, temp, or xmap with whichparent SFD. Should be Beta for whichparent Synch. Should be Rmap for whichparent FINK. Should be clean for whichparent Haslam" if you wanted to push that change.

Some of the logic issues come from a naming convention that Finkbeiner et al changed between the different whichparent files. There is a trailing _4096 for the whichparent SFD files, but none of the others

andrew-saydjari · 2024-07-10T15:45:15Z

- requirements = {'filename': 'SFD_dust_4096_{}.fits'.format(pole)}
- local_fname = os.path.join(
+ if (map_variant == 'SFD') and (component in ('dust', 'i100', 'i60', 'mask')):
+ requirements = {'filename': '{}_{}_4096_{}.fits'.format(map_variant,component,pole)}


See here for the naming complication.

andrew-saydjari · 2024-05-05T01:51:14Z

This PR allows users to query the SFD bitmask as well as all other SFD style data products stored on the Harvard Dataverse under the same DOI. The bitmask is forced to use order 0 interpolation to avoid violating the bit quantization. It appears that the padding done to the data to try to handle distortions at the Galactic Plane was not as successfully applied to the mask data product, leading to some artifacts near b = 0. However, it appears that this is a faithful reproduction of the values in the FITS file and the WCS header and so I am not really in favor of changing it at this level. A lower level change to better handle the distortions etc. would be preferable. In total, I think making these data products, especially the bitmask, is extremely valuable.

andrew-saydjari · 2024-05-05T01:52:12Z

I should say, I did not make any effort to enable this functionality from SFDWebQuery.

andrew-saydjari · 2024-05-05T01:53:01Z

Examples of the issues with the lack of mask padding near b = 0.

gregreen · 2024-07-10T14:03:45Z

+ whichparent='SFD',
+ which='dust'


Can we come up with more intuitive names for these keyword arguments? Maybe map_variant instead of whichparent, and component instead of which? Additionally, is it possible to infer whichparent from which? If so, then one option will suffice.

Happy to make the variable name change. It is true that which is unique, and specifies whichparent, but I hadn't wanted to complicate things with the logic. The logic is fully specified by the docstring "Should be either dust (the default), i100, i60, mask, temp, or xmap with whichparent SFD. Should be Beta for whichparent Synch. Should be Rmap for whichparent FINK. Should be clean for whichparent Haslam" if you wanted to push that change.

Some of the logic issues come from a naming convention that Finkbeiner et al changed between the different whichparent files. There is a trailing _4096 for the whichparent SFD files, but none of the others

andrew-saydjari · 2024-07-10T15:45:15Z

- requirements = {'filename': 'SFD_dust_4096_{}.fits'.format(pole)}
- local_fname = os.path.join(
+ if (map_variant == 'SFD') and (component in ('dust', 'i100', 'i60', 'mask')):
+ requirements = {'filename': '{}_{}_4096_{}.fits'.format(map_variant,component,pole)}


See here for the naming complication.

andrew-saydjari · 2024-05-05T01:51:14Z

This PR allows users to query the SFD bitmask as well as all other SFD style data products stored on the Harvard Dataverse under the same DOI. The bitmask is forced to use order 0 interpolation to avoid violating the bit quantization. It appears that the padding done to the data to try to handle distortions at the Galactic Plane was not as successfully applied to the mask data product, leading to some artifacts near b = 0. However, it appears that this is a faithful reproduction of the values in the FITS file and the WCS header and so I am not really in favor of changing it at this level. A lower level change to better handle the distortions etc. would be preferable. In total, I think making these data products, especially the bitmask, is extremely valuable.

andrew-saydjari · 2024-05-05T01:52:12Z

I should say, I did not make any effort to enable this functionality from SFDWebQuery.

andrew-saydjari · 2024-05-05T01:53:01Z

Examples of the issues with the lack of mask padding near b = 0.

gregreen · 2024-07-10T14:03:45Z

+ whichparent='SFD',
+ which='dust'


Can we come up with more intuitive names for these keyword arguments? Maybe map_variant instead of whichparent, and component instead of which? Additionally, is it possible to infer whichparent from which? If so, then one option will suffice.

Happy to make the variable name change. It is true that which is unique, and specifies whichparent, but I hadn't wanted to complicate things with the logic. The logic is fully specified by the docstring "Should be either dust (the default), i100, i60, mask, temp, or xmap with whichparent SFD. Should be Beta for whichparent Synch. Should be Rmap for whichparent FINK. Should be clean for whichparent Haslam" if you wanted to push that change.

Some of the logic issues come from a naming convention that Finkbeiner et al changed between the different whichparent files. There is a trailing _4096 for the whichparent SFD files, but none of the others

andrew-saydjari · 2024-07-10T15:45:15Z

- requirements = {'filename': 'SFD_dust_4096_{}.fits'.format(pole)}
- local_fname = os.path.join(
+ if (map_variant == 'SFD') and (component in ('dust', 'i100', 'i60', 'mask')):
+ requirements = {'filename': '{}_{}_4096_{}.fits'.format(map_variant,component,pole)}


See here for the naming complication.

andrew-saydjari · 2024-05-05T01:51:14Z

This PR allows users to query the SFD bitmask as well as all other SFD style data products stored on the Harvard Dataverse under the same DOI. The bitmask is forced to use order 0 interpolation to avoid violating the bit quantization. It appears that the padding done to the data to try to handle distortions at the Galactic Plane was not as successfully applied to the mask data product, leading to some artifacts near b = 0. However, it appears that this is a faithful reproduction of the values in the FITS file and the WCS header and so I am not really in favor of changing it at this level. A lower level change to better handle the distortions etc. would be preferable. In total, I think making these data products, especially the bitmask, is extremely valuable.

andrew-saydjari · 2024-05-05T01:52:12Z

I should say, I did not make any effort to enable this functionality from SFDWebQuery.

andrew-saydjari · 2024-05-05T01:53:01Z

Examples of the issues with the lack of mask padding near b = 0.

gregreen · 2024-07-10T14:03:45Z

+ whichparent='SFD',
+ which='dust'


Can we come up with more intuitive names for these keyword arguments? Maybe map_variant instead of whichparent, and component instead of which? Additionally, is it possible to infer whichparent from which? If so, then one option will suffice.

Happy to make the variable name change. It is true that which is unique, and specifies whichparent, but I hadn't wanted to complicate things with the logic. The logic is fully specified by the docstring "Should be either dust (the default), i100, i60, mask, temp, or xmap with whichparent SFD. Should be Beta for whichparent Synch. Should be Rmap for whichparent FINK. Should be clean for whichparent Haslam" if you wanted to push that change.

Some of the logic issues come from a naming convention that Finkbeiner et al changed between the different whichparent files. There is a trailing _4096 for the whichparent SFD files, but none of the others

andrew-saydjari · 2024-07-10T15:45:15Z

- requirements = {'filename': 'SFD_dust_4096_{}.fits'.format(pole)}
- local_fname = os.path.join(
+ if (map_variant == 'SFD') and (component in ('dust', 'i100', 'i60', 'mask')):
+ requirements = {'filename': '{}_{}_4096_{}.fits'.format(map_variant,component,pole)}


See here for the naming complication.

andrew-saydjari · 2024-05-05T01:51:14Z

This PR allows users to query the SFD bitmask as well as all other SFD style data products stored on the Harvard Dataverse under the same DOI. The bitmask is forced to use order 0 interpolation to avoid violating the bit quantization. It appears that the padding done to the data to try to handle distortions at the Galactic Plane was not as successfully applied to the mask data product, leading to some artifacts near b = 0. However, it appears that this is a faithful reproduction of the values in the FITS file and the WCS header and so I am not really in favor of changing it at this level. A lower level change to better handle the distortions etc. would be preferable. In total, I think making these data products, especially the bitmask, is extremely valuable.

andrew-saydjari · 2024-05-05T01:52:12Z

I should say, I did not make any effort to enable this functionality from SFDWebQuery.

andrew-saydjari · 2024-05-05T01:53:01Z

Examples of the issues with the lack of mask padding near b = 0.

gregreen · 2024-07-10T14:03:45Z

+ whichparent='SFD',
+ which='dust'


Can we come up with more intuitive names for these keyword arguments? Maybe map_variant instead of whichparent, and component instead of which? Additionally, is it possible to infer whichparent from which? If so, then one option will suffice.

Happy to make the variable name change. It is true that which is unique, and specifies whichparent, but I hadn't wanted to complicate things with the logic. The logic is fully specified by the docstring "Should be either dust (the default), i100, i60, mask, temp, or xmap with whichparent SFD. Should be Beta for whichparent Synch. Should be Rmap for whichparent FINK. Should be clean for whichparent Haslam" if you wanted to push that change.

Some of the logic issues come from a naming convention that Finkbeiner et al changed between the different whichparent files. There is a trailing _4096 for the whichparent SFD files, but none of the others

andrew-saydjari · 2024-07-10T15:45:15Z

- requirements = {'filename': 'SFD_dust_4096_{}.fits'.format(pole)}
- local_fname = os.path.join(
+ if (map_variant == 'SFD') and (component in ('dust', 'i100', 'i60', 'mask')):
+ requirements = {'filename': '{}_{}_4096_{}.fits'.format(map_variant,component,pole)}


See here for the naming complication.

andrew-saydjari · 2024-05-05T01:51:14Z

This PR allows users to query the SFD bitmask as well as all other SFD style data products stored on the Harvard Dataverse under the same DOI. The bitmask is forced to use order 0 interpolation to avoid violating the bit quantization. It appears that the padding done to the data to try to handle distortions at the Galactic Plane was not as successfully applied to the mask data product, leading to some artifacts near b = 0. However, it appears that this is a faithful reproduction of the values in the FITS file and the WCS header and so I am not really in favor of changing it at this level. A lower level change to better handle the distortions etc. would be preferable. In total, I think making these data products, especially the bitmask, is extremely valuable.

andrew-saydjari · 2024-05-05T01:52:12Z

I should say, I did not make any effort to enable this functionality from SFDWebQuery.

andrew-saydjari · 2024-05-05T01:53:01Z

Examples of the issues with the lack of mask padding near b = 0.

gregreen · 2024-07-10T14:03:45Z

+ whichparent='SFD',
+ which='dust'


Can we come up with more intuitive names for these keyword arguments? Maybe map_variant instead of whichparent, and component instead of which? Additionally, is it possible to infer whichparent from which? If so, then one option will suffice.

Happy to make the variable name change. It is true that which is unique, and specifies whichparent, but I hadn't wanted to complicate things with the logic. The logic is fully specified by the docstring "Should be either dust (the default), i100, i60, mask, temp, or xmap with whichparent SFD. Should be Beta for whichparent Synch. Should be Rmap for whichparent FINK. Should be clean for whichparent Haslam" if you wanted to push that change.

Some of the logic issues come from a naming convention that Finkbeiner et al changed between the different whichparent files. There is a trailing _4096 for the whichparent SFD files, but none of the others

andrew-saydjari · 2024-07-10T15:45:15Z

- requirements = {'filename': 'SFD_dust_4096_{}.fits'.format(pole)}
- local_fname = os.path.join(
+ if (map_variant == 'SFD') and (component in ('dust', 'i100', 'i60', 'mask')):
+ requirements = {'filename': '{}_{}_4096_{}.fits'.format(map_variant,component,pole)}


See here for the naming complication.

andrew-saydjari · 2024-05-05T01:51:14Z

This PR allows users to query the SFD bitmask as well as all other SFD style data products stored on the Harvard Dataverse under the same DOI. The bitmask is forced to use order 0 interpolation to avoid violating the bit quantization. It appears that the padding done to the data to try to handle distortions at the Galactic Plane was not as successfully applied to the mask data product, leading to some artifacts near b = 0. However, it appears that this is a faithful reproduction of the values in the FITS file and the WCS header and so I am not really in favor of changing it at this level. A lower level change to better handle the distortions etc. would be preferable. In total, I think making these data products, especially the bitmask, is extremely valuable.

andrew-saydjari · 2024-05-05T01:52:12Z

I should say, I did not make any effort to enable this functionality from SFDWebQuery.

andrew-saydjari · 2024-05-05T01:53:01Z

Examples of the issues with the lack of mask padding near b = 0.

gregreen · 2024-07-10T14:03:45Z

+ whichparent='SFD',
+ which='dust'


Can we come up with more intuitive names for these keyword arguments? Maybe map_variant instead of whichparent, and component instead of which? Additionally, is it possible to infer whichparent from which? If so, then one option will suffice.

Happy to make the variable name change. It is true that which is unique, and specifies whichparent, but I hadn't wanted to complicate things with the logic. The logic is fully specified by the docstring "Should be either dust (the default), i100, i60, mask, temp, or xmap with whichparent SFD. Should be Beta for whichparent Synch. Should be Rmap for whichparent FINK. Should be clean for whichparent Haslam" if you wanted to push that change.

Some of the logic issues come from a naming convention that Finkbeiner et al changed between the different whichparent files. There is a trailing _4096 for the whichparent SFD files, but none of the others

andrew-saydjari · 2024-07-10T15:45:15Z

- requirements = {'filename': 'SFD_dust_4096_{}.fits'.format(pole)}
- local_fname = os.path.join(
+ if (map_variant == 'SFD') and (component in ('dust', 'i100', 'i60', 'mask')):
+ requirements = {'filename': '{}_{}_4096_{}.fits'.format(map_variant,component,pole)}


See here for the naming complication.

andrew-saydjari · 2024-05-05T01:51:14Z

This PR allows users to query the SFD bitmask as well as all other SFD style data products stored on the Harvard Dataverse under the same DOI. The bitmask is forced to use order 0 interpolation to avoid violating the bit quantization. It appears that the padding done to the data to try to handle distortions at the Galactic Plane was not as successfully applied to the mask data product, leading to some artifacts near b = 0. However, it appears that this is a faithful reproduction of the values in the FITS file and the WCS header and so I am not really in favor of changing it at this level. A lower level change to better handle the distortions etc. would be preferable. In total, I think making these data products, especially the bitmask, is extremely valuable.

andrew-saydjari · 2024-05-05T01:52:12Z

I should say, I did not make any effort to enable this functionality from SFDWebQuery.

andrew-saydjari · 2024-05-05T01:53:01Z

Examples of the issues with the lack of mask padding near b = 0.

gregreen · 2024-07-10T14:03:45Z

+ whichparent='SFD',
+ which='dust'


Can we come up with more intuitive names for these keyword arguments? Maybe map_variant instead of whichparent, and component instead of which? Additionally, is it possible to infer whichparent from which? If so, then one option will suffice.

Happy to make the variable name change. It is true that which is unique, and specifies whichparent, but I hadn't wanted to complicate things with the logic. The logic is fully specified by the docstring "Should be either dust (the default), i100, i60, mask, temp, or xmap with whichparent SFD. Should be Beta for whichparent Synch. Should be Rmap for whichparent FINK. Should be clean for whichparent Haslam" if you wanted to push that change.

Some of the logic issues come from a naming convention that Finkbeiner et al changed between the different whichparent files. There is a trailing _4096 for the whichparent SFD files, but none of the others

andrew-saydjari · 2024-07-10T15:45:15Z

- requirements = {'filename': 'SFD_dust_4096_{}.fits'.format(pole)}
- local_fname = os.path.join(
+ if (map_variant == 'SFD') and (component in ('dust', 'i100', 'i60', 'mask')):
+ requirements = {'filename': '{}_{}_4096_{}.fits'.format(map_variant,component,pole)}


See here for the naming complication.

andrew-saydjari added 5 commits April 27, 2024 20:41

add support for additional sfd maps
c0b81f8

fix temp and xmap SFD files
9d7cc95

initial pass at docs
3484776

edit SFD bit mask description
00e267e

order 0 mask
b70a45f

gregreen reviewed Jul 10, 2024
View reviewed changes

Comment threaddustmaps/sfd.py Outdated

gregreen reviewed Jul 10, 2024
View reviewed changes

change which variable names, pythonic sub on logic
75dbb40

andrew-saydjari commented Jul 10, 2024
View reviewed changes

andrew-saydjari added 5 commits April 27, 2024 20:41

add support for additional sfd maps
c0b81f8

fix temp and xmap SFD files
9d7cc95

initial pass at docs
3484776

edit SFD bit mask description
00e267e

order 0 mask
b70a45f

gregreen reviewed Jul 10, 2024
View reviewed changes

Comment threaddustmaps/sfd.py Outdated

gregreen reviewed Jul 10, 2024
View reviewed changes

change which variable names, pythonic sub on logic
75dbb40

andrew-saydjari commented Jul 10, 2024
View reviewed changes

andrew-saydjari added 5 commits April 27, 2024 20:41

add support for additional sfd maps
c0b81f8

fix temp and xmap SFD files
9d7cc95

initial pass at docs
3484776

edit SFD bit mask description
00e267e

order 0 mask
b70a45f

gregreen reviewed Jul 10, 2024
View reviewed changes

Comment threaddustmaps/sfd.py Outdated

gregreen reviewed Jul 10, 2024
View reviewed changes

change which variable names, pythonic sub on logic
75dbb40

andrew-saydjari commented Jul 10, 2024
View reviewed changes

andrew-saydjari added 5 commits April 27, 2024 20:41

add support for additional sfd maps
c0b81f8

fix temp and xmap SFD files
9d7cc95

initial pass at docs
3484776

edit SFD bit mask description
00e267e

order 0 mask
b70a45f

gregreen reviewed Jul 10, 2024
View reviewed changes

Comment threaddustmaps/sfd.py Outdated

gregreen reviewed Jul 10, 2024
View reviewed changes

change which variable names, pythonic sub on logic
75dbb40

andrew-saydjari commented Jul 10, 2024
View reviewed changes

andrew-saydjari added 5 commits April 27, 2024 20:41

add support for additional sfd maps
c0b81f8

fix temp and xmap SFD files
9d7cc95

initial pass at docs
3484776

edit SFD bit mask description
00e267e

order 0 mask
b70a45f

gregreen reviewed Jul 10, 2024
View reviewed changes

Comment threaddustmaps/sfd.py Outdated

gregreen reviewed Jul 10, 2024
View reviewed changes

change which variable names, pythonic sub on logic
75dbb40

andrew-saydjari commented Jul 10, 2024
View reviewed changes

andrew-saydjari added 5 commits April 27, 2024 20:41

add support for additional sfd maps
c0b81f8

fix temp and xmap SFD files
9d7cc95

initial pass at docs
3484776

edit SFD bit mask description
00e267e

order 0 mask
b70a45f

gregreen reviewed Jul 10, 2024
View reviewed changes

Comment threaddustmaps/sfd.py Outdated

gregreen reviewed Jul 10, 2024
View reviewed changes

change which variable names, pythonic sub on logic
75dbb40

andrew-saydjari commented Jul 10, 2024
View reviewed changes

andrew-saydjari added 5 commits April 27, 2024 20:41

add support for additional sfd maps
c0b81f8

fix temp and xmap SFD files
9d7cc95

initial pass at docs
3484776

edit SFD bit mask description
00e267e

order 0 mask
b70a45f

gregreen reviewed Jul 10, 2024
View reviewed changes

Comment threaddustmaps/sfd.py Outdated

gregreen reviewed Jul 10, 2024
View reviewed changes

change which variable names, pythonic sub on logic
75dbb40

andrew-saydjari commented Jul 10, 2024
View reviewed changes

andrew-saydjari added 5 commits April 27, 2024 20:41

add support for additional sfd maps
c0b81f8

fix temp and xmap SFD files
9d7cc95

initial pass at docs
3484776

edit SFD bit mask description
00e267e

order 0 mask
b70a45f

gregreen reviewed Jul 10, 2024
View reviewed changes

Comment threaddustmaps/sfd.py Outdated

gregreen reviewed Jul 10, 2024
View reviewed changes

change which variable names, pythonic sub on logic
75dbb40

andrew-saydjari commented Jul 10, 2024
View reviewed changes

Conversation

andrew-saydjari commented May 5, 2024

Uh oh!

andrew-saydjari commented May 5, 2024

Uh oh!

andrew-saydjari commented May 5, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

andrew-saydjari commented May 5, 2024

Uh oh!

andrew-saydjari commented May 5, 2024

Uh oh!

andrew-saydjari commented May 5, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

andrew-saydjari commented May 5, 2024

Uh oh!

andrew-saydjari commented May 5, 2024

Uh oh!

andrew-saydjari commented May 5, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

andrew-saydjari commented May 5, 2024

Uh oh!

andrew-saydjari commented May 5, 2024

Uh oh!

andrew-saydjari commented May 5, 2024

Uh oh!

Uh oh!