Skip to content

Fix fixture_xr_image to open earth_day_01d_p directly with rioxarray - #2963

Merged
weiji14 merged 8 commits into
mainfrom
fix/xr_image_fixture
Jan 11, 2024
Merged

Fix fixture_xr_image to open earth_day_01d_p directly with rioxarray#2963
weiji14 merged 8 commits into
mainfrom
fix/xr_image_fixture

Conversation

@weiji14

@weiji14weiji14 commented Jan 7, 2024

Copy link
Copy Markdown
Member

Description of proposed changes

Rioxarray can now read the 3-bands from @earth_day_01d_p directly without needing to parse the colorinterp information from the GeoTIFF file.

Addresses some of the failing tests in #2961 (comment), resolves#2590 (comment)

Reminders

  • Run make format and make check to make sure the code follows the style guide.
  • Add tests for new features or tests that would have caught the bug that you're fixing.
  • Add new public functions/methods/classes to doc/api/index.rst.
  • Write detailed docstrings for all functions/methods.
  • If wrapping a new module, open a 'Wrap new GMT module' issue and submit reasonably-sized PRs.
  • If adding new functionality, add an example to docstrings or tutorials.
  • Use underscores (not hyphens) in names of Python files and directories.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash command is:

  • /format: automatically format and lint the code

@weiji14
Fix fixture_xr_image to open earth_day_01d_p directly with rioxarray
eeb6c63
Rioxarray can now read the 3-bands from `@earth_day_01d_p` directly without needing to parse the colorinterp information from the GeoTIFF file.
@weiji14weiji14 added the maintenance Boring but important stuff for the core devs label Jan 7, 2024
@weiji14weiji14 added this to the 0.11.0 milestone Jan 7, 2024
@weiji14weiji14 self-assigned this Jan 7, 2024
weiji14
weiji14 commented Jan 7, 2024
View reviewed changes
Comment threadpygmt/tests/test_grdimage_image.py
@@ -19,7 +19,9 @@ def fixture_xr_image():
"""
geotiff = which(fname="@earth_day_01d_p", download="c")

@weiji14weiji14Jan 7, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The earth_day_01d_p.tif grid downloaded with GMT 6.5.0 shows this:

$ gmt which @earth_day_01d_p
$ gdalinfo ~/.gmt/cache/earth_day_01d_p.tif
Driver: GTiff/GeoTIFF
Files: ~/.gmt/cache/earth_day_01d_p.tif
Size is 360, 180
Coordinate System is:
GEOGCRS["unknown",
DATUM["unknown",
ELLIPSOID["unknown",6378137,0,
LENGTHUNIT["metre",1,
ID["EPSG",9001]]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433,
ID["EPSG",9122]]],
CS[ellipsoidal,2],
AXIS["latitude",north,
ORDER[1],
ANGLEUNIT["degree",0.0174532925199433,
ID["EPSG",9122]]],
AXIS["longitude",east,
ORDER[2],
ANGLEUNIT["degree",0.0174532925199433,
ID["EPSG",9122]]]]
Data axis to CRS axis mapping: 2,1
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=DEFLATE
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left (-180.0000000, 90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)Lower Left (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)Upper Right ( 180.0000000, 90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)Center ( 0.0000000, 0.0000000) ( 0d 0' 0.01"E, 0d 0' 0.01"N)Band 1 Block=360x7 Type=Byte, ColorInterp=Undefined Min=10.000 Max=255.000  Minimum=10.000, Maximum=255.000, Mean=49.508, StdDev=65.299 NoData Value=0 Metadata: STATISTICS_MAXIMUM=255 STATISTICS_MEAN=49.507654320988 STATISTICS_MINIMUM=10 STATISTICS_STDDEV=65.299010755209 STATISTICS_VALID_PERCENT=100Band 2 Block=360x7 Type=Byte, ColorInterp=Undefined Min=10.000 Max=255.000  Minimum=10.000, Maximum=255.000, Mean=49.911, StdDev=62.129 NoData Value=0 Metadata: STATISTICS_MAXIMUM=255 STATISTICS_MEAN=49.910941358025 STATISTICS_MINIMUM=10 STATISTICS_STDDEV=62.129078727248 STATISTICS_VALID_PERCENT=100Band 3 Block=360x7 Type=Byte, ColorInterp=Undefined Min=10.000 Max=255.000  Minimum=10.000, Maximum=255.000, Mean=65.605, StdDev=44.820 NoData Value=0 Metadata: STATISTICS_MAXIMUM=255 STATISTICS_MEAN=65.605154320988 STATISTICS_MINIMUM=10 STATISTICS_STDDEV=44.819618148349 STATISTICS_VALID_PERCENT=100

If I run gmt which @earth_day_01d_p with GMT 6.4.0, it errors with gmtwhich [ERROR]: File earth_day_01d_p.tif not found!. Did something change on the GMT data server?

@weiji14weiji14Jan 7, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the old @earth_day_01d_p (unzip earth_day_01d_p.tif.zip) from #2937 cached at https://github.com/GenericMappingTools/pygmt/actions/runs/7376169058 that was used in GMT 6.4.0:

$ gdalinfo earth_day_01d_p.tif
Driver: GTiff/GeoTIFF
Files: earth_day_01d_p.tif
Size is 360, 180
Coordinate System is:
GEOGCRS["unknown",
DATUM["unknown",
ELLIPSOID["unknown",6378137,0,
LENGTHUNIT["metre",1,
ID["EPSG",9001]]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433,
ID["EPSG",9122]]],
CS[ellipsoidal,2],
AXIS["latitude",north,
ORDER[1],
ANGLEUNIT["degree",0.0174532925199433,
ID["EPSG",9122]]],
AXIS["longitude",east,
ORDER[2],
ANGLEUNIT["degree",0.0174532925199433,
ID["EPSG",9122]]]]
Data axis to CRS axis mapping: 2,1
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=DEFLATE
INTERLEAVE=BAND
PREDICTOR=2
Corner Coordinates:
Upper Left (-180.0000000, 90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)
Lower Left (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)
Upper Right ( 180.0000000, 90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)
Center ( 0.0000000, 0.0000000) ( 0d 0' 0.01"E, 0d 0' 0.01"N)
Band 1 Block=360x22 Type=Byte, ColorInterp=Palette
Color Table (RGB with 256 entries)
0: 248,224,168,255
1: 240,224,152,255
2: 104,104,96,255
3: 16,24,56,255
4: 24,28,24,255
5: 104,80,48,255
6: 220,224,228,255
7: 48,56,32,255
8: 184,192,176,255
9: 104,104,40,255
10: 16,48,8,255
11: 96,96,56,255
12: 188,184,184,255
13: 128,112,60,255
14: 48,72,24,255
15: 8,16,48,255
16: 216,200,192,255
17: 80,80,28,255
18: 56,48,44,255
19: 72,88,24,255
20: 204,168,140,255
21: 128,136,136,255
22: 72,68,40,255
23: 144,120,68,255
24: 144,144,136,255
25: 88,80,48,255
26: 168,160,152,255
27: 88,104,40,255
28: 8,16,32,255
29: 168,116,84,255
30: 120,100,48,255
31: 200,192,184,255
32: 176,168,160,255
33: 136,112,80,255
34: 204,204,200,255
35: 184,152,124,255
36: 96,76,48,255
37: 88,72,48,255
38: 184,196,192,255
39: 28,56,48,255
40: 92,100,124,255
41: 184,176,176,255
42: 160,160,152,255
43: 132,144,144,255
44: 208,192,184,255
45: 128,100,64,255
46: 72,100,24,255
47: 36,48,16,255
48: 140,140,80,255
49: 152,144,136,255
50: 56,64,16,255
51: 168,168,168,255
52: 224,200,144,255
53: 176,176,168,255
54: 80,88,28,255
55: 152,144,144,255
56: 156,148,84,255
57: 160,160,160,255
58: 16,16,48,255
59: 168,176,168,255
60: 216,200,176,255
61: 16,8,48,255
62: 72,80,24,255
63: 132,140,124,255
64: 88,80,40,255
65: 152,152,176,255
66: 16,16,56,255
67: 20,48,24,255
68: 200,200,192,255
69: 56,48,72,255
70: 40,56,24,255
71: 120,136,56,255
72: 152,160,152,255
73: 8,8,24,255
74: 192,184,176,255
75: 248,244,172,255
76: 116,124,136,255
77: 88,68,40,255
78: 64,72,32,255
79: 172,180,176,255
80: 176,152,132,255
81: 24,40,24,255
82: 60,100,24,255
83: 168,148,128,255
84: 176,132,88,255
85: 200,192,192,255
86: 76,76,48,255
87: 152,116,80,255
88: 8,16,60,255
89: 152,152,144,255
90: 240,216,152,255
91: 52,80,32,255
92: 20,24,40,255
93: 132,144,156,255
94: 88,88,48,255
95: 20,32,8,255
96: 84,68,28,255
97: 172,164,180,255
98: 152,152,152,255
99: 120,112,56,255
100: 88,88,40,255
101: 56,80,8,255
102: 80,80,40,255
103: 176,176,160,255
104: 48,56,8,255
105: 104,84,68,255
106: 48,120,76,255
107: 184,176,168,255
108: 108,96,56,255
109: 232,224,212,255
110: 16,16,40,255
111: 96,88,48,255
112: 144,152,144,255
113: 104,112,68,255
114: 220,180,148,255
115: 28,32,60,255
116: 56,72,8,255
117: 104,96,40,255
118: 56,72,24,255
119: 24,64,24,255
120: 148,164,160,255
121: 180,176,140,255
122: 216,208,196,255
123: 156,128,92,255
124: 72,84,40,255
125: 56,80,16,255
126: 72,72,24,255
127: 48,44,12,255
128: 124,108,96,255
129: 64,80,24,255
130: 132,136,148,255
131: 200,200,184,255
132: 120,100,68,255
133: 16,28,24,255
134: 144,144,144,255
135: 116,120,124,255
136: 248,224,184,255
137: 184,184,168,255
138: 72,80,32,255
139: 152,148,160,255
140: 56,76,44,255
141: 232,208,144,255
142: 176,168,168,255
143: 104,96,88,255
144: 60,56,32,255
145: 168,152,92,255
146: 184,184,176,255
147: 152,144,124,255
148: 232,180,120,255
149: 208,204,192,255
150: 152,152,128,255
151: 188,180,108,255
152: 48,56,16,255
153: 28,60,40,255
154: 56,64,24,255
155: 160,152,144,255
156: 48,72,8,255
157: 168,160,160,255
158: 228,228,184,255
159: 48,64,8,255
160: 160,152,152,255
161: 148,112,64,255
162: 32,48,8,255
163: 80,68,40,255
164: 132,140,100,255
165: 84,104,28,255
166: 36,48,28,255
167: 188,192,184,255
168: 168,168,160,255
169: 240,248,248,255
170: 60,52,24,255
171: 248,248,220,255
172: 36,36,8,255
173: 144,152,152,255
174: 32,64,8,255
175: 96,80,80,255
176: 72,140,148,255
177: 48,72,16,255
178: 200,184,176,255
179: 148,144,152,255
180: 136,140,136,255
181: 244,200,144,255
182: 180,200,212,255
183: 160,168,160,255
184: 40,64,8,255
185: 56,72,16,255
186: 48,56,24,255
187: 64,72,24,255
188: 176,184,168,255
189: 48,64,16,255
190: 104,116,56,255
191: 28,24,56,255
192: 132,124,136,255
193: 72,72,32,255
194: 80,88,40,255
195: 56,88,12,255
196: 236,204,184,255
197: 72,64,28,255
198: 64,84,32,255
199: 24,48,8,255
200: 88,80,84,255
201: 240,232,236,255
202: 148,100,60,255
203: 148,116,100,255
204: 160,144,140,255
205: 208,132,88,255
206: 36,32,24,255
207: 148,132,132,255
208: 52,84,24,255
209: 196,180,168,255
210: 204,188,140,255
211: 24,16,52,255
212: 104,120,40,255
213: 240,184,144,255
214: 72,88,32,255
215: 104,84,40,255
216: 60,52,16,255
217: 208,200,184,255
218: 32,56,8,255
219: 24,56,8,255
220: 160,168,152,255
221: 116,136,164,255
222: 204,196,172,255
223: 48,84,12,255
224: 88,128,80,255
225: 64,64,24,255
226: 248,244,192,255
227: 172,180,192,255
228: 144,144,60,255
229: 40,56,8,255
230: 76,72,88,255
231: 40,72,24,255
232: 196,180,160,255
233: 152,168,132,255
234: 64,88,24,255
235: 64,64,16,255
236: 40,48,8,255
237: 220,152,124,255
238: 92,84,28,255
239: 204,208,220,255
240: 104,128,96,255
241: 132,168,164,255
242: 40,76,8,255
243: 188,184,196,255
244: 56,76,80,255
245: 156,148,104,255
246: 244,240,156,255
247: 68,80,16,255
248: 28,64,80,255
249: 152,176,188,255
250: 132,84,56,255
251: 8,8,48,255
252: 8,8,56,255
253: 8,40,24,255
254: 8,72,96,255
255: 248,248,248,255

See how the GMT 6.4.0 version has 1 band with a Color Table, compared to 3 bands with GMT 6.5.0 release as in https://github.com/GenericMappingTools/pygmt/pull/2963/files#r1444072938.

@weiji14weiji14Jan 7, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, asked about the dataset changes upstream at GenericMappingTools/gmtserver-admin#257.

This was referenced Jan 7, 2024
Open
Closed
@github-actions

github-actionsBot commented Jan 8, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Summary of changed images

This is an auto-generated report of images that have changed on the DVC remote

StatusPath
modifiedpygmt/tests/baseline/test_tilemap_no_clip_False.png
modifiedpygmt/tests/baseline/test_tilemap_no_clip_True.png
modifiedpygmt/tests/baseline/test_tilemap_ogc_wgs84.png
modifiedpygmt/tests/baseline/test_tilemap_web_mercator.png

Image diff(s)

Details

Added images

Modified images

PathOldNew
test_tilemap_no_clip_False.png
test_tilemap_no_clip_True.png
test_tilemap_ogc_wgs84.png
test_tilemap_web_mercator.png

Report last updated at commit 2a41ddb

@weiji14
Trigger re-caching of earth_day_01d_p file
98f30ac
Modify pygmt/helpers/caching.py file slightly to create new cache.
@weiji14
Merge branch 'main' into fix/xr_image_fixture
64cd663
@seisman

seisman commented Jan 8, 2024

Copy link
Copy Markdown
Member

Rioxarray can now read the 3-bands from @earth_day_01d_p directly without needing to parse the colorinterp information from the GeoTIFF file.

What's the required rioxarray version?

@weiji14
Merge branch 'main' into fix/xr_image_fixture
b80652f
@weiji14
Remove elif block for manually creating RGB image
b8c9c3c
Downloading @earth_day_01d_p using GMT 6.4.0 actually works. The new earth_day_01d_p.tif file processed on 2023-09-29 doesn't have the quantized colormap, so no need for the extra handling. See GenericMappingTools/gmtserver-admin#257 for more info.
@weiji14

weiji14 commented Jan 8, 2024•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

Rioxarray can now read the 3-bands from @earth_day_01d_p directly without needing to parse the colorinterp information from the GeoTIFF file.

What's the required rioxarray version?

Sorry, should have worded that differently. This should work with any rioxarray version. The change to the @earth_day_01d_p GeoTIFF file from the quantized/indexed version to a non-quantized/3-band version as mentioned at GenericMappingTools/gmtserver-admin#257 means that we don't need the complicated parsing logic (see also #2590 (comment)). This also means we can easily create a load_blue_marble or load_black_marble (#1442) later.

weiji14
weiji14 commented Jan 9, 2024
View reviewed changes
Comment threadpygmt/tests/test_grdimage_image.py
Comment on lines 17 to +19
with rioxarray.open_rasterio(filename=geotiff) as rda:
if len(rda.band) == 1:
with rasterio.open(fp=geotiff) as src:
df_colormap = pd.DataFrame.from_dict(
data=src.colormap(1), orient="index"
)
array = src.read()

red = np.vectorize(df_colormap[0].get)(array)
green = np.vectorize(df_colormap[1].get)(array)
blue = np.vectorize(df_colormap[2].get)(array)
# alpha = np.vectorize(df_colormap[3].get)(array)

rda.data = red
da_red = rda.astype(dtype=np.uint8).copy()
rda.data = green
da_green = rda.astype(dtype=np.uint8).copy()
rda.data = blue
da_blue = rda.astype(dtype=np.uint8).copy()

xr_image = xr.concat(objs=[da_red, da_green, da_blue], dim="band")
if len(rda.band) == 3:
xr_image = rda.load()

@weiji14weiji14Jan 9, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait for GenericMappingTools/gmtserver-admin#257, to see if the 3-band GeoTIFF file stays, or we revert back to the quantized/indexed GeoTIFF file from before Sep 2023.

@weiji14weiji14Jan 11, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gonna assume that the 3-band GeoTIFF will stay for now, so that we can move on with the GMT 6.5.0 update. Marked this PR as ready for review.

@seismanseismanJan 11, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about splitting this PR into two separate RPs so if the upstream decide to to back to the indexed geotiff file, we can easily revert the changes.

@weiji14weiji14Jan 11, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kinda want to take the code from this fixture and just make a load_blue_marble dataset function after this 🙂 If we do go back to the indexed GeoTIFF files, then the modifications would go in that load_blue_marble function.

@weiji14
weiji14 marked this pull request as ready for review January 11, 2024 02:58
@weiji14weiji14 added the needs review This PR has higher priority and needs review. label Jan 11, 2024
@weiji14
weiji14 merged commit 5bfcbeb into mainJan 11, 2024
@weiji14
weiji14 deleted the fix/xr_image_fixture branch January 11, 2024 04:52
@seismanseisman removed the needs review This PR has higher priority and needs review. label Jan 11, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@seismanseismanseisman approved these changes

Assignees

@weiji14weiji14

Labels

maintenanceBoring but important stuff for the core devs

Projects

None yet

Milestone

0.11.0

Development

Successfully merging this pull request may close these issues.

2 participants

@weiji14@seisman