Example files for GEOMETRY and GEOGRAPHY logical type

[paleolimbot]

As discussed on the mailing list, it's best to get example files early!

Code to generate in details (requires apache/arrow@main...paleolimbot:arrow:parquet-geo-write-files-from-geoarrow , which is a slightly more functional but less appropriate initial version of apache/arrow#45459 ). I've also added the full suite of geoarrow-data files (even the big ones) to that forthcoming release: https://github.com/geoarrow/geoarrow-data .

Details

importurllib.requestimportjsonimportpyarrowaspafrompyarrowimportparquetimportgeoarrow.pyarrowasgamanifest_url= (
"https://raw.githubusercontent.com/geoarrow/geoarrow-data/v0.2.0-rc4/manifest.json"
)
files= {}
withurllib.request.urlopen(manifest_url) asf:
manifest=json.load(f)
forgroupinmanifest["groups"]:
forfileingroup["files"]:
iffile["format"] =="arrows/wkb":
files[group["name"] +"_"+file["name"]] =file["url"]
out_dir="/Users/dewey/gh/parquet-testing/data/geospatial"ones_that_didnt_work= []
forname, urlinfiles.items():
# Skip big files + one CRS example that includes a non-PROJJSON value# on purpose (allowed in GeoArrow), which is rightly rejected# by Parquetif (
"microsoft-buildings"innameor ("ns-water"innameandname!="ns-water_water-point")
or"wkt2"inname
):
print(f"Skipping {name}")
continue# Maintain chunking from IPC into Parquetout=f"{out_dir}/{name}.parquet"with (
urllib.request.urlopen(url) asf,
pa.ipc.open_stream(f) asreader,
parquet.ParquetWriter(
out,
reader.schema,
store_schema=False,
compression="none",
write_geospatial_logical_types=True,
) aswriter,
):
original_schema=reader.schemaprint(f"Reading {url}")
forbatchinreader:
writer.write_batch(batch)
print(f"Wrote {out}")
# Read in original table for comparisonwith (
urllib.request.urlopen(url) asf,
pa.ipc.open_stream(f) asreader
):
original_table=reader.read_all()
print(f"Checking {out}")
withparquet.ParquetFile(out, arrow_extensions_enabled=True) asf:
iff.schema_arrow!=original_table.schema:
print(f"Schema mismatch:\n{f.schema_arrow}\nvs\n{original_schema}")
continuereread=f.read()
ifreread!=original_table:
print("Table mismatch")

[paleolimbot]

@Kontinuation@zhangfengcdt Can you give these a try from Java when you're ready? I'm fairly confident that they are correct, including the "crs" examples that dump the actual payload of the PROJJSON to the file metadata.

[paleolimbot]

I pushed an update to three files here - the original fields that PROJJSON crses were written to were very likely to collide with eachother if you did things like read a Parquet file, filter it, then write it again 😬 . The new files add a hash of the value to the end of the key (e.g., projjson_crs_value_0ffad8372). Totally up for discussion whether that's a good idea or not 🙂 .

[paleolimbot]

I updated these to be a bit more intentional about the corner cases we collectively ran into in apache/parquet-java#2971 and apache/arrow#45459. I'm not sure the Python files to generate them belong in this repo but it does make it easier to see what they contain. I also included CRS examples because that was also something that required some thinking about in the C++ PR...happy to remove or tweak any of these if I didn't get the spirit of the format change right 🙂 .

[alamb]

Today at the Parquet sync @emkornfield said he might have some time to review this PR

[emkornfield]

This all seems reasonable, going to merge.

[alamb]

Thank you @emkornfield and @paleolimbot 🙏

[paleolimbot]

Thank you both!