Support reading skims in Parquet format

[amsamimi]

Description:
Currently, skims are read from OMX files, but based on our experience, storing skims in Parquet format can reduce file size by ~30%. This would significantly reduce storage requirements and improve efficiency.
We already support reading Parquet instead of CSV for the population synthesizer inputs, which has proven very valuable for reducing the size of model inputs. Extending this capability to skims would provide similar benefits.

Proposed enhancement:
Add support for reading skims directly from Parquet format.
Maintain backward compatibility with existing OMX inputs.

Benefits:
Reduced disk usage.
Faster I/O performance.

[lmz]

Just curious - is there a recommended way to store skims in parquet?
Also, any notes on how much faster i/o would be would be useful...

[AmirSamimi-TfNSW]

@lmz
Skims can be stored in Parquet format, similar to how they’re currently handled in OMX. Since our visualizer supports Parquet-formatted skims, we convert them using the same structure. This conversion reduces the file size from 7.23 GB to 4.98 GB.
We expect ActivitySim to load these files more quickly, though the extent of the performance improvement still needs to be tested. The disk space savings are around 30%, and while faster I/O is anticipated, the actual impact will need to be measured.

[jpn--]

There are a few things that the HDF5 format handles transparently which we will need to address to manage skim data in parquet format:

• Lookups, aka row and column headers: This data is one-dim, a different shape from the 2-dim regular matrix tables. Do we expect the lookups to be stored in exploded format in the parquet? e.g. OAZ = [1,2,3,1,2,3,1,2,3] and DTAZ = [1,1,1,2,2,2,3,3,3]. Or should they be provided in a seperate file?
• Do we allow sparse arrays or only dense arrays? Implicitly OMX is only dense so it's probably easiest to start with dense only, but planning to eventually allow sparse could be useful.
• Ordering: are the 2-d arrays to be stored in row major or column major format?

None of this is super hard to handle but I don't think we can ignore having a parquet spec if we want to do parquet, or it will bite us eventually.

[jpn--]

@amsamimi You noted that you are already experimenting with tools that output skims in parquet format. Are you able to describe what that output format looks like for the tools you are using? In particular, addressing the questions outlined above on lookups/taz labels, sparseness, and ordering?

[AmirSamimi-TfNSW]

I used this tool to convert skims.omx to skims.parquet for use in a visualizer. @asiripanich, could you provide more details on the questions Jeff raised in the comment above?

[asiripanich]

@amsamimi You noted that you are already experimenting with tools that output skims in parquet format. Are you able to describe what that output format looks like for the tools you are using? In particular, addressing the questions outlined above on lookups/taz labels, sparseness, and ordering?

Hi @jpn--, the tool that @AmirSamimi-TfNSW mentioned converts an OMX file to a Parquet file with the following format:

Origin	Destination	Matrix 1 Value	Matrix 2 Value	...	Matrix n Value

This format is dense (it includes all OD pairs) and as you can see in the code below, there are no other specific optimizations.

with omx.open_file(omx_path) as matrix:
 zones = matrix.shape()[1]
 od_pairs = list(
 itertools.product(range(1, zones + 1), range(1, zones + 1))
 )
 origins = np.array([t[0] for t in od_pairs], dtype=np.int64)
 destinations = np.array([t[1] for t in od_pairs], dtype=np.int64)
 table_contents = {
 "origin": origins,
 "destination": destinations,
 }
 for table_name in matrix.list_matrices():
 column = extract_matrix(matrix, table_name)
 table_contents[table_name] = column

 pa_table = pa.table(table_contents)
 return pa_table

The main purpose of the skims parquet file is for a dashboard we developed which relies on lazy loading via Polars Lazy DataFrame.