replace openmatrix with h5py by jpn-- · Pull Request #87 · ActivitySim/sharrow

jpn-- · 2026-08-07T19:52:30Z

This pull request removes the use of the openmatrix package throughout the codebase and replaces it with direct usage of the h5py library for reading and writing OMX (Open Matrix) HDF5 files. The change simplifies dependencies, updates documentation and examples, and refactors code to use h5py APIs. Additionally, improvements are made to shared memory handling, including support for custom array storage order and better memory management.

Dependency and Documentation Updates:

Removed openmatrix from all dependency files (envs/development.yml, envs/testing.yml, pyproject.toml, .github/workflows/run-tests.yml) and updated code, documentation, and examples to use h5py directly for OMX file operations. [1][2][3][4][5][6][7][8][9]

Code Refactoring:

Refactored all code that previously used openmatrix to use h5py, including in sharrow/example_data.py, sharrow/omx.py, and sharrow/tests/test_datasets.py. This includes updating file reading/writing logic and dataset creation. [1][2][3][4][5]

OMX File Handling Improvements:

Rewrote the split_omx function in sharrow/omx.py to use h5py for splitting OMX files, added helper functions for copying metadata and datasets, and improved output file organization and lookup management.

Shared Memory Enhancements:

Added support for specifying custom array storage order (e.g., "last-axis-first") in shared memory, improved buffer initialization to be more memory efficient, and enhanced cleanup logic for memory-mapped arrays in sharrow/shared_memory.py. [1][2][3][4][5][6][7][8][9][10]

API and Type Annotations:

Updated function signatures and docstrings to reflect the removal of openmatrix and clarify the use of h5py objects, especially in sharrow/omx_reader.py. [1][2]

Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

Copilot

Pull request overview

Replaces OpenMatrix with direct h5py-based OMX handling and adds optimized eager, shared-memory, and memory-mapped loading.

Changes:

Refactors OMX reading, writing, splitting, and Zarr conversion.
Adds parallel loading and configurable shared-memory array ordering.
Removes OpenMatrix dependencies and updates tests and documentation.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`uv.lock`	Removes OpenMatrix and PyTables dependencies.
`sharrow/translate.py`	Uses the shared OMX reader for Zarr conversion.
`sharrow/tests/test_omx.py`	Adds OMX splitting and conversion tests.
`sharrow/tests/test_datasets.py`	Migrates tests to h5py and covers loading modes.
`sharrow/shared_memory.py`	Adds array ordering and memmap cleanup.
`sharrow/omx.py`	Reimplements OMX splitting with h5py.
`sharrow/omx_reader.py`	Narrows filename handling to h5py inputs.
`sharrow/example_data.py`	Reads example OMX data with h5py.
`sharrow/dataset.py`	Adds h5py-based loading and parallel storage modes.
`pyproject.toml`	Removes the OpenMatrix development dependency.
`envs/testing.yml`	Removes OpenMatrix from testing dependencies.
`envs/development.yml`	Removes OpenMatrix from development dependencies.
`docs/walkthrough/loading-skims.ipynb`	Updates the OMX walkthrough to h5py.
`.github/workflows/run-tests.yml`	Removes OpenMatrix from CI installation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+ if load == "shared":
+ result.shm.release_shared_memory()
+ else:
+ result.shm.delete_shared_memory_files(key)


+ output_paths = []
+ for number, matrix_name in enumerate(matrix_names):
+ output_path = destination / chunk_names[number % len(chunk_names)]
+ output_paths.append(output_path)
+ logger.info(f"writing {matrix_name} to {output_path}")
+ with h5py.File(output_path, "a") as target:
+ _initialize_omx_file(source, target)
+ _copy_dataset(source["data"], target["data"], matrix_name)


- " omx_file.create_carray(\"/lookup\", \"taz\", obj=zones)\n",
- " omx_file.root._v_attrs.SHAPE = np.array([len(zones), len(zones)])"
+ " omx_file.create_dataset(\"lookup/taz\", data=zones)\n",
+ " omx_file.attrs[\"SHAPE\"] = np.array([len(zones), len(zones)])"


Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- · 2026-08-07T19:52:30Z

This pull request removes the use of the openmatrix package throughout the codebase and replaces it with direct usage of the h5py library for reading and writing OMX (Open Matrix) HDF5 files. The change simplifies dependencies, updates documentation and examples, and refactors code to use h5py APIs. Additionally, improvements are made to shared memory handling, including support for custom array storage order and better memory management.

Dependency and Documentation Updates:

Removed openmatrix from all dependency files (envs/development.yml, envs/testing.yml, pyproject.toml, .github/workflows/run-tests.yml) and updated code, documentation, and examples to use h5py directly for OMX file operations. [1][2][3][4][5][6][7][8][9]

Code Refactoring:

Refactored all code that previously used openmatrix to use h5py, including in sharrow/example_data.py, sharrow/omx.py, and sharrow/tests/test_datasets.py. This includes updating file reading/writing logic and dataset creation. [1][2][3][4][5]

OMX File Handling Improvements:

Rewrote the split_omx function in sharrow/omx.py to use h5py for splitting OMX files, added helper functions for copying metadata and datasets, and improved output file organization and lookup management.

Shared Memory Enhancements:

Added support for specifying custom array storage order (e.g., "last-axis-first") in shared memory, improved buffer initialization to be more memory efficient, and enhanced cleanup logic for memory-mapped arrays in sharrow/shared_memory.py. [1][2][3][4][5][6][7][8][9][10]

API and Type Annotations:

Updated function signatures and docstrings to reflect the removal of openmatrix and clarify the use of h5py objects, especially in sharrow/omx_reader.py. [1][2]

Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

Copilot

Pull request overview

Replaces OpenMatrix with direct h5py-based OMX handling and adds optimized eager, shared-memory, and memory-mapped loading.

Changes:

Refactors OMX reading, writing, splitting, and Zarr conversion.
Adds parallel loading and configurable shared-memory array ordering.
Removes OpenMatrix dependencies and updates tests and documentation.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`uv.lock`	Removes OpenMatrix and PyTables dependencies.
`sharrow/translate.py`	Uses the shared OMX reader for Zarr conversion.
`sharrow/tests/test_omx.py`	Adds OMX splitting and conversion tests.
`sharrow/tests/test_datasets.py`	Migrates tests to h5py and covers loading modes.
`sharrow/shared_memory.py`	Adds array ordering and memmap cleanup.
`sharrow/omx.py`	Reimplements OMX splitting with h5py.
`sharrow/omx_reader.py`	Narrows filename handling to h5py inputs.
`sharrow/example_data.py`	Reads example OMX data with h5py.
`sharrow/dataset.py`	Adds h5py-based loading and parallel storage modes.
`pyproject.toml`	Removes the OpenMatrix development dependency.
`envs/testing.yml`	Removes OpenMatrix from testing dependencies.
`envs/development.yml`	Removes OpenMatrix from development dependencies.
`docs/walkthrough/loading-skims.ipynb`	Updates the OMX walkthrough to h5py.
`.github/workflows/run-tests.yml`	Removes OpenMatrix from CI installation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+ if load == "shared":
+ result.shm.release_shared_memory()
+ else:
+ result.shm.delete_shared_memory_files(key)


+ output_paths = []
+ for number, matrix_name in enumerate(matrix_names):
+ output_path = destination / chunk_names[number % len(chunk_names)]
+ output_paths.append(output_path)
+ logger.info(f"writing {matrix_name} to {output_path}")
+ with h5py.File(output_path, "a") as target:
+ _initialize_omx_file(source, target)
+ _copy_dataset(source["data"], target["data"], matrix_name)


- " omx_file.create_carray(\"/lookup\", \"taz\", obj=zones)\n",
- " omx_file.root._v_attrs.SHAPE = np.array([len(zones), len(zones)])"
+ " omx_file.create_dataset(\"lookup/taz\", data=zones)\n",
+ " omx_file.attrs[\"SHAPE\"] = np.array([len(zones), len(zones)])"


Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- · 2026-08-07T19:52:30Z

This pull request removes the use of the openmatrix package throughout the codebase and replaces it with direct usage of the h5py library for reading and writing OMX (Open Matrix) HDF5 files. The change simplifies dependencies, updates documentation and examples, and refactors code to use h5py APIs. Additionally, improvements are made to shared memory handling, including support for custom array storage order and better memory management.

Dependency and Documentation Updates:

Removed openmatrix from all dependency files (envs/development.yml, envs/testing.yml, pyproject.toml, .github/workflows/run-tests.yml) and updated code, documentation, and examples to use h5py directly for OMX file operations. [1][2][3][4][5][6][7][8][9]

Code Refactoring:

Refactored all code that previously used openmatrix to use h5py, including in sharrow/example_data.py, sharrow/omx.py, and sharrow/tests/test_datasets.py. This includes updating file reading/writing logic and dataset creation. [1][2][3][4][5]

OMX File Handling Improvements:

Rewrote the split_omx function in sharrow/omx.py to use h5py for splitting OMX files, added helper functions for copying metadata and datasets, and improved output file organization and lookup management.

Shared Memory Enhancements:

Added support for specifying custom array storage order (e.g., "last-axis-first") in shared memory, improved buffer initialization to be more memory efficient, and enhanced cleanup logic for memory-mapped arrays in sharrow/shared_memory.py. [1][2][3][4][5][6][7][8][9][10]

API and Type Annotations:

Updated function signatures and docstrings to reflect the removal of openmatrix and clarify the use of h5py objects, especially in sharrow/omx_reader.py. [1][2]

Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

Copilot

Pull request overview

Replaces OpenMatrix with direct h5py-based OMX handling and adds optimized eager, shared-memory, and memory-mapped loading.

Changes:

Refactors OMX reading, writing, splitting, and Zarr conversion.
Adds parallel loading and configurable shared-memory array ordering.
Removes OpenMatrix dependencies and updates tests and documentation.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`uv.lock`	Removes OpenMatrix and PyTables dependencies.
`sharrow/translate.py`	Uses the shared OMX reader for Zarr conversion.
`sharrow/tests/test_omx.py`	Adds OMX splitting and conversion tests.
`sharrow/tests/test_datasets.py`	Migrates tests to h5py and covers loading modes.
`sharrow/shared_memory.py`	Adds array ordering and memmap cleanup.
`sharrow/omx.py`	Reimplements OMX splitting with h5py.
`sharrow/omx_reader.py`	Narrows filename handling to h5py inputs.
`sharrow/example_data.py`	Reads example OMX data with h5py.
`sharrow/dataset.py`	Adds h5py-based loading and parallel storage modes.
`pyproject.toml`	Removes the OpenMatrix development dependency.
`envs/testing.yml`	Removes OpenMatrix from testing dependencies.
`envs/development.yml`	Removes OpenMatrix from development dependencies.
`docs/walkthrough/loading-skims.ipynb`	Updates the OMX walkthrough to h5py.
`.github/workflows/run-tests.yml`	Removes OpenMatrix from CI installation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+ if load == "shared":
+ result.shm.release_shared_memory()
+ else:
+ result.shm.delete_shared_memory_files(key)


+ output_paths = []
+ for number, matrix_name in enumerate(matrix_names):
+ output_path = destination / chunk_names[number % len(chunk_names)]
+ output_paths.append(output_path)
+ logger.info(f"writing {matrix_name} to {output_path}")
+ with h5py.File(output_path, "a") as target:
+ _initialize_omx_file(source, target)
+ _copy_dataset(source["data"], target["data"], matrix_name)


- " omx_file.create_carray(\"/lookup\", \"taz\", obj=zones)\n",
- " omx_file.root._v_attrs.SHAPE = np.array([len(zones), len(zones)])"
+ " omx_file.create_dataset(\"lookup/taz\", data=zones)\n",
+ " omx_file.attrs[\"SHAPE\"] = np.array([len(zones), len(zones)])"


Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- · 2026-08-07T19:52:30Z

This pull request removes the use of the openmatrix package throughout the codebase and replaces it with direct usage of the h5py library for reading and writing OMX (Open Matrix) HDF5 files. The change simplifies dependencies, updates documentation and examples, and refactors code to use h5py APIs. Additionally, improvements are made to shared memory handling, including support for custom array storage order and better memory management.

Dependency and Documentation Updates:

Removed openmatrix from all dependency files (envs/development.yml, envs/testing.yml, pyproject.toml, .github/workflows/run-tests.yml) and updated code, documentation, and examples to use h5py directly for OMX file operations. [1][2][3][4][5][6][7][8][9]

Code Refactoring:

Refactored all code that previously used openmatrix to use h5py, including in sharrow/example_data.py, sharrow/omx.py, and sharrow/tests/test_datasets.py. This includes updating file reading/writing logic and dataset creation. [1][2][3][4][5]

OMX File Handling Improvements:

Rewrote the split_omx function in sharrow/omx.py to use h5py for splitting OMX files, added helper functions for copying metadata and datasets, and improved output file organization and lookup management.

Shared Memory Enhancements:

Added support for specifying custom array storage order (e.g., "last-axis-first") in shared memory, improved buffer initialization to be more memory efficient, and enhanced cleanup logic for memory-mapped arrays in sharrow/shared_memory.py. [1][2][3][4][5][6][7][8][9][10]

API and Type Annotations:

Updated function signatures and docstrings to reflect the removal of openmatrix and clarify the use of h5py objects, especially in sharrow/omx_reader.py. [1][2]

Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

Copilot

Pull request overview

Replaces OpenMatrix with direct h5py-based OMX handling and adds optimized eager, shared-memory, and memory-mapped loading.

Changes:

Refactors OMX reading, writing, splitting, and Zarr conversion.
Adds parallel loading and configurable shared-memory array ordering.
Removes OpenMatrix dependencies and updates tests and documentation.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`uv.lock`	Removes OpenMatrix and PyTables dependencies.
`sharrow/translate.py`	Uses the shared OMX reader for Zarr conversion.
`sharrow/tests/test_omx.py`	Adds OMX splitting and conversion tests.
`sharrow/tests/test_datasets.py`	Migrates tests to h5py and covers loading modes.
`sharrow/shared_memory.py`	Adds array ordering and memmap cleanup.
`sharrow/omx.py`	Reimplements OMX splitting with h5py.
`sharrow/omx_reader.py`	Narrows filename handling to h5py inputs.
`sharrow/example_data.py`	Reads example OMX data with h5py.
`sharrow/dataset.py`	Adds h5py-based loading and parallel storage modes.
`pyproject.toml`	Removes the OpenMatrix development dependency.
`envs/testing.yml`	Removes OpenMatrix from testing dependencies.
`envs/development.yml`	Removes OpenMatrix from development dependencies.
`docs/walkthrough/loading-skims.ipynb`	Updates the OMX walkthrough to h5py.
`.github/workflows/run-tests.yml`	Removes OpenMatrix from CI installation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+ if load == "shared":
+ result.shm.release_shared_memory()
+ else:
+ result.shm.delete_shared_memory_files(key)


+ output_paths = []
+ for number, matrix_name in enumerate(matrix_names):
+ output_path = destination / chunk_names[number % len(chunk_names)]
+ output_paths.append(output_path)
+ logger.info(f"writing {matrix_name} to {output_path}")
+ with h5py.File(output_path, "a") as target:
+ _initialize_omx_file(source, target)
+ _copy_dataset(source["data"], target["data"], matrix_name)


- " omx_file.create_carray(\"/lookup\", \"taz\", obj=zones)\n",
- " omx_file.root._v_attrs.SHAPE = np.array([len(zones), len(zones)])"
+ " omx_file.create_dataset(\"lookup/taz\", data=zones)\n",
+ " omx_file.attrs[\"SHAPE\"] = np.array([len(zones), len(zones)])"


Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- · 2026-08-07T19:52:30Z

This pull request removes the use of the openmatrix package throughout the codebase and replaces it with direct usage of the h5py library for reading and writing OMX (Open Matrix) HDF5 files. The change simplifies dependencies, updates documentation and examples, and refactors code to use h5py APIs. Additionally, improvements are made to shared memory handling, including support for custom array storage order and better memory management.

Dependency and Documentation Updates:

Removed openmatrix from all dependency files (envs/development.yml, envs/testing.yml, pyproject.toml, .github/workflows/run-tests.yml) and updated code, documentation, and examples to use h5py directly for OMX file operations. [1][2][3][4][5][6][7][8][9]

Code Refactoring:

Refactored all code that previously used openmatrix to use h5py, including in sharrow/example_data.py, sharrow/omx.py, and sharrow/tests/test_datasets.py. This includes updating file reading/writing logic and dataset creation. [1][2][3][4][5]

OMX File Handling Improvements:

Rewrote the split_omx function in sharrow/omx.py to use h5py for splitting OMX files, added helper functions for copying metadata and datasets, and improved output file organization and lookup management.

Shared Memory Enhancements:

Added support for specifying custom array storage order (e.g., "last-axis-first") in shared memory, improved buffer initialization to be more memory efficient, and enhanced cleanup logic for memory-mapped arrays in sharrow/shared_memory.py. [1][2][3][4][5][6][7][8][9][10]

API and Type Annotations:

Updated function signatures and docstrings to reflect the removal of openmatrix and clarify the use of h5py objects, especially in sharrow/omx_reader.py. [1][2]

Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

Copilot

Pull request overview

Replaces OpenMatrix with direct h5py-based OMX handling and adds optimized eager, shared-memory, and memory-mapped loading.

Changes:

Refactors OMX reading, writing, splitting, and Zarr conversion.
Adds parallel loading and configurable shared-memory array ordering.
Removes OpenMatrix dependencies and updates tests and documentation.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`uv.lock`	Removes OpenMatrix and PyTables dependencies.
`sharrow/translate.py`	Uses the shared OMX reader for Zarr conversion.
`sharrow/tests/test_omx.py`	Adds OMX splitting and conversion tests.
`sharrow/tests/test_datasets.py`	Migrates tests to h5py and covers loading modes.
`sharrow/shared_memory.py`	Adds array ordering and memmap cleanup.
`sharrow/omx.py`	Reimplements OMX splitting with h5py.
`sharrow/omx_reader.py`	Narrows filename handling to h5py inputs.
`sharrow/example_data.py`	Reads example OMX data with h5py.
`sharrow/dataset.py`	Adds h5py-based loading and parallel storage modes.
`pyproject.toml`	Removes the OpenMatrix development dependency.
`envs/testing.yml`	Removes OpenMatrix from testing dependencies.
`envs/development.yml`	Removes OpenMatrix from development dependencies.
`docs/walkthrough/loading-skims.ipynb`	Updates the OMX walkthrough to h5py.
`.github/workflows/run-tests.yml`	Removes OpenMatrix from CI installation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+ if load == "shared":
+ result.shm.release_shared_memory()
+ else:
+ result.shm.delete_shared_memory_files(key)


+ output_paths = []
+ for number, matrix_name in enumerate(matrix_names):
+ output_path = destination / chunk_names[number % len(chunk_names)]
+ output_paths.append(output_path)
+ logger.info(f"writing {matrix_name} to {output_path}")
+ with h5py.File(output_path, "a") as target:
+ _initialize_omx_file(source, target)
+ _copy_dataset(source["data"], target["data"], matrix_name)


- " omx_file.create_carray(\"/lookup\", \"taz\", obj=zones)\n",
- " omx_file.root._v_attrs.SHAPE = np.array([len(zones), len(zones)])"
+ " omx_file.create_dataset(\"lookup/taz\", data=zones)\n",
+ " omx_file.attrs[\"SHAPE\"] = np.array([len(zones), len(zones)])"


Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- · 2026-08-07T19:52:30Z

This pull request removes the use of the openmatrix package throughout the codebase and replaces it with direct usage of the h5py library for reading and writing OMX (Open Matrix) HDF5 files. The change simplifies dependencies, updates documentation and examples, and refactors code to use h5py APIs. Additionally, improvements are made to shared memory handling, including support for custom array storage order and better memory management.

Dependency and Documentation Updates:

Removed openmatrix from all dependency files (envs/development.yml, envs/testing.yml, pyproject.toml, .github/workflows/run-tests.yml) and updated code, documentation, and examples to use h5py directly for OMX file operations. [1][2][3][4][5][6][7][8][9]

Code Refactoring:

Refactored all code that previously used openmatrix to use h5py, including in sharrow/example_data.py, sharrow/omx.py, and sharrow/tests/test_datasets.py. This includes updating file reading/writing logic and dataset creation. [1][2][3][4][5]

OMX File Handling Improvements:

Rewrote the split_omx function in sharrow/omx.py to use h5py for splitting OMX files, added helper functions for copying metadata and datasets, and improved output file organization and lookup management.

Shared Memory Enhancements:

Added support for specifying custom array storage order (e.g., "last-axis-first") in shared memory, improved buffer initialization to be more memory efficient, and enhanced cleanup logic for memory-mapped arrays in sharrow/shared_memory.py. [1][2][3][4][5][6][7][8][9][10]

API and Type Annotations:

Updated function signatures and docstrings to reflect the removal of openmatrix and clarify the use of h5py objects, especially in sharrow/omx_reader.py. [1][2]

Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

Copilot

Pull request overview

Replaces OpenMatrix with direct h5py-based OMX handling and adds optimized eager, shared-memory, and memory-mapped loading.

Changes:

Refactors OMX reading, writing, splitting, and Zarr conversion.
Adds parallel loading and configurable shared-memory array ordering.
Removes OpenMatrix dependencies and updates tests and documentation.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`uv.lock`	Removes OpenMatrix and PyTables dependencies.
`sharrow/translate.py`	Uses the shared OMX reader for Zarr conversion.
`sharrow/tests/test_omx.py`	Adds OMX splitting and conversion tests.
`sharrow/tests/test_datasets.py`	Migrates tests to h5py and covers loading modes.
`sharrow/shared_memory.py`	Adds array ordering and memmap cleanup.
`sharrow/omx.py`	Reimplements OMX splitting with h5py.
`sharrow/omx_reader.py`	Narrows filename handling to h5py inputs.
`sharrow/example_data.py`	Reads example OMX data with h5py.
`sharrow/dataset.py`	Adds h5py-based loading and parallel storage modes.
`pyproject.toml`	Removes the OpenMatrix development dependency.
`envs/testing.yml`	Removes OpenMatrix from testing dependencies.
`envs/development.yml`	Removes OpenMatrix from development dependencies.
`docs/walkthrough/loading-skims.ipynb`	Updates the OMX walkthrough to h5py.
`.github/workflows/run-tests.yml`	Removes OpenMatrix from CI installation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+ if load == "shared":
+ result.shm.release_shared_memory()
+ else:
+ result.shm.delete_shared_memory_files(key)


+ output_paths = []
+ for number, matrix_name in enumerate(matrix_names):
+ output_path = destination / chunk_names[number % len(chunk_names)]
+ output_paths.append(output_path)
+ logger.info(f"writing {matrix_name} to {output_path}")
+ with h5py.File(output_path, "a") as target:
+ _initialize_omx_file(source, target)
+ _copy_dataset(source["data"], target["data"], matrix_name)


- " omx_file.create_carray(\"/lookup\", \"taz\", obj=zones)\n",
- " omx_file.root._v_attrs.SHAPE = np.array([len(zones), len(zones)])"
+ " omx_file.create_dataset(\"lookup/taz\", data=zones)\n",
+ " omx_file.attrs[\"SHAPE\"] = np.array([len(zones), len(zones)])"


Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- · 2026-08-07T19:52:30Z

This pull request removes the use of the openmatrix package throughout the codebase and replaces it with direct usage of the h5py library for reading and writing OMX (Open Matrix) HDF5 files. The change simplifies dependencies, updates documentation and examples, and refactors code to use h5py APIs. Additionally, improvements are made to shared memory handling, including support for custom array storage order and better memory management.

Dependency and Documentation Updates:

Removed openmatrix from all dependency files (envs/development.yml, envs/testing.yml, pyproject.toml, .github/workflows/run-tests.yml) and updated code, documentation, and examples to use h5py directly for OMX file operations. [1][2][3][4][5][6][7][8][9]

Code Refactoring:

Refactored all code that previously used openmatrix to use h5py, including in sharrow/example_data.py, sharrow/omx.py, and sharrow/tests/test_datasets.py. This includes updating file reading/writing logic and dataset creation. [1][2][3][4][5]

OMX File Handling Improvements:

Rewrote the split_omx function in sharrow/omx.py to use h5py for splitting OMX files, added helper functions for copying metadata and datasets, and improved output file organization and lookup management.

Shared Memory Enhancements:

Added support for specifying custom array storage order (e.g., "last-axis-first") in shared memory, improved buffer initialization to be more memory efficient, and enhanced cleanup logic for memory-mapped arrays in sharrow/shared_memory.py. [1][2][3][4][5][6][7][8][9][10]

API and Type Annotations:

Updated function signatures and docstrings to reflect the removal of openmatrix and clarify the use of h5py objects, especially in sharrow/omx_reader.py. [1][2]

Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

Copilot

Pull request overview

Replaces OpenMatrix with direct h5py-based OMX handling and adds optimized eager, shared-memory, and memory-mapped loading.

Changes:

Refactors OMX reading, writing, splitting, and Zarr conversion.
Adds parallel loading and configurable shared-memory array ordering.
Removes OpenMatrix dependencies and updates tests and documentation.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`uv.lock`	Removes OpenMatrix and PyTables dependencies.
`sharrow/translate.py`	Uses the shared OMX reader for Zarr conversion.
`sharrow/tests/test_omx.py`	Adds OMX splitting and conversion tests.
`sharrow/tests/test_datasets.py`	Migrates tests to h5py and covers loading modes.
`sharrow/shared_memory.py`	Adds array ordering and memmap cleanup.
`sharrow/omx.py`	Reimplements OMX splitting with h5py.
`sharrow/omx_reader.py`	Narrows filename handling to h5py inputs.
`sharrow/example_data.py`	Reads example OMX data with h5py.
`sharrow/dataset.py`	Adds h5py-based loading and parallel storage modes.
`pyproject.toml`	Removes the OpenMatrix development dependency.
`envs/testing.yml`	Removes OpenMatrix from testing dependencies.
`envs/development.yml`	Removes OpenMatrix from development dependencies.
`docs/walkthrough/loading-skims.ipynb`	Updates the OMX walkthrough to h5py.
`.github/workflows/run-tests.yml`	Removes OpenMatrix from CI installation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+ if load == "shared":
+ result.shm.release_shared_memory()
+ else:
+ result.shm.delete_shared_memory_files(key)


+ output_paths = []
+ for number, matrix_name in enumerate(matrix_names):
+ output_path = destination / chunk_names[number % len(chunk_names)]
+ output_paths.append(output_path)
+ logger.info(f"writing {matrix_name} to {output_path}")
+ with h5py.File(output_path, "a") as target:
+ _initialize_omx_file(source, target)
+ _copy_dataset(source["data"], target["data"], matrix_name)


- " omx_file.create_carray(\"/lookup\", \"taz\", obj=zones)\n",
- " omx_file.root._v_attrs.SHAPE = np.array([len(zones), len(zones)])"
+ " omx_file.create_dataset(\"lookup/taz\", data=zones)\n",
+ " omx_file.attrs[\"SHAPE\"] = np.array([len(zones), len(zones)])"


Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- · 2026-08-07T19:52:30Z

This pull request removes the use of the openmatrix package throughout the codebase and replaces it with direct usage of the h5py library for reading and writing OMX (Open Matrix) HDF5 files. The change simplifies dependencies, updates documentation and examples, and refactors code to use h5py APIs. Additionally, improvements are made to shared memory handling, including support for custom array storage order and better memory management.

Dependency and Documentation Updates:

Removed openmatrix from all dependency files (envs/development.yml, envs/testing.yml, pyproject.toml, .github/workflows/run-tests.yml) and updated code, documentation, and examples to use h5py directly for OMX file operations. [1][2][3][4][5][6][7][8][9]

Code Refactoring:

Refactored all code that previously used openmatrix to use h5py, including in sharrow/example_data.py, sharrow/omx.py, and sharrow/tests/test_datasets.py. This includes updating file reading/writing logic and dataset creation. [1][2][3][4][5]

OMX File Handling Improvements:

Rewrote the split_omx function in sharrow/omx.py to use h5py for splitting OMX files, added helper functions for copying metadata and datasets, and improved output file organization and lookup management.

Shared Memory Enhancements:

Added support for specifying custom array storage order (e.g., "last-axis-first") in shared memory, improved buffer initialization to be more memory efficient, and enhanced cleanup logic for memory-mapped arrays in sharrow/shared_memory.py. [1][2][3][4][5][6][7][8][9][10]

API and Type Annotations:

Updated function signatures and docstrings to reflect the removal of openmatrix and clarify the use of h5py objects, especially in sharrow/omx_reader.py. [1][2]

Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

Copilot

Pull request overview

Replaces OpenMatrix with direct h5py-based OMX handling and adds optimized eager, shared-memory, and memory-mapped loading.

Changes:

Refactors OMX reading, writing, splitting, and Zarr conversion.
Adds parallel loading and configurable shared-memory array ordering.
Removes OpenMatrix dependencies and updates tests and documentation.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`uv.lock`	Removes OpenMatrix and PyTables dependencies.
`sharrow/translate.py`	Uses the shared OMX reader for Zarr conversion.
`sharrow/tests/test_omx.py`	Adds OMX splitting and conversion tests.
`sharrow/tests/test_datasets.py`	Migrates tests to h5py and covers loading modes.
`sharrow/shared_memory.py`	Adds array ordering and memmap cleanup.
`sharrow/omx.py`	Reimplements OMX splitting with h5py.
`sharrow/omx_reader.py`	Narrows filename handling to h5py inputs.
`sharrow/example_data.py`	Reads example OMX data with h5py.
`sharrow/dataset.py`	Adds h5py-based loading and parallel storage modes.
`pyproject.toml`	Removes the OpenMatrix development dependency.
`envs/testing.yml`	Removes OpenMatrix from testing dependencies.
`envs/development.yml`	Removes OpenMatrix from development dependencies.
`docs/walkthrough/loading-skims.ipynb`	Updates the OMX walkthrough to h5py.
`.github/workflows/run-tests.yml`	Removes OpenMatrix from CI installation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+ if load == "shared":
+ result.shm.release_shared_memory()
+ else:
+ result.shm.delete_shared_memory_files(key)


+ output_paths = []
+ for number, matrix_name in enumerate(matrix_names):
+ output_path = destination / chunk_names[number % len(chunk_names)]
+ output_paths.append(output_path)
+ logger.info(f"writing {matrix_name} to {output_path}")
+ with h5py.File(output_path, "a") as target:
+ _initialize_omx_file(source, target)
+ _copy_dataset(source["data"], target["data"], matrix_name)


- " omx_file.create_carray(\"/lookup\", \"taz\", obj=zones)\n",
- " omx_file.root._v_attrs.SHAPE = np.array([len(zones), len(zones)])"
+ " omx_file.create_dataset(\"lookup/taz\", data=zones)\n",
+ " omx_file.attrs[\"SHAPE\"] = np.array([len(zones), len(zones)])"


Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- added 5 commits August 5, 2026 23:33

h5py replace openmatrix entirely
c816dc7

faster read speed
bcf8e98

Support filename-bearing OMX handles
3453dcc
Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Release memmap files before cleanup
9d82dcf

Parallelize OMX reloads for shared-memory datasets
bfcf33f
Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

jpn-- requested a balanced review from Copilot August 11, 2026 21:42

Copilotstarted reviewing on behalf of jpn--August 11, 2026 21:42 View session

CopilotAI reviewed Aug 11, 2026
View reviewed changes

jpn-- added 3 commits August 19, 2026 12:23

Fix OMX splitting and memmap cleanup
52895a2
Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Improve parallel OMX loading and compatibility
f739be5
Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Improve OMX loading portability and memory handling
5e238a8
Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

jpn-- mentioned this pull request Aug 24, 2026
Support reading skims from Parquet files, in addition to OMX ActivitySim/activitysim#1101
Open

Add reproducible MTC OMX benchmark
b024fb3
Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- added 5 commits August 5, 2026 23:33

h5py replace openmatrix entirely
c816dc7

faster read speed
bcf8e98

Support filename-bearing OMX handles
3453dcc
Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Release memmap files before cleanup
9d82dcf

Parallelize OMX reloads for shared-memory datasets
bfcf33f
Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

jpn-- requested a balanced review from Copilot August 11, 2026 21:42

Copilotstarted reviewing on behalf of jpn--August 11, 2026 21:42 View session

CopilotAI reviewed Aug 11, 2026
View reviewed changes

jpn-- added 3 commits August 19, 2026 12:23

Fix OMX splitting and memmap cleanup
52895a2
Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Improve parallel OMX loading and compatibility
f739be5
Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Improve OMX loading portability and memory handling
5e238a8
Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

jpn-- mentioned this pull request Aug 24, 2026
Support reading skims from Parquet files, in addition to OMX ActivitySim/activitysim#1101
Open

Add reproducible MTC OMX benchmark
b024fb3
Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- added 5 commits August 5, 2026 23:33

h5py replace openmatrix entirely
c816dc7

faster read speed
bcf8e98

Support filename-bearing OMX handles
3453dcc
Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Release memmap files before cleanup
9d82dcf

Parallelize OMX reloads for shared-memory datasets
bfcf33f
Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

jpn-- requested a balanced review from Copilot August 11, 2026 21:42

Copilotstarted reviewing on behalf of jpn--August 11, 2026 21:42 View session

CopilotAI reviewed Aug 11, 2026
View reviewed changes

jpn-- added 3 commits August 19, 2026 12:23

Fix OMX splitting and memmap cleanup
52895a2
Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Improve parallel OMX loading and compatibility
f739be5
Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Improve OMX loading portability and memory handling
5e238a8
Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

jpn-- mentioned this pull request Aug 24, 2026
Support reading skims from Parquet files, in addition to OMX ActivitySim/activitysim#1101
Open

Add reproducible MTC OMX benchmark
b024fb3
Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- added 5 commits August 5, 2026 23:33

h5py replace openmatrix entirely
c816dc7

faster read speed
bcf8e98

Support filename-bearing OMX handles
3453dcc
Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Release memmap files before cleanup
9d82dcf

Parallelize OMX reloads for shared-memory datasets
bfcf33f
Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

jpn-- requested a balanced review from Copilot August 11, 2026 21:42

Copilotstarted reviewing on behalf of jpn--August 11, 2026 21:42 View session

CopilotAI reviewed Aug 11, 2026
View reviewed changes

jpn-- added 3 commits August 19, 2026 12:23

Fix OMX splitting and memmap cleanup
52895a2
Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Improve parallel OMX loading and compatibility
f739be5
Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Improve OMX loading portability and memory handling
5e238a8
Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

jpn-- mentioned this pull request Aug 24, 2026
Support reading skims from Parquet files, in addition to OMX ActivitySim/activitysim#1101
Open

Add reproducible MTC OMX benchmark
b024fb3
Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- added 5 commits August 5, 2026 23:33

h5py replace openmatrix entirely
c816dc7

faster read speed
bcf8e98

Support filename-bearing OMX handles
3453dcc
Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Release memmap files before cleanup
9d82dcf

Parallelize OMX reloads for shared-memory datasets
bfcf33f
Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

jpn-- requested a balanced review from Copilot August 11, 2026 21:42

Copilotstarted reviewing on behalf of jpn--August 11, 2026 21:42 View session

CopilotAI reviewed Aug 11, 2026
View reviewed changes

jpn-- added 3 commits August 19, 2026 12:23

Fix OMX splitting and memmap cleanup
52895a2
Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Improve parallel OMX loading and compatibility
f739be5
Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Improve OMX loading portability and memory handling
5e238a8
Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

jpn-- mentioned this pull request Aug 24, 2026
Support reading skims from Parquet files, in addition to OMX ActivitySim/activitysim#1101
Open

Add reproducible MTC OMX benchmark
b024fb3
Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- added 5 commits August 5, 2026 23:33

h5py replace openmatrix entirely
c816dc7

faster read speed
bcf8e98

Support filename-bearing OMX handles
3453dcc
Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Release memmap files before cleanup
9d82dcf

Parallelize OMX reloads for shared-memory datasets
bfcf33f
Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

jpn-- requested a balanced review from Copilot August 11, 2026 21:42

Copilotstarted reviewing on behalf of jpn--August 11, 2026 21:42 View session

CopilotAI reviewed Aug 11, 2026
View reviewed changes

jpn-- added 3 commits August 19, 2026 12:23

Fix OMX splitting and memmap cleanup
52895a2
Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Improve parallel OMX loading and compatibility
f739be5
Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Improve OMX loading portability and memory handling
5e238a8
Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

jpn-- mentioned this pull request Aug 24, 2026
Support reading skims from Parquet files, in addition to OMX ActivitySim/activitysim#1101
Open

Add reproducible MTC OMX benchmark
b024fb3
Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- added 5 commits August 5, 2026 23:33

h5py replace openmatrix entirely
c816dc7

faster read speed
bcf8e98

Support filename-bearing OMX handles
3453dcc
Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Release memmap files before cleanup
9d82dcf

Parallelize OMX reloads for shared-memory datasets
bfcf33f
Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

jpn-- requested a balanced review from Copilot August 11, 2026 21:42

Copilotstarted reviewing on behalf of jpn--August 11, 2026 21:42 View session

CopilotAI reviewed Aug 11, 2026
View reviewed changes

jpn-- added 3 commits August 19, 2026 12:23

Fix OMX splitting and memmap cleanup
52895a2
Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Improve parallel OMX loading and compatibility
f739be5
Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Improve OMX loading portability and memory handling
5e238a8
Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

jpn-- mentioned this pull request Aug 24, 2026
Support reading skims from Parquet files, in addition to OMX ActivitySim/activitysim#1101
Open

Add reproducible MTC OMX benchmark
b024fb3
Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

jpn-- added 5 commits August 5, 2026 23:33

h5py replace openmatrix entirely
c816dc7

faster read speed
bcf8e98

Support filename-bearing OMX handles
3453dcc
Reopen compatible OMX wrappers through h5py while preserving caller ownership. Add coverage for 2D, 3D, and reload workflows.

Release memmap files before cleanup
9d82dcf

Parallelize OMX reloads for shared-memory datasets
bfcf33f
Add configurable worker support while preserving last-source precedence and serial fallbacks. Validate worker settings and cover parallel reload behavior.

jpn-- requested a balanced review from Copilot August 11, 2026 21:42

Copilotstarted reviewing on behalf of jpn--August 11, 2026 21:42 View session

CopilotAI reviewed Aug 11, 2026
View reviewed changes

jpn-- added 3 commits August 19, 2026 12:23

Fix OMX splitting and memmap cleanup
52895a2
Truncate split outputs on reruns to remove stale data. Release failed memmaps before deleting their backing files.

Improve parallel OMX loading and compatibility
f739be5
Share a bounded decoder pool across matrix loads and support dtype conversion into noncontiguous outputs. Infer OMX dimensions when the SHAPE attribute is missing.

Improve OMX loading portability and memory handling
5e238a8
Make HDF5 filter plugins optional, use platform-safe parallel loading on Windows, and clean up failed memmap allocations. Add portability tests, documentation, and an isolated OMX memory benchmark.

jpn-- mentioned this pull request Aug 24, 2026
Support reading skims from Parquet files, in addition to OMX ActivitySim/activitysim#1101
Open

Add reproducible MTC OMX benchmark
b024fb3
Compare release and development loaders in isolated environments, tracking runtime and peak memory across loading strategies. Document setup, caching, source selection, and benchmark options.

Uh oh!

Conversation

jpn-- commented Aug 7, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jpn-- commented Aug 7, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jpn-- commented Aug 7, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jpn-- commented Aug 7, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jpn-- commented Aug 7, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels