Add Pyodide support and CI jobs for Zarr

[agriyakhetarpal]

Important

Please view #1903 instead

[agriyakhetarpal]

I added a couple of review comments below to ease the code review process for project maintainers and core developers. Besides that, there is another test case that fails and I have had a hard time debugging it so far. I have managed to get down to two failing tests across the entire test suite by skipping tests where applicable using @pytest.mark.skipif() decorators.

Additionally, I suppose that proper updates to the Zarr documentation and a CHANGELOG update would be required to inform people about the limitations of running Zarr in a Pyodide runtime and which features are not supported. I would be happy to make those changes as I am directed to; and edit the docstrings for the relevant sections in the public API, in subsequent commits and review comments as necessary.

[agriyakhetarpal]

I am unaware of the reason why numcodecs.tests.common cannot import this to the namespace – for this while, I have added this list of strings verbatim here. I will investigate the cause for the error.

[agriyakhetarpal]

This is one of the tests I would like to receive help on – I am not entirely aware of the differences in the Pyodide (Emscripten MemFS to be specific) and the CPython file systems, and I see that retrieving the 0o777 octal (execute permissions) seems to bring some insights. I have marked this as xfail for now.

[agriyakhetarpal]

Both of these lines currently do not work for some reason – I am unable to import MsgPack from numcodecs even after I added them, which is why I am explicitly installing msgpack inside the Pyodide virtual environment for now, which seems to make this work. I will take a look at why these options do not work as I continue to work on this PR.

[jhamman]

@agriyakhetarpal -- thanks for this PR. I want to point you to #1777 and see if we can focus your efforts toward the v3 branch rather than main.

[agriyakhetarpal]

Thank you for the redirection, @jhamman. I shall change the base branch of the PR, fix any conflicts, and rebase as necessary in a moment.

[agriyakhetarpal]

Okay, wayyy too many conflicts! 😄 I think this will take me a while, but should be doable without repeating these changes as a separate PR.

[agriyakhetarpal]

I cherry-picked and backed up my commits in a different branch and a stash, I'll recover them from there. I think a separate PR would be better, I'll copy what I documented over to the other one.

[agriyakhetarpal]

I have a suggestion which I hope is not unsolicited: please make the v3 branch the default branch for the repository in the GitHub settings? It would be helpful in the context of new contributors – my Git TUI used the main branch to create a new branch. :)

Please ignore if this has been evaluated before or if there is a nuance to the discussion that I might have missed as a newcomer who has not contributed to Zarr before. Thanks!

[agriyakhetarpal]

Important

Please view #1903 instead

[agriyakhetarpal]

I added a couple of review comments below to ease the code review process for project maintainers and core developers. Besides that, there is another test case that fails and I have had a hard time debugging it so far. I have managed to get down to two failing tests across the entire test suite by skipping tests where applicable using @pytest.mark.skipif() decorators.

Additionally, I suppose that proper updates to the Zarr documentation and a CHANGELOG update would be required to inform people about the limitations of running Zarr in a Pyodide runtime and which features are not supported. I would be happy to make those changes as I am directed to; and edit the docstrings for the relevant sections in the public API, in subsequent commits and review comments as necessary.

[agriyakhetarpal]

I am unaware of the reason why numcodecs.tests.common cannot import this to the namespace – for this while, I have added this list of strings verbatim here. I will investigate the cause for the error.

[agriyakhetarpal]

This is one of the tests I would like to receive help on – I am not entirely aware of the differences in the Pyodide (Emscripten MemFS to be specific) and the CPython file systems, and I see that retrieving the 0o777 octal (execute permissions) seems to bring some insights. I have marked this as xfail for now.

[agriyakhetarpal]

Both of these lines currently do not work for some reason – I am unable to import MsgPack from numcodecs even after I added them, which is why I am explicitly installing msgpack inside the Pyodide virtual environment for now, which seems to make this work. I will take a look at why these options do not work as I continue to work on this PR.

[jhamman]

@agriyakhetarpal -- thanks for this PR. I want to point you to #1777 and see if we can focus your efforts toward the v3 branch rather than main.

[agriyakhetarpal]

Thank you for the redirection, @jhamman. I shall change the base branch of the PR, fix any conflicts, and rebase as necessary in a moment.

[agriyakhetarpal]

Okay, wayyy too many conflicts! 😄 I think this will take me a while, but should be doable without repeating these changes as a separate PR.

[agriyakhetarpal]

I cherry-picked and backed up my commits in a different branch and a stash, I'll recover them from there. I think a separate PR would be better, I'll copy what I documented over to the other one.

[agriyakhetarpal]

I have a suggestion which I hope is not unsolicited: please make the v3 branch the default branch for the repository in the GitHub settings? It would be helpful in the context of new contributors – my Git TUI used the main branch to create a new branch. :)

Please ignore if this has been evaluated before or if there is a nuance to the discussion that I might have missed as a newcomer who has not contributed to Zarr before. Thanks!

[agriyakhetarpal]

Important

Please view #1903 instead

[agriyakhetarpal]

I added a couple of review comments below to ease the code review process for project maintainers and core developers. Besides that, there is another test case that fails and I have had a hard time debugging it so far. I have managed to get down to two failing tests across the entire test suite by skipping tests where applicable using @pytest.mark.skipif() decorators.

Additionally, I suppose that proper updates to the Zarr documentation and a CHANGELOG update would be required to inform people about the limitations of running Zarr in a Pyodide runtime and which features are not supported. I would be happy to make those changes as I am directed to; and edit the docstrings for the relevant sections in the public API, in subsequent commits and review comments as necessary.

[agriyakhetarpal]

I am unaware of the reason why numcodecs.tests.common cannot import this to the namespace – for this while, I have added this list of strings verbatim here. I will investigate the cause for the error.

[agriyakhetarpal]

This is one of the tests I would like to receive help on – I am not entirely aware of the differences in the Pyodide (Emscripten MemFS to be specific) and the CPython file systems, and I see that retrieving the 0o777 octal (execute permissions) seems to bring some insights. I have marked this as xfail for now.

[agriyakhetarpal]

Both of these lines currently do not work for some reason – I am unable to import MsgPack from numcodecs even after I added them, which is why I am explicitly installing msgpack inside the Pyodide virtual environment for now, which seems to make this work. I will take a look at why these options do not work as I continue to work on this PR.

[jhamman]

@agriyakhetarpal -- thanks for this PR. I want to point you to #1777 and see if we can focus your efforts toward the v3 branch rather than main.

[agriyakhetarpal]

Thank you for the redirection, @jhamman. I shall change the base branch of the PR, fix any conflicts, and rebase as necessary in a moment.

[agriyakhetarpal]

Okay, wayyy too many conflicts! 😄 I think this will take me a while, but should be doable without repeating these changes as a separate PR.

[agriyakhetarpal]

I cherry-picked and backed up my commits in a different branch and a stash, I'll recover them from there. I think a separate PR would be better, I'll copy what I documented over to the other one.

[agriyakhetarpal]

I have a suggestion which I hope is not unsolicited: please make the v3 branch the default branch for the repository in the GitHub settings? It would be helpful in the context of new contributors – my Git TUI used the main branch to create a new branch. :)

Please ignore if this has been evaluated before or if there is a nuance to the discussion that I might have missed as a newcomer who has not contributed to Zarr before. Thanks!

[agriyakhetarpal]

Important

Please view #1903 instead

[agriyakhetarpal]

I added a couple of review comments below to ease the code review process for project maintainers and core developers. Besides that, there is another test case that fails and I have had a hard time debugging it so far. I have managed to get down to two failing tests across the entire test suite by skipping tests where applicable using @pytest.mark.skipif() decorators.

Additionally, I suppose that proper updates to the Zarr documentation and a CHANGELOG update would be required to inform people about the limitations of running Zarr in a Pyodide runtime and which features are not supported. I would be happy to make those changes as I am directed to; and edit the docstrings for the relevant sections in the public API, in subsequent commits and review comments as necessary.

[agriyakhetarpal]

I am unaware of the reason why numcodecs.tests.common cannot import this to the namespace – for this while, I have added this list of strings verbatim here. I will investigate the cause for the error.

[agriyakhetarpal]

This is one of the tests I would like to receive help on – I am not entirely aware of the differences in the Pyodide (Emscripten MemFS to be specific) and the CPython file systems, and I see that retrieving the 0o777 octal (execute permissions) seems to bring some insights. I have marked this as xfail for now.

[agriyakhetarpal]

Both of these lines currently do not work for some reason – I am unable to import MsgPack from numcodecs even after I added them, which is why I am explicitly installing msgpack inside the Pyodide virtual environment for now, which seems to make this work. I will take a look at why these options do not work as I continue to work on this PR.

[jhamman]

@agriyakhetarpal -- thanks for this PR. I want to point you to #1777 and see if we can focus your efforts toward the v3 branch rather than main.

[agriyakhetarpal]

Thank you for the redirection, @jhamman. I shall change the base branch of the PR, fix any conflicts, and rebase as necessary in a moment.

[agriyakhetarpal]

Okay, wayyy too many conflicts! 😄 I think this will take me a while, but should be doable without repeating these changes as a separate PR.

[agriyakhetarpal]

I cherry-picked and backed up my commits in a different branch and a stash, I'll recover them from there. I think a separate PR would be better, I'll copy what I documented over to the other one.

[agriyakhetarpal]

I have a suggestion which I hope is not unsolicited: please make the v3 branch the default branch for the repository in the GitHub settings? It would be helpful in the context of new contributors – my Git TUI used the main branch to create a new branch. :)

Please ignore if this has been evaluated before or if there is a nuance to the discussion that I might have missed as a newcomer who has not contributed to Zarr before. Thanks!

[agriyakhetarpal]

Important

Please view #1903 instead

[agriyakhetarpal]

I added a couple of review comments below to ease the code review process for project maintainers and core developers. Besides that, there is another test case that fails and I have had a hard time debugging it so far. I have managed to get down to two failing tests across the entire test suite by skipping tests where applicable using @pytest.mark.skipif() decorators.

Additionally, I suppose that proper updates to the Zarr documentation and a CHANGELOG update would be required to inform people about the limitations of running Zarr in a Pyodide runtime and which features are not supported. I would be happy to make those changes as I am directed to; and edit the docstrings for the relevant sections in the public API, in subsequent commits and review comments as necessary.

[agriyakhetarpal]

I am unaware of the reason why numcodecs.tests.common cannot import this to the namespace – for this while, I have added this list of strings verbatim here. I will investigate the cause for the error.

[agriyakhetarpal]

This is one of the tests I would like to receive help on – I am not entirely aware of the differences in the Pyodide (Emscripten MemFS to be specific) and the CPython file systems, and I see that retrieving the 0o777 octal (execute permissions) seems to bring some insights. I have marked this as xfail for now.

[agriyakhetarpal]

Both of these lines currently do not work for some reason – I am unable to import MsgPack from numcodecs even after I added them, which is why I am explicitly installing msgpack inside the Pyodide virtual environment for now, which seems to make this work. I will take a look at why these options do not work as I continue to work on this PR.

[jhamman]

@agriyakhetarpal -- thanks for this PR. I want to point you to #1777 and see if we can focus your efforts toward the v3 branch rather than main.

[agriyakhetarpal]

Thank you for the redirection, @jhamman. I shall change the base branch of the PR, fix any conflicts, and rebase as necessary in a moment.

[agriyakhetarpal]

Okay, wayyy too many conflicts! 😄 I think this will take me a while, but should be doable without repeating these changes as a separate PR.

[agriyakhetarpal]

I cherry-picked and backed up my commits in a different branch and a stash, I'll recover them from there. I think a separate PR would be better, I'll copy what I documented over to the other one.

[agriyakhetarpal]

I have a suggestion which I hope is not unsolicited: please make the v3 branch the default branch for the repository in the GitHub settings? It would be helpful in the context of new contributors – my Git TUI used the main branch to create a new branch. :)

Please ignore if this has been evaluated before or if there is a nuance to the discussion that I might have missed as a newcomer who has not contributed to Zarr before. Thanks!

[agriyakhetarpal]

Important

Please view #1903 instead

[agriyakhetarpal]

I added a couple of review comments below to ease the code review process for project maintainers and core developers. Besides that, there is another test case that fails and I have had a hard time debugging it so far. I have managed to get down to two failing tests across the entire test suite by skipping tests where applicable using @pytest.mark.skipif() decorators.

Additionally, I suppose that proper updates to the Zarr documentation and a CHANGELOG update would be required to inform people about the limitations of running Zarr in a Pyodide runtime and which features are not supported. I would be happy to make those changes as I am directed to; and edit the docstrings for the relevant sections in the public API, in subsequent commits and review comments as necessary.

[agriyakhetarpal]

I am unaware of the reason why numcodecs.tests.common cannot import this to the namespace – for this while, I have added this list of strings verbatim here. I will investigate the cause for the error.

[agriyakhetarpal]

This is one of the tests I would like to receive help on – I am not entirely aware of the differences in the Pyodide (Emscripten MemFS to be specific) and the CPython file systems, and I see that retrieving the 0o777 octal (execute permissions) seems to bring some insights. I have marked this as xfail for now.

[agriyakhetarpal]

Both of these lines currently do not work for some reason – I am unable to import MsgPack from numcodecs even after I added them, which is why I am explicitly installing msgpack inside the Pyodide virtual environment for now, which seems to make this work. I will take a look at why these options do not work as I continue to work on this PR.

[jhamman]

@agriyakhetarpal -- thanks for this PR. I want to point you to #1777 and see if we can focus your efforts toward the v3 branch rather than main.

[agriyakhetarpal]

Thank you for the redirection, @jhamman. I shall change the base branch of the PR, fix any conflicts, and rebase as necessary in a moment.

[agriyakhetarpal]

Okay, wayyy too many conflicts! 😄 I think this will take me a while, but should be doable without repeating these changes as a separate PR.

[agriyakhetarpal]

I cherry-picked and backed up my commits in a different branch and a stash, I'll recover them from there. I think a separate PR would be better, I'll copy what I documented over to the other one.

[agriyakhetarpal]

I have a suggestion which I hope is not unsolicited: please make the v3 branch the default branch for the repository in the GitHub settings? It would be helpful in the context of new contributors – my Git TUI used the main branch to create a new branch. :)

Please ignore if this has been evaluated before or if there is a nuance to the discussion that I might have missed as a newcomer who has not contributed to Zarr before. Thanks!

[agriyakhetarpal]

Important

Please view #1903 instead

[agriyakhetarpal]

I added a couple of review comments below to ease the code review process for project maintainers and core developers. Besides that, there is another test case that fails and I have had a hard time debugging it so far. I have managed to get down to two failing tests across the entire test suite by skipping tests where applicable using @pytest.mark.skipif() decorators.

Additionally, I suppose that proper updates to the Zarr documentation and a CHANGELOG update would be required to inform people about the limitations of running Zarr in a Pyodide runtime and which features are not supported. I would be happy to make those changes as I am directed to; and edit the docstrings for the relevant sections in the public API, in subsequent commits and review comments as necessary.

[agriyakhetarpal]

I am unaware of the reason why numcodecs.tests.common cannot import this to the namespace – for this while, I have added this list of strings verbatim here. I will investigate the cause for the error.

[agriyakhetarpal]

This is one of the tests I would like to receive help on – I am not entirely aware of the differences in the Pyodide (Emscripten MemFS to be specific) and the CPython file systems, and I see that retrieving the 0o777 octal (execute permissions) seems to bring some insights. I have marked this as xfail for now.

[agriyakhetarpal]

Both of these lines currently do not work for some reason – I am unable to import MsgPack from numcodecs even after I added them, which is why I am explicitly installing msgpack inside the Pyodide virtual environment for now, which seems to make this work. I will take a look at why these options do not work as I continue to work on this PR.

[jhamman]

@agriyakhetarpal -- thanks for this PR. I want to point you to #1777 and see if we can focus your efforts toward the v3 branch rather than main.

[agriyakhetarpal]

Thank you for the redirection, @jhamman. I shall change the base branch of the PR, fix any conflicts, and rebase as necessary in a moment.

[agriyakhetarpal]

Okay, wayyy too many conflicts! 😄 I think this will take me a while, but should be doable without repeating these changes as a separate PR.

[agriyakhetarpal]

I cherry-picked and backed up my commits in a different branch and a stash, I'll recover them from there. I think a separate PR would be better, I'll copy what I documented over to the other one.

[agriyakhetarpal]

I have a suggestion which I hope is not unsolicited: please make the v3 branch the default branch for the repository in the GitHub settings? It would be helpful in the context of new contributors – my Git TUI used the main branch to create a new branch. :)

Please ignore if this has been evaluated before or if there is a nuance to the discussion that I might have missed as a newcomer who has not contributed to Zarr before. Thanks!

[agriyakhetarpal]

Important

Please view #1903 instead

[agriyakhetarpal]

I added a couple of review comments below to ease the code review process for project maintainers and core developers. Besides that, there is another test case that fails and I have had a hard time debugging it so far. I have managed to get down to two failing tests across the entire test suite by skipping tests where applicable using @pytest.mark.skipif() decorators.

Additionally, I suppose that proper updates to the Zarr documentation and a CHANGELOG update would be required to inform people about the limitations of running Zarr in a Pyodide runtime and which features are not supported. I would be happy to make those changes as I am directed to; and edit the docstrings for the relevant sections in the public API, in subsequent commits and review comments as necessary.

[agriyakhetarpal]

I am unaware of the reason why numcodecs.tests.common cannot import this to the namespace – for this while, I have added this list of strings verbatim here. I will investigate the cause for the error.

[agriyakhetarpal]

This is one of the tests I would like to receive help on – I am not entirely aware of the differences in the Pyodide (Emscripten MemFS to be specific) and the CPython file systems, and I see that retrieving the 0o777 octal (execute permissions) seems to bring some insights. I have marked this as xfail for now.

[agriyakhetarpal]

Both of these lines currently do not work for some reason – I am unable to import MsgPack from numcodecs even after I added them, which is why I am explicitly installing msgpack inside the Pyodide virtual environment for now, which seems to make this work. I will take a look at why these options do not work as I continue to work on this PR.

[jhamman]

@agriyakhetarpal -- thanks for this PR. I want to point you to #1777 and see if we can focus your efforts toward the v3 branch rather than main.

[agriyakhetarpal]

Thank you for the redirection, @jhamman. I shall change the base branch of the PR, fix any conflicts, and rebase as necessary in a moment.

[agriyakhetarpal]

Okay, wayyy too many conflicts! 😄 I think this will take me a while, but should be doable without repeating these changes as a separate PR.

[agriyakhetarpal]

I cherry-picked and backed up my commits in a different branch and a stash, I'll recover them from there. I think a separate PR would be better, I'll copy what I documented over to the other one.

[agriyakhetarpal]

I have a suggestion which I hope is not unsolicited: please make the v3 branch the default branch for the repository in the GitHub settings? It would be helpful in the context of new contributors – my Git TUI used the main branch to create a new branch. :)

Please ignore if this has been evaluated before or if there is a nuance to the discussion that I might have missed as a newcomer who has not contributed to Zarr before. Thanks!