Use donfig for V3 configuration

[maxrjones]

Progress towards the proposal in #1772 to use Dask style config (donfig) for configuration in v3.

TODO:

• Migrate sync configuration to donfig
• Migrate runtime configuration to donfig
• Add tests for setting configuration
• Add docs for listing and setting configuration

[maxrjones]

@jhamman fyi I've gotten as far as I can right now on the configuration and will be offline until wednesday. I started working on the byte order in 5de1db2, but while the tests pass I don't think it's fully correct which is why it's left out of this PR.

You're welcome to take over this PR or close if you think it'd be better to make progress separately. I'm free Wednesday 5/15 from 1-3 PM ET to work on this next if it's still an open issue at that time.

[jhamman]

@normanrz - I picked up where @maxrjones left off on this one. Things are going well but I have one design question for you.

The question is what to do with the order config option. For now, I've added it to the ArraySpec object which works but, to some extent, mixes array metadata w/ array runtime options. The relevant constraints that I see are:

1. We want to decide on an order at the time we create the AsyncArray object. From then on, all chunks and output arrays should use this. You'll notice a new attribute AsyncArray.order defined in the __init__ method. This is important because we don't wouldn't want the following to change the order:

   array=AsyncArray.open(...)
   config.set({'array.order': 'C'}}
   a1=array.getitem(...)
   config.set({'array.order': 'F'}}
   a2=array.getitem(...)
   asserta1.order==a2.order

2. order does not make sense to include in the ArrayMetadata. It would be convenient to attach to that object but I don't think its the right choice -- we would have to special case serialization of that object so 👎 .

What do you think about the approach I've taken? Do you have other suggestions?

[normanrz]

I like it! I think ArraySpec is a good place to put it, because it is the specification of how an array shall be constructed. That makes sense to me!

[normanrz]

Suggested change

	order="C",# TODO!!!!
	order="C",

A hard-coded C is fine here, because it is just the shard index. It doesn't surface into user code.

[pep8speaks]

Hello @maxrjones! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file src/zarr/codecs/sharding.py:

Line 624:101: E501 line too long (102 > 100 characters)

[maxrjones]

Progress towards the proposal in #1772 to use Dask style config (donfig) for configuration in v3.

TODO:

• Migrate sync configuration to donfig
• Migrate runtime configuration to donfig
• Add tests for setting configuration
• Add docs for listing and setting configuration

[maxrjones]

@jhamman fyi I've gotten as far as I can right now on the configuration and will be offline until wednesday. I started working on the byte order in 5de1db2, but while the tests pass I don't think it's fully correct which is why it's left out of this PR.

You're welcome to take over this PR or close if you think it'd be better to make progress separately. I'm free Wednesday 5/15 from 1-3 PM ET to work on this next if it's still an open issue at that time.

[jhamman]

@normanrz - I picked up where @maxrjones left off on this one. Things are going well but I have one design question for you.

The question is what to do with the order config option. For now, I've added it to the ArraySpec object which works but, to some extent, mixes array metadata w/ array runtime options. The relevant constraints that I see are:

1. We want to decide on an order at the time we create the AsyncArray object. From then on, all chunks and output arrays should use this. You'll notice a new attribute AsyncArray.order defined in the __init__ method. This is important because we don't wouldn't want the following to change the order:

   array=AsyncArray.open(...)
   config.set({'array.order': 'C'}}
   a1=array.getitem(...)
   config.set({'array.order': 'F'}}
   a2=array.getitem(...)
   asserta1.order==a2.order

2. order does not make sense to include in the ArrayMetadata. It would be convenient to attach to that object but I don't think its the right choice -- we would have to special case serialization of that object so 👎 .

What do you think about the approach I've taken? Do you have other suggestions?

[normanrz]

I like it! I think ArraySpec is a good place to put it, because it is the specification of how an array shall be constructed. That makes sense to me!

[normanrz]

Suggested change

	order="C",# TODO!!!!
	order="C",

A hard-coded C is fine here, because it is just the shard index. It doesn't surface into user code.

[pep8speaks]

Hello @maxrjones! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file src/zarr/codecs/sharding.py:

Line 624:101: E501 line too long (102 > 100 characters)

[maxrjones]

Progress towards the proposal in #1772 to use Dask style config (donfig) for configuration in v3.

TODO:

• Migrate sync configuration to donfig
• Migrate runtime configuration to donfig
• Add tests for setting configuration
• Add docs for listing and setting configuration

[maxrjones]

@jhamman fyi I've gotten as far as I can right now on the configuration and will be offline until wednesday. I started working on the byte order in 5de1db2, but while the tests pass I don't think it's fully correct which is why it's left out of this PR.

You're welcome to take over this PR or close if you think it'd be better to make progress separately. I'm free Wednesday 5/15 from 1-3 PM ET to work on this next if it's still an open issue at that time.

[jhamman]

@normanrz - I picked up where @maxrjones left off on this one. Things are going well but I have one design question for you.

The question is what to do with the order config option. For now, I've added it to the ArraySpec object which works but, to some extent, mixes array metadata w/ array runtime options. The relevant constraints that I see are:

1. We want to decide on an order at the time we create the AsyncArray object. From then on, all chunks and output arrays should use this. You'll notice a new attribute AsyncArray.order defined in the __init__ method. This is important because we don't wouldn't want the following to change the order:

   array=AsyncArray.open(...)
   config.set({'array.order': 'C'}}
   a1=array.getitem(...)
   config.set({'array.order': 'F'}}
   a2=array.getitem(...)
   asserta1.order==a2.order

2. order does not make sense to include in the ArrayMetadata. It would be convenient to attach to that object but I don't think its the right choice -- we would have to special case serialization of that object so 👎 .

What do you think about the approach I've taken? Do you have other suggestions?

[normanrz]

I like it! I think ArraySpec is a good place to put it, because it is the specification of how an array shall be constructed. That makes sense to me!

[normanrz]

Suggested change

	order="C",# TODO!!!!
	order="C",

A hard-coded C is fine here, because it is just the shard index. It doesn't surface into user code.

[pep8speaks]

Hello @maxrjones! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file src/zarr/codecs/sharding.py:

Line 624:101: E501 line too long (102 > 100 characters)

[maxrjones]

Progress towards the proposal in #1772 to use Dask style config (donfig) for configuration in v3.

TODO:

• Migrate sync configuration to donfig
• Migrate runtime configuration to donfig
• Add tests for setting configuration
• Add docs for listing and setting configuration

[maxrjones]

@jhamman fyi I've gotten as far as I can right now on the configuration and will be offline until wednesday. I started working on the byte order in 5de1db2, but while the tests pass I don't think it's fully correct which is why it's left out of this PR.

You're welcome to take over this PR or close if you think it'd be better to make progress separately. I'm free Wednesday 5/15 from 1-3 PM ET to work on this next if it's still an open issue at that time.

[jhamman]

@normanrz - I picked up where @maxrjones left off on this one. Things are going well but I have one design question for you.

The question is what to do with the order config option. For now, I've added it to the ArraySpec object which works but, to some extent, mixes array metadata w/ array runtime options. The relevant constraints that I see are:

1. We want to decide on an order at the time we create the AsyncArray object. From then on, all chunks and output arrays should use this. You'll notice a new attribute AsyncArray.order defined in the __init__ method. This is important because we don't wouldn't want the following to change the order:

   array=AsyncArray.open(...)
   config.set({'array.order': 'C'}}
   a1=array.getitem(...)
   config.set({'array.order': 'F'}}
   a2=array.getitem(...)
   asserta1.order==a2.order

2. order does not make sense to include in the ArrayMetadata. It would be convenient to attach to that object but I don't think its the right choice -- we would have to special case serialization of that object so 👎 .

What do you think about the approach I've taken? Do you have other suggestions?

[normanrz]

I like it! I think ArraySpec is a good place to put it, because it is the specification of how an array shall be constructed. That makes sense to me!

[normanrz]

Suggested change

	order="C",# TODO!!!!
	order="C",

A hard-coded C is fine here, because it is just the shard index. It doesn't surface into user code.

[pep8speaks]

Hello @maxrjones! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file src/zarr/codecs/sharding.py:

Line 624:101: E501 line too long (102 > 100 characters)

[maxrjones]

Progress towards the proposal in #1772 to use Dask style config (donfig) for configuration in v3.

TODO:

• Migrate sync configuration to donfig
• Migrate runtime configuration to donfig
• Add tests for setting configuration
• Add docs for listing and setting configuration

[maxrjones]

@jhamman fyi I've gotten as far as I can right now on the configuration and will be offline until wednesday. I started working on the byte order in 5de1db2, but while the tests pass I don't think it's fully correct which is why it's left out of this PR.

You're welcome to take over this PR or close if you think it'd be better to make progress separately. I'm free Wednesday 5/15 from 1-3 PM ET to work on this next if it's still an open issue at that time.

[jhamman]

@normanrz - I picked up where @maxrjones left off on this one. Things are going well but I have one design question for you.

The question is what to do with the order config option. For now, I've added it to the ArraySpec object which works but, to some extent, mixes array metadata w/ array runtime options. The relevant constraints that I see are:

1. We want to decide on an order at the time we create the AsyncArray object. From then on, all chunks and output arrays should use this. You'll notice a new attribute AsyncArray.order defined in the __init__ method. This is important because we don't wouldn't want the following to change the order:

   array=AsyncArray.open(...)
   config.set({'array.order': 'C'}}
   a1=array.getitem(...)
   config.set({'array.order': 'F'}}
   a2=array.getitem(...)
   asserta1.order==a2.order

2. order does not make sense to include in the ArrayMetadata. It would be convenient to attach to that object but I don't think its the right choice -- we would have to special case serialization of that object so 👎 .

What do you think about the approach I've taken? Do you have other suggestions?

[normanrz]

I like it! I think ArraySpec is a good place to put it, because it is the specification of how an array shall be constructed. That makes sense to me!

[normanrz]

Suggested change

	order="C",# TODO!!!!
	order="C",

A hard-coded C is fine here, because it is just the shard index. It doesn't surface into user code.

[pep8speaks]

Hello @maxrjones! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file src/zarr/codecs/sharding.py:

Line 624:101: E501 line too long (102 > 100 characters)

[maxrjones]

Progress towards the proposal in #1772 to use Dask style config (donfig) for configuration in v3.

TODO:

• Migrate sync configuration to donfig
• Migrate runtime configuration to donfig
• Add tests for setting configuration
• Add docs for listing and setting configuration

[maxrjones]

@jhamman fyi I've gotten as far as I can right now on the configuration and will be offline until wednesday. I started working on the byte order in 5de1db2, but while the tests pass I don't think it's fully correct which is why it's left out of this PR.

You're welcome to take over this PR or close if you think it'd be better to make progress separately. I'm free Wednesday 5/15 from 1-3 PM ET to work on this next if it's still an open issue at that time.

[jhamman]

@normanrz - I picked up where @maxrjones left off on this one. Things are going well but I have one design question for you.

The question is what to do with the order config option. For now, I've added it to the ArraySpec object which works but, to some extent, mixes array metadata w/ array runtime options. The relevant constraints that I see are:

1. We want to decide on an order at the time we create the AsyncArray object. From then on, all chunks and output arrays should use this. You'll notice a new attribute AsyncArray.order defined in the __init__ method. This is important because we don't wouldn't want the following to change the order:

   array=AsyncArray.open(...)
   config.set({'array.order': 'C'}}
   a1=array.getitem(...)
   config.set({'array.order': 'F'}}
   a2=array.getitem(...)
   asserta1.order==a2.order

2. order does not make sense to include in the ArrayMetadata. It would be convenient to attach to that object but I don't think its the right choice -- we would have to special case serialization of that object so 👎 .

What do you think about the approach I've taken? Do you have other suggestions?

[normanrz]

I like it! I think ArraySpec is a good place to put it, because it is the specification of how an array shall be constructed. That makes sense to me!

[normanrz]

Suggested change

	order="C",# TODO!!!!
	order="C",

A hard-coded C is fine here, because it is just the shard index. It doesn't surface into user code.

[pep8speaks]

Hello @maxrjones! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file src/zarr/codecs/sharding.py:

Line 624:101: E501 line too long (102 > 100 characters)

[maxrjones]

Progress towards the proposal in #1772 to use Dask style config (donfig) for configuration in v3.

TODO:

• Migrate sync configuration to donfig
• Migrate runtime configuration to donfig
• Add tests for setting configuration
• Add docs for listing and setting configuration

[maxrjones]

@jhamman fyi I've gotten as far as I can right now on the configuration and will be offline until wednesday. I started working on the byte order in 5de1db2, but while the tests pass I don't think it's fully correct which is why it's left out of this PR.

You're welcome to take over this PR or close if you think it'd be better to make progress separately. I'm free Wednesday 5/15 from 1-3 PM ET to work on this next if it's still an open issue at that time.

[jhamman]

@normanrz - I picked up where @maxrjones left off on this one. Things are going well but I have one design question for you.

The question is what to do with the order config option. For now, I've added it to the ArraySpec object which works but, to some extent, mixes array metadata w/ array runtime options. The relevant constraints that I see are:

1. We want to decide on an order at the time we create the AsyncArray object. From then on, all chunks and output arrays should use this. You'll notice a new attribute AsyncArray.order defined in the __init__ method. This is important because we don't wouldn't want the following to change the order:

   array=AsyncArray.open(...)
   config.set({'array.order': 'C'}}
   a1=array.getitem(...)
   config.set({'array.order': 'F'}}
   a2=array.getitem(...)
   asserta1.order==a2.order

2. order does not make sense to include in the ArrayMetadata. It would be convenient to attach to that object but I don't think its the right choice -- we would have to special case serialization of that object so 👎 .

What do you think about the approach I've taken? Do you have other suggestions?

[normanrz]

I like it! I think ArraySpec is a good place to put it, because it is the specification of how an array shall be constructed. That makes sense to me!

[normanrz]

Suggested change

	order="C",# TODO!!!!
	order="C",

A hard-coded C is fine here, because it is just the shard index. It doesn't surface into user code.

[pep8speaks]

Hello @maxrjones! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file src/zarr/codecs/sharding.py:

Line 624:101: E501 line too long (102 > 100 characters)

[maxrjones]

Progress towards the proposal in #1772 to use Dask style config (donfig) for configuration in v3.

TODO:

• Migrate sync configuration to donfig
• Migrate runtime configuration to donfig
• Add tests for setting configuration
• Add docs for listing and setting configuration

[maxrjones]

@jhamman fyi I've gotten as far as I can right now on the configuration and will be offline until wednesday. I started working on the byte order in 5de1db2, but while the tests pass I don't think it's fully correct which is why it's left out of this PR.

You're welcome to take over this PR or close if you think it'd be better to make progress separately. I'm free Wednesday 5/15 from 1-3 PM ET to work on this next if it's still an open issue at that time.

[jhamman]

@normanrz - I picked up where @maxrjones left off on this one. Things are going well but I have one design question for you.

The question is what to do with the order config option. For now, I've added it to the ArraySpec object which works but, to some extent, mixes array metadata w/ array runtime options. The relevant constraints that I see are:

1. We want to decide on an order at the time we create the AsyncArray object. From then on, all chunks and output arrays should use this. You'll notice a new attribute AsyncArray.order defined in the __init__ method. This is important because we don't wouldn't want the following to change the order:

   array=AsyncArray.open(...)
   config.set({'array.order': 'C'}}
   a1=array.getitem(...)
   config.set({'array.order': 'F'}}
   a2=array.getitem(...)
   asserta1.order==a2.order

2. order does not make sense to include in the ArrayMetadata. It would be convenient to attach to that object but I don't think its the right choice -- we would have to special case serialization of that object so 👎 .

What do you think about the approach I've taken? Do you have other suggestions?

[normanrz]

I like it! I think ArraySpec is a good place to put it, because it is the specification of how an array shall be constructed. That makes sense to me!

[normanrz]

Suggested change

	order="C",# TODO!!!!
	order="C",

A hard-coded C is fine here, because it is just the shard index. It doesn't surface into user code.

[pep8speaks]

Hello @maxrjones! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file src/zarr/codecs/sharding.py:

Line 624:101: E501 line too long (102 > 100 characters)