Feat/read write direct

[konstibob]

implements #85 now, direct reads and writes for arrays.
These methods skip the codec pipeline for neccesary methods, avoiding the decode, encode round trip, which would imply a quality loss for lossy codecs.

[normanrz]

Maybe the arg could be a long[2][] addressing first the shard and then the inner chunk?

[konstibob]

I would disagree here. The recursive top to bottom descent already happens at other points in the program, and adressing each nested shard, would be a complete overkill here!

Array.readInnerChunkDirect -> Gives us back the shard that it is on
ShardingIndexedCodec -> gives us the mid-shards
while nestedcodec L2 -> gives us the leaf where it is, which gves us the complete recursive top to bottom descent, already.

Therefore no need for another parameter!

[normanrz]

Although rarely used, it is spec-compliant to have multiple nested sharding codecs. A single innerChunkShape would not suffice.

[konstibob]

innerChunkShape() recurses through nested sharding codecs and returns the innermost
addressable shape readInnerChunkEncoded splits the coordinate per level with.
So arbitrary nesting depth works see testDirectInnerChunkReadNestedSharding

[normanrz]

I think a writeInnerChunkDirect could also be interesting. It would read and parse (no decode) an existing shard, swap out the bytes for one provided inner chunk, and write out the shard again.

However, users would likely want to write multiple inner chunks for which this would be inefficient. Instead, this could also be a builder pattern, where you start a session for a shard (read and parse existing), then write multiple inner chunks to a buffer, and finally write (commit) the whole shard. Similar to what ShardingIndexedCodec.encode does internally.

[konstibob]

completly agree, editing multiple chunks directly was done pretty inefficient. Added functionality to directly write to multiple chunks at the same time!

[konstibob]

adjusted write to be more efficient. ÌnnerChunkWriter.java is now doing any sort of inner chunk writing, collecting all needed chunks and reconstructing the shard + metadata by going through all of it iteratively.

[konstibob]

implements #85 now, direct reads and writes for arrays.
These methods skip the codec pipeline for neccesary methods, avoiding the decode, encode round trip, which would imply a quality loss for lossy codecs.

[normanrz]

Maybe the arg could be a long[2][] addressing first the shard and then the inner chunk?

[konstibob]

I would disagree here. The recursive top to bottom descent already happens at other points in the program, and adressing each nested shard, would be a complete overkill here!

Array.readInnerChunkDirect -> Gives us back the shard that it is on
ShardingIndexedCodec -> gives us the mid-shards
while nestedcodec L2 -> gives us the leaf where it is, which gves us the complete recursive top to bottom descent, already.

Therefore no need for another parameter!

[normanrz]

Although rarely used, it is spec-compliant to have multiple nested sharding codecs. A single innerChunkShape would not suffice.

[konstibob]

innerChunkShape() recurses through nested sharding codecs and returns the innermost
addressable shape readInnerChunkEncoded splits the coordinate per level with.
So arbitrary nesting depth works see testDirectInnerChunkReadNestedSharding

[normanrz]

I think a writeInnerChunkDirect could also be interesting. It would read and parse (no decode) an existing shard, swap out the bytes for one provided inner chunk, and write out the shard again.

However, users would likely want to write multiple inner chunks for which this would be inefficient. Instead, this could also be a builder pattern, where you start a session for a shard (read and parse existing), then write multiple inner chunks to a buffer, and finally write (commit) the whole shard. Similar to what ShardingIndexedCodec.encode does internally.

[konstibob]

completly agree, editing multiple chunks directly was done pretty inefficient. Added functionality to directly write to multiple chunks at the same time!

[konstibob]

adjusted write to be more efficient. ÌnnerChunkWriter.java is now doing any sort of inner chunk writing, collecting all needed chunks and reconstructing the shard + metadata by going through all of it iteratively.

[konstibob]

implements #85 now, direct reads and writes for arrays.
These methods skip the codec pipeline for neccesary methods, avoiding the decode, encode round trip, which would imply a quality loss for lossy codecs.

[normanrz]

Maybe the arg could be a long[2][] addressing first the shard and then the inner chunk?

[konstibob]

I would disagree here. The recursive top to bottom descent already happens at other points in the program, and adressing each nested shard, would be a complete overkill here!

Array.readInnerChunkDirect -> Gives us back the shard that it is on
ShardingIndexedCodec -> gives us the mid-shards
while nestedcodec L2 -> gives us the leaf where it is, which gves us the complete recursive top to bottom descent, already.

Therefore no need for another parameter!

[normanrz]

Although rarely used, it is spec-compliant to have multiple nested sharding codecs. A single innerChunkShape would not suffice.

[konstibob]

innerChunkShape() recurses through nested sharding codecs and returns the innermost
addressable shape readInnerChunkEncoded splits the coordinate per level with.
So arbitrary nesting depth works see testDirectInnerChunkReadNestedSharding

[normanrz]

I think a writeInnerChunkDirect could also be interesting. It would read and parse (no decode) an existing shard, swap out the bytes for one provided inner chunk, and write out the shard again.

However, users would likely want to write multiple inner chunks for which this would be inefficient. Instead, this could also be a builder pattern, where you start a session for a shard (read and parse existing), then write multiple inner chunks to a buffer, and finally write (commit) the whole shard. Similar to what ShardingIndexedCodec.encode does internally.

[konstibob]

completly agree, editing multiple chunks directly was done pretty inefficient. Added functionality to directly write to multiple chunks at the same time!

[konstibob]

adjusted write to be more efficient. ÌnnerChunkWriter.java is now doing any sort of inner chunk writing, collecting all needed chunks and reconstructing the shard + metadata by going through all of it iteratively.

[konstibob]

implements #85 now, direct reads and writes for arrays.
These methods skip the codec pipeline for neccesary methods, avoiding the decode, encode round trip, which would imply a quality loss for lossy codecs.

[normanrz]

Maybe the arg could be a long[2][] addressing first the shard and then the inner chunk?

[konstibob]

I would disagree here. The recursive top to bottom descent already happens at other points in the program, and adressing each nested shard, would be a complete overkill here!

Array.readInnerChunkDirect -> Gives us back the shard that it is on
ShardingIndexedCodec -> gives us the mid-shards
while nestedcodec L2 -> gives us the leaf where it is, which gves us the complete recursive top to bottom descent, already.

Therefore no need for another parameter!

[normanrz]

Although rarely used, it is spec-compliant to have multiple nested sharding codecs. A single innerChunkShape would not suffice.

[konstibob]

innerChunkShape() recurses through nested sharding codecs and returns the innermost
addressable shape readInnerChunkEncoded splits the coordinate per level with.
So arbitrary nesting depth works see testDirectInnerChunkReadNestedSharding

[normanrz]

I think a writeInnerChunkDirect could also be interesting. It would read and parse (no decode) an existing shard, swap out the bytes for one provided inner chunk, and write out the shard again.

However, users would likely want to write multiple inner chunks for which this would be inefficient. Instead, this could also be a builder pattern, where you start a session for a shard (read and parse existing), then write multiple inner chunks to a buffer, and finally write (commit) the whole shard. Similar to what ShardingIndexedCodec.encode does internally.

[konstibob]

completly agree, editing multiple chunks directly was done pretty inefficient. Added functionality to directly write to multiple chunks at the same time!

[konstibob]

adjusted write to be more efficient. ÌnnerChunkWriter.java is now doing any sort of inner chunk writing, collecting all needed chunks and reconstructing the shard + metadata by going through all of it iteratively.

[konstibob]

implements #85 now, direct reads and writes for arrays.
These methods skip the codec pipeline for neccesary methods, avoiding the decode, encode round trip, which would imply a quality loss for lossy codecs.

[normanrz]

Maybe the arg could be a long[2][] addressing first the shard and then the inner chunk?

[konstibob]

I would disagree here. The recursive top to bottom descent already happens at other points in the program, and adressing each nested shard, would be a complete overkill here!

Array.readInnerChunkDirect -> Gives us back the shard that it is on
ShardingIndexedCodec -> gives us the mid-shards
while nestedcodec L2 -> gives us the leaf where it is, which gves us the complete recursive top to bottom descent, already.

Therefore no need for another parameter!

[normanrz]

Although rarely used, it is spec-compliant to have multiple nested sharding codecs. A single innerChunkShape would not suffice.

[konstibob]

innerChunkShape() recurses through nested sharding codecs and returns the innermost
addressable shape readInnerChunkEncoded splits the coordinate per level with.
So arbitrary nesting depth works see testDirectInnerChunkReadNestedSharding

[normanrz]

I think a writeInnerChunkDirect could also be interesting. It would read and parse (no decode) an existing shard, swap out the bytes for one provided inner chunk, and write out the shard again.

However, users would likely want to write multiple inner chunks for which this would be inefficient. Instead, this could also be a builder pattern, where you start a session for a shard (read and parse existing), then write multiple inner chunks to a buffer, and finally write (commit) the whole shard. Similar to what ShardingIndexedCodec.encode does internally.

[konstibob]

completly agree, editing multiple chunks directly was done pretty inefficient. Added functionality to directly write to multiple chunks at the same time!

[konstibob]

adjusted write to be more efficient. ÌnnerChunkWriter.java is now doing any sort of inner chunk writing, collecting all needed chunks and reconstructing the shard + metadata by going through all of it iteratively.

[konstibob]

implements #85 now, direct reads and writes for arrays.
These methods skip the codec pipeline for neccesary methods, avoiding the decode, encode round trip, which would imply a quality loss for lossy codecs.

[normanrz]

Maybe the arg could be a long[2][] addressing first the shard and then the inner chunk?

[konstibob]

I would disagree here. The recursive top to bottom descent already happens at other points in the program, and adressing each nested shard, would be a complete overkill here!

Array.readInnerChunkDirect -> Gives us back the shard that it is on
ShardingIndexedCodec -> gives us the mid-shards
while nestedcodec L2 -> gives us the leaf where it is, which gves us the complete recursive top to bottom descent, already.

Therefore no need for another parameter!

[normanrz]

Although rarely used, it is spec-compliant to have multiple nested sharding codecs. A single innerChunkShape would not suffice.

[konstibob]

innerChunkShape() recurses through nested sharding codecs and returns the innermost
addressable shape readInnerChunkEncoded splits the coordinate per level with.
So arbitrary nesting depth works see testDirectInnerChunkReadNestedSharding

[normanrz]

I think a writeInnerChunkDirect could also be interesting. It would read and parse (no decode) an existing shard, swap out the bytes for one provided inner chunk, and write out the shard again.

However, users would likely want to write multiple inner chunks for which this would be inefficient. Instead, this could also be a builder pattern, where you start a session for a shard (read and parse existing), then write multiple inner chunks to a buffer, and finally write (commit) the whole shard. Similar to what ShardingIndexedCodec.encode does internally.

[konstibob]

completly agree, editing multiple chunks directly was done pretty inefficient. Added functionality to directly write to multiple chunks at the same time!

[konstibob]

adjusted write to be more efficient. ÌnnerChunkWriter.java is now doing any sort of inner chunk writing, collecting all needed chunks and reconstructing the shard + metadata by going through all of it iteratively.

[konstibob]

implements #85 now, direct reads and writes for arrays.
These methods skip the codec pipeline for neccesary methods, avoiding the decode, encode round trip, which would imply a quality loss for lossy codecs.

[normanrz]

Maybe the arg could be a long[2][] addressing first the shard and then the inner chunk?

[konstibob]

I would disagree here. The recursive top to bottom descent already happens at other points in the program, and adressing each nested shard, would be a complete overkill here!

Array.readInnerChunkDirect -> Gives us back the shard that it is on
ShardingIndexedCodec -> gives us the mid-shards
while nestedcodec L2 -> gives us the leaf where it is, which gves us the complete recursive top to bottom descent, already.

Therefore no need for another parameter!

[normanrz]

Although rarely used, it is spec-compliant to have multiple nested sharding codecs. A single innerChunkShape would not suffice.

[konstibob]

innerChunkShape() recurses through nested sharding codecs and returns the innermost
addressable shape readInnerChunkEncoded splits the coordinate per level with.
So arbitrary nesting depth works see testDirectInnerChunkReadNestedSharding

[normanrz]

I think a writeInnerChunkDirect could also be interesting. It would read and parse (no decode) an existing shard, swap out the bytes for one provided inner chunk, and write out the shard again.

However, users would likely want to write multiple inner chunks for which this would be inefficient. Instead, this could also be a builder pattern, where you start a session for a shard (read and parse existing), then write multiple inner chunks to a buffer, and finally write (commit) the whole shard. Similar to what ShardingIndexedCodec.encode does internally.

[konstibob]

completly agree, editing multiple chunks directly was done pretty inefficient. Added functionality to directly write to multiple chunks at the same time!

[konstibob]

adjusted write to be more efficient. ÌnnerChunkWriter.java is now doing any sort of inner chunk writing, collecting all needed chunks and reconstructing the shard + metadata by going through all of it iteratively.

[konstibob]

implements #85 now, direct reads and writes for arrays.
These methods skip the codec pipeline for neccesary methods, avoiding the decode, encode round trip, which would imply a quality loss for lossy codecs.

[normanrz]

Maybe the arg could be a long[2][] addressing first the shard and then the inner chunk?

[konstibob]

I would disagree here. The recursive top to bottom descent already happens at other points in the program, and adressing each nested shard, would be a complete overkill here!

Array.readInnerChunkDirect -> Gives us back the shard that it is on
ShardingIndexedCodec -> gives us the mid-shards
while nestedcodec L2 -> gives us the leaf where it is, which gves us the complete recursive top to bottom descent, already.

Therefore no need for another parameter!

[normanrz]

Although rarely used, it is spec-compliant to have multiple nested sharding codecs. A single innerChunkShape would not suffice.

[konstibob]

innerChunkShape() recurses through nested sharding codecs and returns the innermost
addressable shape readInnerChunkEncoded splits the coordinate per level with.
So arbitrary nesting depth works see testDirectInnerChunkReadNestedSharding

[normanrz]

I think a writeInnerChunkDirect could also be interesting. It would read and parse (no decode) an existing shard, swap out the bytes for one provided inner chunk, and write out the shard again.

However, users would likely want to write multiple inner chunks for which this would be inefficient. Instead, this could also be a builder pattern, where you start a session for a shard (read and parse existing), then write multiple inner chunks to a buffer, and finally write (commit) the whole shard. Similar to what ShardingIndexedCodec.encode does internally.

[konstibob]

completly agree, editing multiple chunks directly was done pretty inefficient. Added functionality to directly write to multiple chunks at the same time!

[konstibob]

adjusted write to be more efficient. ÌnnerChunkWriter.java is now doing any sort of inner chunk writing, collecting all needed chunks and reconstructing the shard + metadata by going through all of it iteratively.