ARROW-5357: [Rust] Change Buffer::len to represent total bytes instead of used bytes

[sunchao]

No description provided.

[sunchao]

After discussing with @paddyhoran on the JIRA, I'm now inclined to keep both len and capacity in Buffer, which allows us to still maintain most of the existing functionalities. To fix the flatbuffer conversion issue, we'll need ARROW-5358 which allows us to compare ArrayData properly.

@nevi-me let me know if this sounds good to you.

[nevi-me]

I saw the conversation, would we be able to use capacity when writing to IPC from the current Buffer?

[sunchao]

Yes, both reading and writing to IPC buffer will use capacity instead of len.

[nevi-me]

Apologies for not responding sooner @sunchao

It's only mutable buffer that has capacity, and when we freeze it to create a buffer, we 'lose' the capacity.

#[derive(PartialEq,Debug)]pubstructBuffer{/// Reference-counted pointer to the internal byte buffer.data:Arc<BufferData>,/// The offset into the buffer.offset:usize,}structBufferData{ptr:*constu8,len:usize,// no capacity}

[sunchao]

@nevi-me yes that's right. As I mentioned above, I think it's better to keep both len and capacity in Buffer (same as MutableBuffer).

I'm currently working on https://issues.apache.org/jira/browse/ARROW-5358 which hopefully will allow us to compare Array and ArrayData. It should be able to help resolve the issue you are seeing. I plan to come back to this after that is done.

[nevi-me]

Okay, great! I'll keep #4167 open, and continue #4330 without ArrayData comparisons for now

[codecov-io]

Codecov Report

Merging #4331 into master will decrease coverage by 4.92%.
The diff coverage is 88.88%.

@@ Coverage Diff @@## master #4331 +/- ##
===========================================
- Coverage 87.5% 82.57% -4.93% 
===========================================
Files 998 86 -912 Lines 141784 24935 -116849 Branches 1418 0 -1418 ===========================================
- Hits 124065 20591 -103474 + Misses 17357 4344 -13013 + Partials 362 0 -362

Impacted Files	Coverage Δ
rust/arrow/src/memory.rs	100% <100%> (ø)	⬆️
rust/arrow/src/array/array.rs	92.57% <100%> (ø)	⬆️
rust/arrow/src/tensor.rs	92.3% <80%> (-0.48%)	⬇️
rust/arrow/src/buffer.rs	91.3% <88.88%> (+1.52%)	⬆️
python/pyarrow/ipc.pxi
cpp/src/arrow/csv/chunker-test.cc
cpp/src/parquet/column_page.h
cpp/src/parquet/bloom_filter-test.cc
cpp/src/arrow/array/builder_decimal.cc
r/src/symbols.cpp
... and 907 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 38b0176...7eac173. Read the comment docs.

[paddyhoran]

@nevi-me@sunchao I know there was some related discussion here but are we ok to move forward with this PR (i.e. adding capacity to Buffer). Is this ready for review?

[sunchao]

I think it is ready for review. It'd be great if you can take a look.

[paddyhoran]

Yep, sounds good. I'll take a look when I get a chance.

[paddyhoran]

LGTM, just a few nits.

[paddyhoran]

I'm not against the current implementation, but I wonder if we should only compare the "meaningful" data?

[andygrove]

I agree. It seems wrong that on one hand the memcmp is comparing up to len and we bypass that if there is a mismatch on capacity

[sunchao]

Thanks guys! I'm slightly confused - here we are comparing meaningful data, no? and if capacity mismatch then it is considered not equal and we skip comparing the data content.

The equality is defined as: 1) both length should be equal, 2) both capacity should be equal, and 3) data content up to length should be equal. Let me know if this definition sounds good to you.

[paddyhoran]

To me if two arrays only differ by the amount of padding that they have then I would consider them equal. When I perform operations using these two arrays I will get the same answer (because the padding, or rather amount of padding, does not impact the result). However:

• I am focused on computation, maybe there are other implication in IPC, etc.
• In practice, this probably won't come up as we round up padding to a multiple of 64 bytes, but it could happen.

[sunchao]

Array equality is defined separately in array/equal.rs and yes, it does take account on what you said (it compares buffer content with data()). In the current context we are discussing equality of buffers though, which IMO, when looking at in isolation, should consider both len and capacity.

[paddyhoran]

Ok, yes I agree.

[paddyhoran]

We should also check that we are padded to 64 bytes. We assume this in the SIMD implementations and it's recommended here.

[paddyhoran]

We should probably test for this also.

[andygrove]

I agree. It seems wrong that on one hand the memcmp is comparing up to len and we bypass that if there is a mismatch on capacity

[andygrove]

Same issue .. memcmp compares up to len so it doesn't matter if the capacities match or not

[nevi-me]

I initially thought this would be a blocker for IPC, but I've managed to implement the reader and writer without this. I'd vote for closing this PR @sunchao.

[paddyhoran]

I'd vote for closing this PR @sunchao.

I would actually like to see this PR merged (maybe after further review). Length and capacity are two distinct concepts in Arrow and having capacity be explicit is easier to understand especially for newcomers.

I also think it will be useful when converting from other libraries to Arrow. In these cases, we will have to check the alignment and padding (capacity) of the data that is passed to us.

[sunchao]

Yes I think it is still useful to have size and capacity as two separate properties of buffer. @andygrove could you take another look on this? I think your concern has been addressed via the comments.

[emkornfield]

@andygrove want to take look?

[paddyhoran]

@sunchao can you rebase this if you get a chance (or let me know if you are stuck for time and I will)? It turns out that this PR fixes UB as pointed out by @jturner314 on #6397.

I'll re-review early next week and hopefully we can get this merged.

[sunchao]

Sure @paddyhoran . Updated.

[paddyhoran]

LGTM - @andygrove you requested changes previously, do you want to re-review or can I merge this one?

[andygrove]

Apologies for missing the earlier mentions. I really need to figure out a better way of managing all the emails I get so that these mentions stand out.

[wesm]

@andygrove the easiest thing is to set up a gmail filter that sends any GitHub e-mail with "you were mentioned" directly to Inbox (vice versa, to auto-archive any GitHub e-mail that does not contain this). A separate filter rule should be used to apply a label unconditionally to Arrow-related GitHub e-mails