ARROW-6110: [Java][Integration] Support LargeList Type and add integration test with C++

[rymurr]

Add large list and ensure it works with Integration tests. As noted in the JIRA
ticket this is rather limited as the underlying vector doesn't support int64 addressing

The important downcasts to int32 have been noted for a follow up once vectors with
long addresses are supported

[github-actions]

https://issues.apache.org/jira/browse/ARROW-6110

[emkornfield]

@BryanCutler would you have time to review?

[emkornfield]

@rymurr looks like this needs a rebase

[rymurr]

@rymurr looks like this needs a rebase

Thanks for the reminder @emkornfield, done.

[BryanCutler]

I'm a little swamped right now, but I'll try to review sometime this week

[liyafan82]

Here we need to cast the argument of getLong to long, to avoid integer overflow.

[rymurr]

done

[liyafan82]

Here we need to cast the first argument, too.

[rymurr]

done

[liyafan82]

I think we can replace the old implementation

void setBit(ArrowBuf validityBuffer, int index)

with this one, as now ArrowBuf is based on 64-bit index.

[rymurr]

done

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[rymurr]

done

[liyafan82]

Here we need to cast getLong argument to long.

[rymurr]

done

[liyafan82]

We need to cast (numRecords + 1) to long. Otherwise, integer overflow may happen before promoting the result to long.

[rymurr]

done

[nealrichardson]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

[BryanCutler]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

I'm taking a look now, I'd like to get it in for 1.0 too.

[BryanCutler]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

[BryanCutler]

It would be nice not to add a new template and combine with the UnionListWriter if possible. That doesn't have to be done here though, it can be looked at later.

[rymurr]

agreed, I fixed this...was just being lazy ;-)

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[BryanCutler]

Since this is the relative bit index, it's not possible to return long. Can you change return value to an int?

[rymurr]

done

[BryanCutler]

From the comment above, I think this should still be an int

[rymurr]

done

[BryanCutler]

shouldn't this be long?

[rymurr]

yup! Done

[BryanCutler]

return int

[rymurr]

done

[BryanCutler]

Could you add a TODO to revisit once 64 bit vectors are supported?

[BryanCutler]

Also indicate this in the javadoc here and probably for the class. It might even be a good idea to raise an error if the user tries to add too many elements, otherwise things might just start looking wrong.

[rymurr]

done

[BryanCutler]

This should return long and not be casted to int

[rymurr]

done

[BryanCutler]

same here

[rymurr]

done

[BryanCutler]

could you add a check for Types.MinorType.LARGELIST here are remove the changes below?

[BryanCutler]

you also won't need typeBitWidth as an arg

[rymurr]

thanks, done

[rymurr]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

Thanks a lot for the thorough review. I have fixed up everything you mentioned. It appears some of the confusion was related to changes for 64-bit allocations that were recently merged and the rest was my ignorance!

I have pushed a change with all your recommended fixes.

[BryanCutler]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

[BryanCutler]

does this arg need to be cast to long?

[BryanCutler]

also here

[BryanCutler]

cast to long?

[BryanCutler]

was there a reason for this change?

[rymurr]

IDE autoformatting...reverted!

[BryanCutler]

why is this needed? the vector only supports integer max_value number of elements right?

[rymurr]

you are correct. Removed

[BryanCutler]

doesn't value need to be passed in here?

[rymurr]

value is always of type Void. None of the other visitors pass value to children.

[rymurr]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

Thanks for another thorough review @BryanCutler ! I think that I addressed everything. I may have went over the top w/ the casting but it doesn't hurt :-)

[BryanCutler]

LGTM

[BryanCutler]

merged to master, thanks @rymurr !

[rymurr]

Add large list and ensure it works with Integration tests. As noted in the JIRA
ticket this is rather limited as the underlying vector doesn't support int64 addressing

The important downcasts to int32 have been noted for a follow up once vectors with
long addresses are supported

[github-actions]

https://issues.apache.org/jira/browse/ARROW-6110

[emkornfield]

@BryanCutler would you have time to review?

[emkornfield]

@rymurr looks like this needs a rebase

[rymurr]

@rymurr looks like this needs a rebase

Thanks for the reminder @emkornfield, done.

[BryanCutler]

I'm a little swamped right now, but I'll try to review sometime this week

[liyafan82]

Here we need to cast the argument of getLong to long, to avoid integer overflow.

[rymurr]

done

[liyafan82]

Here we need to cast the first argument, too.

[rymurr]

done

[liyafan82]

I think we can replace the old implementation

void setBit(ArrowBuf validityBuffer, int index)

with this one, as now ArrowBuf is based on 64-bit index.

[rymurr]

done

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[rymurr]

done

[liyafan82]

Here we need to cast getLong argument to long.

[rymurr]

done

[liyafan82]

We need to cast (numRecords + 1) to long. Otherwise, integer overflow may happen before promoting the result to long.

[rymurr]

done

[nealrichardson]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

[BryanCutler]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

I'm taking a look now, I'd like to get it in for 1.0 too.

[BryanCutler]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

[BryanCutler]

It would be nice not to add a new template and combine with the UnionListWriter if possible. That doesn't have to be done here though, it can be looked at later.

[rymurr]

agreed, I fixed this...was just being lazy ;-)

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[BryanCutler]

Since this is the relative bit index, it's not possible to return long. Can you change return value to an int?

[rymurr]

done

[BryanCutler]

From the comment above, I think this should still be an int

[rymurr]

done

[BryanCutler]

shouldn't this be long?

[rymurr]

yup! Done

[BryanCutler]

return int

[rymurr]

done

[BryanCutler]

Could you add a TODO to revisit once 64 bit vectors are supported?

[BryanCutler]

Also indicate this in the javadoc here and probably for the class. It might even be a good idea to raise an error if the user tries to add too many elements, otherwise things might just start looking wrong.

[rymurr]

done

[BryanCutler]

This should return long and not be casted to int

[rymurr]

done

[BryanCutler]

same here

[rymurr]

done

[BryanCutler]

could you add a check for Types.MinorType.LARGELIST here are remove the changes below?

[BryanCutler]

you also won't need typeBitWidth as an arg

[rymurr]

thanks, done

[rymurr]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

Thanks a lot for the thorough review. I have fixed up everything you mentioned. It appears some of the confusion was related to changes for 64-bit allocations that were recently merged and the rest was my ignorance!

I have pushed a change with all your recommended fixes.

[BryanCutler]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

[BryanCutler]

does this arg need to be cast to long?

[BryanCutler]

also here

[BryanCutler]

cast to long?

[BryanCutler]

was there a reason for this change?

[rymurr]

IDE autoformatting...reverted!

[BryanCutler]

why is this needed? the vector only supports integer max_value number of elements right?

[rymurr]

you are correct. Removed

[BryanCutler]

doesn't value need to be passed in here?

[rymurr]

value is always of type Void. None of the other visitors pass value to children.

[rymurr]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

Thanks for another thorough review @BryanCutler ! I think that I addressed everything. I may have went over the top w/ the casting but it doesn't hurt :-)

[BryanCutler]

LGTM

[BryanCutler]

merged to master, thanks @rymurr !

[rymurr]

Add large list and ensure it works with Integration tests. As noted in the JIRA
ticket this is rather limited as the underlying vector doesn't support int64 addressing

The important downcasts to int32 have been noted for a follow up once vectors with
long addresses are supported

[github-actions]

https://issues.apache.org/jira/browse/ARROW-6110

[emkornfield]

@BryanCutler would you have time to review?

[emkornfield]

@rymurr looks like this needs a rebase

[rymurr]

@rymurr looks like this needs a rebase

Thanks for the reminder @emkornfield, done.

[BryanCutler]

I'm a little swamped right now, but I'll try to review sometime this week

[liyafan82]

Here we need to cast the argument of getLong to long, to avoid integer overflow.

[rymurr]

done

[liyafan82]

Here we need to cast the first argument, too.

[rymurr]

done

[liyafan82]

I think we can replace the old implementation

void setBit(ArrowBuf validityBuffer, int index)

with this one, as now ArrowBuf is based on 64-bit index.

[rymurr]

done

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[rymurr]

done

[liyafan82]

Here we need to cast getLong argument to long.

[rymurr]

done

[liyafan82]

We need to cast (numRecords + 1) to long. Otherwise, integer overflow may happen before promoting the result to long.

[rymurr]

done

[nealrichardson]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

[BryanCutler]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

I'm taking a look now, I'd like to get it in for 1.0 too.

[BryanCutler]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

[BryanCutler]

It would be nice not to add a new template and combine with the UnionListWriter if possible. That doesn't have to be done here though, it can be looked at later.

[rymurr]

agreed, I fixed this...was just being lazy ;-)

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[BryanCutler]

Since this is the relative bit index, it's not possible to return long. Can you change return value to an int?

[rymurr]

done

[BryanCutler]

From the comment above, I think this should still be an int

[rymurr]

done

[BryanCutler]

shouldn't this be long?

[rymurr]

yup! Done

[BryanCutler]

return int

[rymurr]

done

[BryanCutler]

Could you add a TODO to revisit once 64 bit vectors are supported?

[BryanCutler]

Also indicate this in the javadoc here and probably for the class. It might even be a good idea to raise an error if the user tries to add too many elements, otherwise things might just start looking wrong.

[rymurr]

done

[BryanCutler]

This should return long and not be casted to int

[rymurr]

done

[BryanCutler]

same here

[rymurr]

done

[BryanCutler]

could you add a check for Types.MinorType.LARGELIST here are remove the changes below?

[BryanCutler]

you also won't need typeBitWidth as an arg

[rymurr]

thanks, done

[rymurr]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

Thanks a lot for the thorough review. I have fixed up everything you mentioned. It appears some of the confusion was related to changes for 64-bit allocations that were recently merged and the rest was my ignorance!

I have pushed a change with all your recommended fixes.

[BryanCutler]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

[BryanCutler]

does this arg need to be cast to long?

[BryanCutler]

also here

[BryanCutler]

cast to long?

[BryanCutler]

was there a reason for this change?

[rymurr]

IDE autoformatting...reverted!

[BryanCutler]

why is this needed? the vector only supports integer max_value number of elements right?

[rymurr]

you are correct. Removed

[BryanCutler]

doesn't value need to be passed in here?

[rymurr]

value is always of type Void. None of the other visitors pass value to children.

[rymurr]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

Thanks for another thorough review @BryanCutler ! I think that I addressed everything. I may have went over the top w/ the casting but it doesn't hurt :-)

[BryanCutler]

LGTM

[BryanCutler]

merged to master, thanks @rymurr !

[rymurr]

Add large list and ensure it works with Integration tests. As noted in the JIRA
ticket this is rather limited as the underlying vector doesn't support int64 addressing

The important downcasts to int32 have been noted for a follow up once vectors with
long addresses are supported

[github-actions]

https://issues.apache.org/jira/browse/ARROW-6110

[emkornfield]

@BryanCutler would you have time to review?

[emkornfield]

@rymurr looks like this needs a rebase

[rymurr]

@rymurr looks like this needs a rebase

Thanks for the reminder @emkornfield, done.

[BryanCutler]

I'm a little swamped right now, but I'll try to review sometime this week

[liyafan82]

Here we need to cast the argument of getLong to long, to avoid integer overflow.

[rymurr]

done

[liyafan82]

Here we need to cast the first argument, too.

[rymurr]

done

[liyafan82]

I think we can replace the old implementation

void setBit(ArrowBuf validityBuffer, int index)

with this one, as now ArrowBuf is based on 64-bit index.

[rymurr]

done

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[rymurr]

done

[liyafan82]

Here we need to cast getLong argument to long.

[rymurr]

done

[liyafan82]

We need to cast (numRecords + 1) to long. Otherwise, integer overflow may happen before promoting the result to long.

[rymurr]

done

[nealrichardson]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

[BryanCutler]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

I'm taking a look now, I'd like to get it in for 1.0 too.

[BryanCutler]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

[BryanCutler]

It would be nice not to add a new template and combine with the UnionListWriter if possible. That doesn't have to be done here though, it can be looked at later.

[rymurr]

agreed, I fixed this...was just being lazy ;-)

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[BryanCutler]

Since this is the relative bit index, it's not possible to return long. Can you change return value to an int?

[rymurr]

done

[BryanCutler]

From the comment above, I think this should still be an int

[rymurr]

done

[BryanCutler]

shouldn't this be long?

[rymurr]

yup! Done

[BryanCutler]

return int

[rymurr]

done

[BryanCutler]

Could you add a TODO to revisit once 64 bit vectors are supported?

[BryanCutler]

Also indicate this in the javadoc here and probably for the class. It might even be a good idea to raise an error if the user tries to add too many elements, otherwise things might just start looking wrong.

[rymurr]

done

[BryanCutler]

This should return long and not be casted to int

[rymurr]

done

[BryanCutler]

same here

[rymurr]

done

[BryanCutler]

could you add a check for Types.MinorType.LARGELIST here are remove the changes below?

[BryanCutler]

you also won't need typeBitWidth as an arg

[rymurr]

thanks, done

[rymurr]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

Thanks a lot for the thorough review. I have fixed up everything you mentioned. It appears some of the confusion was related to changes for 64-bit allocations that were recently merged and the rest was my ignorance!

I have pushed a change with all your recommended fixes.

[BryanCutler]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

[BryanCutler]

does this arg need to be cast to long?

[BryanCutler]

also here

[BryanCutler]

cast to long?

[BryanCutler]

was there a reason for this change?

[rymurr]

IDE autoformatting...reverted!

[BryanCutler]

why is this needed? the vector only supports integer max_value number of elements right?

[rymurr]

you are correct. Removed

[BryanCutler]

doesn't value need to be passed in here?

[rymurr]

value is always of type Void. None of the other visitors pass value to children.

[rymurr]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

Thanks for another thorough review @BryanCutler ! I think that I addressed everything. I may have went over the top w/ the casting but it doesn't hurt :-)

[BryanCutler]

LGTM

[BryanCutler]

merged to master, thanks @rymurr !

[rymurr]

Add large list and ensure it works with Integration tests. As noted in the JIRA
ticket this is rather limited as the underlying vector doesn't support int64 addressing

The important downcasts to int32 have been noted for a follow up once vectors with
long addresses are supported

[github-actions]

https://issues.apache.org/jira/browse/ARROW-6110

[emkornfield]

@BryanCutler would you have time to review?

[emkornfield]

@rymurr looks like this needs a rebase

[rymurr]

@rymurr looks like this needs a rebase

Thanks for the reminder @emkornfield, done.

[BryanCutler]

I'm a little swamped right now, but I'll try to review sometime this week

[liyafan82]

Here we need to cast the argument of getLong to long, to avoid integer overflow.

[rymurr]

done

[liyafan82]

Here we need to cast the first argument, too.

[rymurr]

done

[liyafan82]

I think we can replace the old implementation

void setBit(ArrowBuf validityBuffer, int index)

with this one, as now ArrowBuf is based on 64-bit index.

[rymurr]

done

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[rymurr]

done

[liyafan82]

Here we need to cast getLong argument to long.

[rymurr]

done

[liyafan82]

We need to cast (numRecords + 1) to long. Otherwise, integer overflow may happen before promoting the result to long.

[rymurr]

done

[nealrichardson]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

[BryanCutler]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

I'm taking a look now, I'd like to get it in for 1.0 too.

[BryanCutler]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

[BryanCutler]

It would be nice not to add a new template and combine with the UnionListWriter if possible. That doesn't have to be done here though, it can be looked at later.

[rymurr]

agreed, I fixed this...was just being lazy ;-)

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[BryanCutler]

Since this is the relative bit index, it's not possible to return long. Can you change return value to an int?

[rymurr]

done

[BryanCutler]

From the comment above, I think this should still be an int

[rymurr]

done

[BryanCutler]

shouldn't this be long?

[rymurr]

yup! Done

[BryanCutler]

return int

[rymurr]

done

[BryanCutler]

Could you add a TODO to revisit once 64 bit vectors are supported?

[BryanCutler]

Also indicate this in the javadoc here and probably for the class. It might even be a good idea to raise an error if the user tries to add too many elements, otherwise things might just start looking wrong.

[rymurr]

done

[BryanCutler]

This should return long and not be casted to int

[rymurr]

done

[BryanCutler]

same here

[rymurr]

done

[BryanCutler]

could you add a check for Types.MinorType.LARGELIST here are remove the changes below?

[BryanCutler]

you also won't need typeBitWidth as an arg

[rymurr]

thanks, done

[rymurr]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

Thanks a lot for the thorough review. I have fixed up everything you mentioned. It appears some of the confusion was related to changes for 64-bit allocations that were recently merged and the rest was my ignorance!

I have pushed a change with all your recommended fixes.

[BryanCutler]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

[BryanCutler]

does this arg need to be cast to long?

[BryanCutler]

also here

[BryanCutler]

cast to long?

[BryanCutler]

was there a reason for this change?

[rymurr]

IDE autoformatting...reverted!

[BryanCutler]

why is this needed? the vector only supports integer max_value number of elements right?

[rymurr]

you are correct. Removed

[BryanCutler]

doesn't value need to be passed in here?

[rymurr]

value is always of type Void. None of the other visitors pass value to children.

[rymurr]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

Thanks for another thorough review @BryanCutler ! I think that I addressed everything. I may have went over the top w/ the casting but it doesn't hurt :-)

[BryanCutler]

LGTM

[BryanCutler]

merged to master, thanks @rymurr !

[rymurr]

Add large list and ensure it works with Integration tests. As noted in the JIRA
ticket this is rather limited as the underlying vector doesn't support int64 addressing

The important downcasts to int32 have been noted for a follow up once vectors with
long addresses are supported

[github-actions]

https://issues.apache.org/jira/browse/ARROW-6110

[emkornfield]

@BryanCutler would you have time to review?

[emkornfield]

@rymurr looks like this needs a rebase

[rymurr]

@rymurr looks like this needs a rebase

Thanks for the reminder @emkornfield, done.

[BryanCutler]

I'm a little swamped right now, but I'll try to review sometime this week

[liyafan82]

Here we need to cast the argument of getLong to long, to avoid integer overflow.

[rymurr]

done

[liyafan82]

Here we need to cast the first argument, too.

[rymurr]

done

[liyafan82]

I think we can replace the old implementation

void setBit(ArrowBuf validityBuffer, int index)

with this one, as now ArrowBuf is based on 64-bit index.

[rymurr]

done

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[rymurr]

done

[liyafan82]

Here we need to cast getLong argument to long.

[rymurr]

done

[liyafan82]

We need to cast (numRecords + 1) to long. Otherwise, integer overflow may happen before promoting the result to long.

[rymurr]

done

[nealrichardson]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

[BryanCutler]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

I'm taking a look now, I'd like to get it in for 1.0 too.

[BryanCutler]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

[BryanCutler]

It would be nice not to add a new template and combine with the UnionListWriter if possible. That doesn't have to be done here though, it can be looked at later.

[rymurr]

agreed, I fixed this...was just being lazy ;-)

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[BryanCutler]

Since this is the relative bit index, it's not possible to return long. Can you change return value to an int?

[rymurr]

done

[BryanCutler]

From the comment above, I think this should still be an int

[rymurr]

done

[BryanCutler]

shouldn't this be long?

[rymurr]

yup! Done

[BryanCutler]

return int

[rymurr]

done

[BryanCutler]

Could you add a TODO to revisit once 64 bit vectors are supported?

[BryanCutler]

Also indicate this in the javadoc here and probably for the class. It might even be a good idea to raise an error if the user tries to add too many elements, otherwise things might just start looking wrong.

[rymurr]

done

[BryanCutler]

This should return long and not be casted to int

[rymurr]

done

[BryanCutler]

same here

[rymurr]

done

[BryanCutler]

could you add a check for Types.MinorType.LARGELIST here are remove the changes below?

[BryanCutler]

you also won't need typeBitWidth as an arg

[rymurr]

thanks, done

[rymurr]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

Thanks a lot for the thorough review. I have fixed up everything you mentioned. It appears some of the confusion was related to changes for 64-bit allocations that were recently merged and the rest was my ignorance!

I have pushed a change with all your recommended fixes.

[BryanCutler]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

[BryanCutler]

does this arg need to be cast to long?

[BryanCutler]

also here

[BryanCutler]

cast to long?

[BryanCutler]

was there a reason for this change?

[rymurr]

IDE autoformatting...reverted!

[BryanCutler]

why is this needed? the vector only supports integer max_value number of elements right?

[rymurr]

you are correct. Removed

[BryanCutler]

doesn't value need to be passed in here?

[rymurr]

value is always of type Void. None of the other visitors pass value to children.

[rymurr]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

Thanks for another thorough review @BryanCutler ! I think that I addressed everything. I may have went over the top w/ the casting but it doesn't hurt :-)

[BryanCutler]

LGTM

[BryanCutler]

merged to master, thanks @rymurr !

[rymurr]

Add large list and ensure it works with Integration tests. As noted in the JIRA
ticket this is rather limited as the underlying vector doesn't support int64 addressing

The important downcasts to int32 have been noted for a follow up once vectors with
long addresses are supported

[github-actions]

https://issues.apache.org/jira/browse/ARROW-6110

[emkornfield]

@BryanCutler would you have time to review?

[emkornfield]

@rymurr looks like this needs a rebase

[rymurr]

@rymurr looks like this needs a rebase

Thanks for the reminder @emkornfield, done.

[BryanCutler]

I'm a little swamped right now, but I'll try to review sometime this week

[liyafan82]

Here we need to cast the argument of getLong to long, to avoid integer overflow.

[rymurr]

done

[liyafan82]

Here we need to cast the first argument, too.

[rymurr]

done

[liyafan82]

I think we can replace the old implementation

void setBit(ArrowBuf validityBuffer, int index)

with this one, as now ArrowBuf is based on 64-bit index.

[rymurr]

done

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[rymurr]

done

[liyafan82]

Here we need to cast getLong argument to long.

[rymurr]

done

[liyafan82]

We need to cast (numRecords + 1) to long. Otherwise, integer overflow may happen before promoting the result to long.

[rymurr]

done

[nealrichardson]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

[BryanCutler]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

I'm taking a look now, I'd like to get it in for 1.0 too.

[BryanCutler]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

[BryanCutler]

It would be nice not to add a new template and combine with the UnionListWriter if possible. That doesn't have to be done here though, it can be looked at later.

[rymurr]

agreed, I fixed this...was just being lazy ;-)

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[BryanCutler]

Since this is the relative bit index, it's not possible to return long. Can you change return value to an int?

[rymurr]

done

[BryanCutler]

From the comment above, I think this should still be an int

[rymurr]

done

[BryanCutler]

shouldn't this be long?

[rymurr]

yup! Done

[BryanCutler]

return int

[rymurr]

done

[BryanCutler]

Could you add a TODO to revisit once 64 bit vectors are supported?

[BryanCutler]

Also indicate this in the javadoc here and probably for the class. It might even be a good idea to raise an error if the user tries to add too many elements, otherwise things might just start looking wrong.

[rymurr]

done

[BryanCutler]

This should return long and not be casted to int

[rymurr]

done

[BryanCutler]

same here

[rymurr]

done

[BryanCutler]

could you add a check for Types.MinorType.LARGELIST here are remove the changes below?

[BryanCutler]

you also won't need typeBitWidth as an arg

[rymurr]

thanks, done

[rymurr]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

Thanks a lot for the thorough review. I have fixed up everything you mentioned. It appears some of the confusion was related to changes for 64-bit allocations that were recently merged and the rest was my ignorance!

I have pushed a change with all your recommended fixes.

[BryanCutler]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

[BryanCutler]

does this arg need to be cast to long?

[BryanCutler]

also here

[BryanCutler]

cast to long?

[BryanCutler]

was there a reason for this change?

[rymurr]

IDE autoformatting...reverted!

[BryanCutler]

why is this needed? the vector only supports integer max_value number of elements right?

[rymurr]

you are correct. Removed

[BryanCutler]

doesn't value need to be passed in here?

[rymurr]

value is always of type Void. None of the other visitors pass value to children.

[rymurr]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

Thanks for another thorough review @BryanCutler ! I think that I addressed everything. I may have went over the top w/ the casting but it doesn't hurt :-)

[BryanCutler]

LGTM

[BryanCutler]

merged to master, thanks @rymurr !

[rymurr]

Add large list and ensure it works with Integration tests. As noted in the JIRA
ticket this is rather limited as the underlying vector doesn't support int64 addressing

The important downcasts to int32 have been noted for a follow up once vectors with
long addresses are supported

[github-actions]

https://issues.apache.org/jira/browse/ARROW-6110

[emkornfield]

@BryanCutler would you have time to review?

[emkornfield]

@rymurr looks like this needs a rebase

[rymurr]

@rymurr looks like this needs a rebase

Thanks for the reminder @emkornfield, done.

[BryanCutler]

I'm a little swamped right now, but I'll try to review sometime this week

[liyafan82]

Here we need to cast the argument of getLong to long, to avoid integer overflow.

[rymurr]

done

[liyafan82]

Here we need to cast the first argument, too.

[rymurr]

done

[liyafan82]

I think we can replace the old implementation

void setBit(ArrowBuf validityBuffer, int index)

with this one, as now ArrowBuf is based on 64-bit index.

[rymurr]

done

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[rymurr]

done

[liyafan82]

Here we need to cast getLong argument to long.

[rymurr]

done

[liyafan82]

We need to cast (numRecords + 1) to long. Otherwise, integer overflow may happen before promoting the result to long.

[rymurr]

done

[nealrichardson]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

[BryanCutler]

Is this good to merge now? @BryanCutler are you still planning to review this? Would like to get this in 1.0.

I'm taking a look now, I'd like to get it in for 1.0 too.

[BryanCutler]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

[BryanCutler]

It would be nice not to add a new template and combine with the UnionListWriter if possible. That doesn't have to be done here though, it can be looked at later.

[rymurr]

agreed, I fixed this...was just being lazy ;-)

[BryanCutler]

please update the link in setValidityBitToOne here https://github.com/apache/arrow/pull/7275/files#diff-bc5e2a2a1b6b348c39f57c91defb855bL79

[BryanCutler]

Since this is the relative bit index, it's not possible to return long. Can you change return value to an int?

[rymurr]

done

[BryanCutler]

From the comment above, I think this should still be an int

[rymurr]

done

[BryanCutler]

shouldn't this be long?

[rymurr]

yup! Done

[BryanCutler]

return int

[rymurr]

done

[BryanCutler]

Could you add a TODO to revisit once 64 bit vectors are supported?

[BryanCutler]

Also indicate this in the javadoc here and probably for the class. It might even be a good idea to raise an error if the user tries to add too many elements, otherwise things might just start looking wrong.

[rymurr]

done

[BryanCutler]

This should return long and not be casted to int

[rymurr]

done

[BryanCutler]

same here

[rymurr]

done

[BryanCutler]

could you add a check for Types.MinorType.LARGELIST here are remove the changes below?

[BryanCutler]

you also won't need typeBitWidth as an arg

[rymurr]

thanks, done

[rymurr]

Thanks for working on this @rymurr ! Apologies for taking so long to review.. It looks pretty good, but I saw what looked like inconsistencies in the LargeListVector APIs using ints vs longs to me, and otherwise only minor things to fix up.

Thanks a lot for the thorough review. I have fixed up everything you mentioned. It appears some of the confusion was related to changes for 64-bit allocations that were recently merged and the rest was my ignorance!

I have pushed a change with all your recommended fixes.

[BryanCutler]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

[BryanCutler]

does this arg need to be cast to long?

[BryanCutler]

also here

[BryanCutler]

cast to long?

[BryanCutler]

was there a reason for this change?

[rymurr]

IDE autoformatting...reverted!

[BryanCutler]

why is this needed? the vector only supports integer max_value number of elements right?

[rymurr]

you are correct. Removed

[BryanCutler]

doesn't value need to be passed in here?

[rymurr]

value is always of type Void. None of the other visitors pass value to children.

[rymurr]

Thanks for the quick update @rymurr , it looks pretty good! Only a couple minor things. I see quite a few instances of offsetBuffer.getLong/setLong(i * OFFSET_WIDTH) that I believe need to be cast to long to avoid overflow. Could you take a quick pass and fix those up? I think we will be good to go after that.

Thanks for another thorough review @BryanCutler ! I think that I addressed everything. I may have went over the top w/ the casting but it doesn't hurt :-)

[BryanCutler]

LGTM

[BryanCutler]

merged to master, thanks @rymurr !