ARROW-17869: [Java][Gandiva] Implement Reserve for JavaBuffer

[js8544]

Java's buffer type doesn't support reserve, which causes java-jars to fail.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

https://issues.apache.org/jira/browse/ARROW-17869

[github-actions]

⚠️ Ticket has no components in JIRA, make sure you assign one.

[github-actions]

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

[github-actions]

Revision: 7f6d36cdbbf89f56f53e132198c86b0c004d119d

Submitted crossbow builds: ursacomputing/crossbow @ actions-d26b84437c

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@kou Hi kou, sorry to bother. I requested two crossbow builds today but they were ignored by the bot. Do you have any idea why this happened?

[kou]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 263c2956ee7c09ab1e74fc5e5329e0c55e28d84c

Submitted crossbow builds: ursacomputing/crossbow @ actions-7ce585f8a8

Task	Status
java-jars

[kou]

Hmm... It's strange...

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 1e7610799c2ee7724bb7a225bdfd2cafd0e811c5

Submitted crossbow builds: ursacomputing/crossbow @ actions-478721e321

Task	Status
java-jars

[js8544]

Brew's LLVM 15 enables z3 by default, and it's shared linked. So at task Build Java Jars, it's required to have libz3 installed on the system. There are several ways to solve this:

1. Add a brew install z3 task to Build Java Jars task
2. Pin LLVM for java to 14. Currently it uses arrow/cpp/Brewfile. We can create another Brewfile in arrow/java and pin it in the new file.
3. We can also pin LLVM to 14 for all MacOS builds at arrow/cpp/Brewfile. Because LLVM 15 is noticably slower than LLVM 14 somehow. It can also speed up the build time for other MacOS CI tasks.

Which way is preferred? cc @kou@pitrou

[kou]

3. for now but we need a new Jira issue for unpinning.

[js8544]

3. for now but we need a new Jira issue for unpinning.

I've pinned Brew llvm version to 14. Also filed https://issues.apache.org/jira/browse/ARROW-17902 to track current workarounds for LLVM 15 related issues.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 39437b059b466a00de6778034610600de6e15d0c

Submitted crossbow builds: ursacomputing/crossbow @ actions-e128cce1c4

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: dfb7e2d71450a3bff7191d2dda633c85d618b035

Submitted crossbow builds: ursacomputing/crossbow @ actions-77a61b58b9

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 5710f8b

Submitted crossbow builds: ursacomputing/crossbow @ actions-632d4e01ac

Task	Status
java-jars

[pitrou]

Hmm... it would be much more desirable to implement Reserve on Java-exported buffers instead. @lwhite1@lidavidm Could one of you help perhaps?

[lidavidm]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

[js8544]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

Because the resize implementation for SimpleBuffer doesn't amotize the realloc costs, if we don't reserve, there will be N reallocs, one for each resize call. Now it's reduced to log2(N), since with each reserve we double the buffer size. We still need to call resize to track the actual size.

[lidavidm]

Ah, thanks for the explanation. I think it would be best if we could implement reserve for Java-allocated buffers, but I am not familiar with the code and don't have time to look at it.

[lidavidm]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

[js8544]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

It seems like we can. I can take a look later today if you don't have time for it.

[pitrou]

@js8544 Are you closing this PR in favor of another one?

[js8544]

@js8544 Are you closing this PR in favor of another one?

@pitrou Yes, I will submit another PR once #14310 is merged since this change requires llvm@14.

[js8544]

Java's buffer type doesn't support reserve, which causes java-jars to fail.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

https://issues.apache.org/jira/browse/ARROW-17869

[github-actions]

⚠️ Ticket has no components in JIRA, make sure you assign one.

[github-actions]

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

[github-actions]

Revision: 7f6d36cdbbf89f56f53e132198c86b0c004d119d

Submitted crossbow builds: ursacomputing/crossbow @ actions-d26b84437c

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@kou Hi kou, sorry to bother. I requested two crossbow builds today but they were ignored by the bot. Do you have any idea why this happened?

[kou]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 263c2956ee7c09ab1e74fc5e5329e0c55e28d84c

Submitted crossbow builds: ursacomputing/crossbow @ actions-7ce585f8a8

Task	Status
java-jars

[kou]

Hmm... It's strange...

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 1e7610799c2ee7724bb7a225bdfd2cafd0e811c5

Submitted crossbow builds: ursacomputing/crossbow @ actions-478721e321

Task	Status
java-jars

[js8544]

Brew's LLVM 15 enables z3 by default, and it's shared linked. So at task Build Java Jars, it's required to have libz3 installed on the system. There are several ways to solve this:

1. Add a brew install z3 task to Build Java Jars task
2. Pin LLVM for java to 14. Currently it uses arrow/cpp/Brewfile. We can create another Brewfile in arrow/java and pin it in the new file.
3. We can also pin LLVM to 14 for all MacOS builds at arrow/cpp/Brewfile. Because LLVM 15 is noticably slower than LLVM 14 somehow. It can also speed up the build time for other MacOS CI tasks.

Which way is preferred? cc @kou@pitrou

[kou]

3. for now but we need a new Jira issue for unpinning.

[js8544]

3. for now but we need a new Jira issue for unpinning.

I've pinned Brew llvm version to 14. Also filed https://issues.apache.org/jira/browse/ARROW-17902 to track current workarounds for LLVM 15 related issues.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 39437b059b466a00de6778034610600de6e15d0c

Submitted crossbow builds: ursacomputing/crossbow @ actions-e128cce1c4

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: dfb7e2d71450a3bff7191d2dda633c85d618b035

Submitted crossbow builds: ursacomputing/crossbow @ actions-77a61b58b9

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 5710f8b

Submitted crossbow builds: ursacomputing/crossbow @ actions-632d4e01ac

Task	Status
java-jars

[pitrou]

Hmm... it would be much more desirable to implement Reserve on Java-exported buffers instead. @lwhite1@lidavidm Could one of you help perhaps?

[lidavidm]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

[js8544]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

Because the resize implementation for SimpleBuffer doesn't amotize the realloc costs, if we don't reserve, there will be N reallocs, one for each resize call. Now it's reduced to log2(N), since with each reserve we double the buffer size. We still need to call resize to track the actual size.

[lidavidm]

Ah, thanks for the explanation. I think it would be best if we could implement reserve for Java-allocated buffers, but I am not familiar with the code and don't have time to look at it.

[lidavidm]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

[js8544]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

It seems like we can. I can take a look later today if you don't have time for it.

[pitrou]

@js8544 Are you closing this PR in favor of another one?

[js8544]

@js8544 Are you closing this PR in favor of another one?

@pitrou Yes, I will submit another PR once #14310 is merged since this change requires llvm@14.

[js8544]

Java's buffer type doesn't support reserve, which causes java-jars to fail.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

https://issues.apache.org/jira/browse/ARROW-17869

[github-actions]

⚠️ Ticket has no components in JIRA, make sure you assign one.

[github-actions]

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

[github-actions]

Revision: 7f6d36cdbbf89f56f53e132198c86b0c004d119d

Submitted crossbow builds: ursacomputing/crossbow @ actions-d26b84437c

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@kou Hi kou, sorry to bother. I requested two crossbow builds today but they were ignored by the bot. Do you have any idea why this happened?

[kou]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 263c2956ee7c09ab1e74fc5e5329e0c55e28d84c

Submitted crossbow builds: ursacomputing/crossbow @ actions-7ce585f8a8

Task	Status
java-jars

[kou]

Hmm... It's strange...

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 1e7610799c2ee7724bb7a225bdfd2cafd0e811c5

Submitted crossbow builds: ursacomputing/crossbow @ actions-478721e321

Task	Status
java-jars

[js8544]

Brew's LLVM 15 enables z3 by default, and it's shared linked. So at task Build Java Jars, it's required to have libz3 installed on the system. There are several ways to solve this:

1. Add a brew install z3 task to Build Java Jars task
2. Pin LLVM for java to 14. Currently it uses arrow/cpp/Brewfile. We can create another Brewfile in arrow/java and pin it in the new file.
3. We can also pin LLVM to 14 for all MacOS builds at arrow/cpp/Brewfile. Because LLVM 15 is noticably slower than LLVM 14 somehow. It can also speed up the build time for other MacOS CI tasks.

Which way is preferred? cc @kou@pitrou

[kou]

3. for now but we need a new Jira issue for unpinning.

[js8544]

3. for now but we need a new Jira issue for unpinning.

I've pinned Brew llvm version to 14. Also filed https://issues.apache.org/jira/browse/ARROW-17902 to track current workarounds for LLVM 15 related issues.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 39437b059b466a00de6778034610600de6e15d0c

Submitted crossbow builds: ursacomputing/crossbow @ actions-e128cce1c4

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: dfb7e2d71450a3bff7191d2dda633c85d618b035

Submitted crossbow builds: ursacomputing/crossbow @ actions-77a61b58b9

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 5710f8b

Submitted crossbow builds: ursacomputing/crossbow @ actions-632d4e01ac

Task	Status
java-jars

[pitrou]

Hmm... it would be much more desirable to implement Reserve on Java-exported buffers instead. @lwhite1@lidavidm Could one of you help perhaps?

[lidavidm]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

[js8544]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

Because the resize implementation for SimpleBuffer doesn't amotize the realloc costs, if we don't reserve, there will be N reallocs, one for each resize call. Now it's reduced to log2(N), since with each reserve we double the buffer size. We still need to call resize to track the actual size.

[lidavidm]

Ah, thanks for the explanation. I think it would be best if we could implement reserve for Java-allocated buffers, but I am not familiar with the code and don't have time to look at it.

[lidavidm]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

[js8544]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

It seems like we can. I can take a look later today if you don't have time for it.

[pitrou]

@js8544 Are you closing this PR in favor of another one?

[js8544]

@js8544 Are you closing this PR in favor of another one?

@pitrou Yes, I will submit another PR once #14310 is merged since this change requires llvm@14.

[js8544]

Java's buffer type doesn't support reserve, which causes java-jars to fail.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

https://issues.apache.org/jira/browse/ARROW-17869

[github-actions]

⚠️ Ticket has no components in JIRA, make sure you assign one.

[github-actions]

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

[github-actions]

Revision: 7f6d36cdbbf89f56f53e132198c86b0c004d119d

Submitted crossbow builds: ursacomputing/crossbow @ actions-d26b84437c

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@kou Hi kou, sorry to bother. I requested two crossbow builds today but they were ignored by the bot. Do you have any idea why this happened?

[kou]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 263c2956ee7c09ab1e74fc5e5329e0c55e28d84c

Submitted crossbow builds: ursacomputing/crossbow @ actions-7ce585f8a8

Task	Status
java-jars

[kou]

Hmm... It's strange...

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 1e7610799c2ee7724bb7a225bdfd2cafd0e811c5

Submitted crossbow builds: ursacomputing/crossbow @ actions-478721e321

Task	Status
java-jars

[js8544]

Brew's LLVM 15 enables z3 by default, and it's shared linked. So at task Build Java Jars, it's required to have libz3 installed on the system. There are several ways to solve this:

1. Add a brew install z3 task to Build Java Jars task
2. Pin LLVM for java to 14. Currently it uses arrow/cpp/Brewfile. We can create another Brewfile in arrow/java and pin it in the new file.
3. We can also pin LLVM to 14 for all MacOS builds at arrow/cpp/Brewfile. Because LLVM 15 is noticably slower than LLVM 14 somehow. It can also speed up the build time for other MacOS CI tasks.

Which way is preferred? cc @kou@pitrou

[kou]

3. for now but we need a new Jira issue for unpinning.

[js8544]

3. for now but we need a new Jira issue for unpinning.

I've pinned Brew llvm version to 14. Also filed https://issues.apache.org/jira/browse/ARROW-17902 to track current workarounds for LLVM 15 related issues.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 39437b059b466a00de6778034610600de6e15d0c

Submitted crossbow builds: ursacomputing/crossbow @ actions-e128cce1c4

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: dfb7e2d71450a3bff7191d2dda633c85d618b035

Submitted crossbow builds: ursacomputing/crossbow @ actions-77a61b58b9

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 5710f8b

Submitted crossbow builds: ursacomputing/crossbow @ actions-632d4e01ac

Task	Status
java-jars

[pitrou]

Hmm... it would be much more desirable to implement Reserve on Java-exported buffers instead. @lwhite1@lidavidm Could one of you help perhaps?

[lidavidm]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

[js8544]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

Because the resize implementation for SimpleBuffer doesn't amotize the realloc costs, if we don't reserve, there will be N reallocs, one for each resize call. Now it's reduced to log2(N), since with each reserve we double the buffer size. We still need to call resize to track the actual size.

[lidavidm]

Ah, thanks for the explanation. I think it would be best if we could implement reserve for Java-allocated buffers, but I am not familiar with the code and don't have time to look at it.

[lidavidm]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

[js8544]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

It seems like we can. I can take a look later today if you don't have time for it.

[pitrou]

@js8544 Are you closing this PR in favor of another one?

[js8544]

@js8544 Are you closing this PR in favor of another one?

@pitrou Yes, I will submit another PR once #14310 is merged since this change requires llvm@14.

[js8544]

Java's buffer type doesn't support reserve, which causes java-jars to fail.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

https://issues.apache.org/jira/browse/ARROW-17869

[github-actions]

⚠️ Ticket has no components in JIRA, make sure you assign one.

[github-actions]

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

[github-actions]

Revision: 7f6d36cdbbf89f56f53e132198c86b0c004d119d

Submitted crossbow builds: ursacomputing/crossbow @ actions-d26b84437c

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@kou Hi kou, sorry to bother. I requested two crossbow builds today but they were ignored by the bot. Do you have any idea why this happened?

[kou]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 263c2956ee7c09ab1e74fc5e5329e0c55e28d84c

Submitted crossbow builds: ursacomputing/crossbow @ actions-7ce585f8a8

Task	Status
java-jars

[kou]

Hmm... It's strange...

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 1e7610799c2ee7724bb7a225bdfd2cafd0e811c5

Submitted crossbow builds: ursacomputing/crossbow @ actions-478721e321

Task	Status
java-jars

[js8544]

Brew's LLVM 15 enables z3 by default, and it's shared linked. So at task Build Java Jars, it's required to have libz3 installed on the system. There are several ways to solve this:

1. Add a brew install z3 task to Build Java Jars task
2. Pin LLVM for java to 14. Currently it uses arrow/cpp/Brewfile. We can create another Brewfile in arrow/java and pin it in the new file.
3. We can also pin LLVM to 14 for all MacOS builds at arrow/cpp/Brewfile. Because LLVM 15 is noticably slower than LLVM 14 somehow. It can also speed up the build time for other MacOS CI tasks.

Which way is preferred? cc @kou@pitrou

[kou]

3. for now but we need a new Jira issue for unpinning.

[js8544]

3. for now but we need a new Jira issue for unpinning.

I've pinned Brew llvm version to 14. Also filed https://issues.apache.org/jira/browse/ARROW-17902 to track current workarounds for LLVM 15 related issues.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 39437b059b466a00de6778034610600de6e15d0c

Submitted crossbow builds: ursacomputing/crossbow @ actions-e128cce1c4

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: dfb7e2d71450a3bff7191d2dda633c85d618b035

Submitted crossbow builds: ursacomputing/crossbow @ actions-77a61b58b9

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 5710f8b

Submitted crossbow builds: ursacomputing/crossbow @ actions-632d4e01ac

Task	Status
java-jars

[pitrou]

Hmm... it would be much more desirable to implement Reserve on Java-exported buffers instead. @lwhite1@lidavidm Could one of you help perhaps?

[lidavidm]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

[js8544]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

Because the resize implementation for SimpleBuffer doesn't amotize the realloc costs, if we don't reserve, there will be N reallocs, one for each resize call. Now it's reduced to log2(N), since with each reserve we double the buffer size. We still need to call resize to track the actual size.

[lidavidm]

Ah, thanks for the explanation. I think it would be best if we could implement reserve for Java-allocated buffers, but I am not familiar with the code and don't have time to look at it.

[lidavidm]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

[js8544]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

It seems like we can. I can take a look later today if you don't have time for it.

[pitrou]

@js8544 Are you closing this PR in favor of another one?

[js8544]

@js8544 Are you closing this PR in favor of another one?

@pitrou Yes, I will submit another PR once #14310 is merged since this change requires llvm@14.

[js8544]

Java's buffer type doesn't support reserve, which causes java-jars to fail.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

https://issues.apache.org/jira/browse/ARROW-17869

[github-actions]

⚠️ Ticket has no components in JIRA, make sure you assign one.

[github-actions]

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

[github-actions]

Revision: 7f6d36cdbbf89f56f53e132198c86b0c004d119d

Submitted crossbow builds: ursacomputing/crossbow @ actions-d26b84437c

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@kou Hi kou, sorry to bother. I requested two crossbow builds today but they were ignored by the bot. Do you have any idea why this happened?

[kou]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 263c2956ee7c09ab1e74fc5e5329e0c55e28d84c

Submitted crossbow builds: ursacomputing/crossbow @ actions-7ce585f8a8

Task	Status
java-jars

[kou]

Hmm... It's strange...

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 1e7610799c2ee7724bb7a225bdfd2cafd0e811c5

Submitted crossbow builds: ursacomputing/crossbow @ actions-478721e321

Task	Status
java-jars

[js8544]

Brew's LLVM 15 enables z3 by default, and it's shared linked. So at task Build Java Jars, it's required to have libz3 installed on the system. There are several ways to solve this:

1. Add a brew install z3 task to Build Java Jars task
2. Pin LLVM for java to 14. Currently it uses arrow/cpp/Brewfile. We can create another Brewfile in arrow/java and pin it in the new file.
3. We can also pin LLVM to 14 for all MacOS builds at arrow/cpp/Brewfile. Because LLVM 15 is noticably slower than LLVM 14 somehow. It can also speed up the build time for other MacOS CI tasks.

Which way is preferred? cc @kou@pitrou

[kou]

3. for now but we need a new Jira issue for unpinning.

[js8544]

3. for now but we need a new Jira issue for unpinning.

I've pinned Brew llvm version to 14. Also filed https://issues.apache.org/jira/browse/ARROW-17902 to track current workarounds for LLVM 15 related issues.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 39437b059b466a00de6778034610600de6e15d0c

Submitted crossbow builds: ursacomputing/crossbow @ actions-e128cce1c4

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: dfb7e2d71450a3bff7191d2dda633c85d618b035

Submitted crossbow builds: ursacomputing/crossbow @ actions-77a61b58b9

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 5710f8b

Submitted crossbow builds: ursacomputing/crossbow @ actions-632d4e01ac

Task	Status
java-jars

[pitrou]

Hmm... it would be much more desirable to implement Reserve on Java-exported buffers instead. @lwhite1@lidavidm Could one of you help perhaps?

[lidavidm]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

[js8544]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

Because the resize implementation for SimpleBuffer doesn't amotize the realloc costs, if we don't reserve, there will be N reallocs, one for each resize call. Now it's reduced to log2(N), since with each reserve we double the buffer size. We still need to call resize to track the actual size.

[lidavidm]

Ah, thanks for the explanation. I think it would be best if we could implement reserve for Java-allocated buffers, but I am not familiar with the code and don't have time to look at it.

[lidavidm]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

[js8544]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

It seems like we can. I can take a look later today if you don't have time for it.

[pitrou]

@js8544 Are you closing this PR in favor of another one?

[js8544]

@js8544 Are you closing this PR in favor of another one?

@pitrou Yes, I will submit another PR once #14310 is merged since this change requires llvm@14.

[js8544]

Java's buffer type doesn't support reserve, which causes java-jars to fail.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

https://issues.apache.org/jira/browse/ARROW-17869

[github-actions]

⚠️ Ticket has no components in JIRA, make sure you assign one.

[github-actions]

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

[github-actions]

Revision: 7f6d36cdbbf89f56f53e132198c86b0c004d119d

Submitted crossbow builds: ursacomputing/crossbow @ actions-d26b84437c

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@kou Hi kou, sorry to bother. I requested two crossbow builds today but they were ignored by the bot. Do you have any idea why this happened?

[kou]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 263c2956ee7c09ab1e74fc5e5329e0c55e28d84c

Submitted crossbow builds: ursacomputing/crossbow @ actions-7ce585f8a8

Task	Status
java-jars

[kou]

Hmm... It's strange...

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 1e7610799c2ee7724bb7a225bdfd2cafd0e811c5

Submitted crossbow builds: ursacomputing/crossbow @ actions-478721e321

Task	Status
java-jars

[js8544]

Brew's LLVM 15 enables z3 by default, and it's shared linked. So at task Build Java Jars, it's required to have libz3 installed on the system. There are several ways to solve this:

1. Add a brew install z3 task to Build Java Jars task
2. Pin LLVM for java to 14. Currently it uses arrow/cpp/Brewfile. We can create another Brewfile in arrow/java and pin it in the new file.
3. We can also pin LLVM to 14 for all MacOS builds at arrow/cpp/Brewfile. Because LLVM 15 is noticably slower than LLVM 14 somehow. It can also speed up the build time for other MacOS CI tasks.

Which way is preferred? cc @kou@pitrou

[kou]

3. for now but we need a new Jira issue for unpinning.

[js8544]

3. for now but we need a new Jira issue for unpinning.

I've pinned Brew llvm version to 14. Also filed https://issues.apache.org/jira/browse/ARROW-17902 to track current workarounds for LLVM 15 related issues.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 39437b059b466a00de6778034610600de6e15d0c

Submitted crossbow builds: ursacomputing/crossbow @ actions-e128cce1c4

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: dfb7e2d71450a3bff7191d2dda633c85d618b035

Submitted crossbow builds: ursacomputing/crossbow @ actions-77a61b58b9

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 5710f8b

Submitted crossbow builds: ursacomputing/crossbow @ actions-632d4e01ac

Task	Status
java-jars

[pitrou]

Hmm... it would be much more desirable to implement Reserve on Java-exported buffers instead. @lwhite1@lidavidm Could one of you help perhaps?

[lidavidm]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

[js8544]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

Because the resize implementation for SimpleBuffer doesn't amotize the realloc costs, if we don't reserve, there will be N reallocs, one for each resize call. Now it's reduced to log2(N), since with each reserve we double the buffer size. We still need to call resize to track the actual size.

[lidavidm]

Ah, thanks for the explanation. I think it would be best if we could implement reserve for Java-allocated buffers, but I am not familiar with the code and don't have time to look at it.

[lidavidm]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

[js8544]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

It seems like we can. I can take a look later today if you don't have time for it.

[pitrou]

@js8544 Are you closing this PR in favor of another one?

[js8544]

@js8544 Are you closing this PR in favor of another one?

@pitrou Yes, I will submit another PR once #14310 is merged since this change requires llvm@14.

[js8544]

Java's buffer type doesn't support reserve, which causes java-jars to fail.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

https://issues.apache.org/jira/browse/ARROW-17869

[github-actions]

⚠️ Ticket has no components in JIRA, make sure you assign one.

[github-actions]

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

[github-actions]

Revision: 7f6d36cdbbf89f56f53e132198c86b0c004d119d

Submitted crossbow builds: ursacomputing/crossbow @ actions-d26b84437c

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@kou Hi kou, sorry to bother. I requested two crossbow builds today but they were ignored by the bot. Do you have any idea why this happened?

[kou]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 263c2956ee7c09ab1e74fc5e5329e0c55e28d84c

Submitted crossbow builds: ursacomputing/crossbow @ actions-7ce585f8a8

Task	Status
java-jars

[kou]

Hmm... It's strange...

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 1e7610799c2ee7724bb7a225bdfd2cafd0e811c5

Submitted crossbow builds: ursacomputing/crossbow @ actions-478721e321

Task	Status
java-jars

[js8544]

Brew's LLVM 15 enables z3 by default, and it's shared linked. So at task Build Java Jars, it's required to have libz3 installed on the system. There are several ways to solve this:

1. Add a brew install z3 task to Build Java Jars task
2. Pin LLVM for java to 14. Currently it uses arrow/cpp/Brewfile. We can create another Brewfile in arrow/java and pin it in the new file.
3. We can also pin LLVM to 14 for all MacOS builds at arrow/cpp/Brewfile. Because LLVM 15 is noticably slower than LLVM 14 somehow. It can also speed up the build time for other MacOS CI tasks.

Which way is preferred? cc @kou@pitrou

[kou]

3. for now but we need a new Jira issue for unpinning.

[js8544]

3. for now but we need a new Jira issue for unpinning.

I've pinned Brew llvm version to 14. Also filed https://issues.apache.org/jira/browse/ARROW-17902 to track current workarounds for LLVM 15 related issues.

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 39437b059b466a00de6778034610600de6e15d0c

Submitted crossbow builds: ursacomputing/crossbow @ actions-e128cce1c4

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: dfb7e2d71450a3bff7191d2dda633c85d618b035

Submitted crossbow builds: ursacomputing/crossbow @ actions-77a61b58b9

Task	Status
java-jars

[js8544]

@github-actions crossbow submit java-jars

[js8544]

@github-actions crossbow submit java-jars

[github-actions]

Revision: 5710f8b

Submitted crossbow builds: ursacomputing/crossbow @ actions-632d4e01ac

Task	Status
java-jars

[pitrou]

Hmm... it would be much more desirable to implement Reserve on Java-exported buffers instead. @lwhite1@lidavidm Could one of you help perhaps?

[lidavidm]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

[js8544]

From a quick glance only: why are we reserving if we're resizing immediately afterwards anyways?

Because the resize implementation for SimpleBuffer doesn't amotize the realloc costs, if we don't reserve, there will be N reallocs, one for each resize call. Now it's reduced to log2(N), since with each reserve we double the buffer size. We still need to call resize to track the actual size.

[lidavidm]

Ah, thanks for the explanation. I think it would be best if we could implement reserve for Java-allocated buffers, but I am not familiar with the code and don't have time to look at it.

[lidavidm]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

[js8544]

That said: it seems like we could recycle the logic for resize here to also implement reserve?

arrow/java/gandiva/src/main/cpp/jni_common.cc

Lines 715 to 753 in 5fd54bc

	Status Reserve(constint64_t new_capacity) override {
	returnStatus::NotImplemented("reserve not implemented");
	}
	private:
	JNIEnv* env_;
	jobject jexpander_;
	int32_t vector_idx_;
	};
	Status JavaResizableBuffer::Resize(constint64_t new_size, bool shrink_to_fit) {
	if (shrink_to_fit == true) {
	returnStatus::NotImplemented("shrink not implemented");
	}
	if (ARROW_PREDICT_TRUE(new_size < capacity())) {
	// no need to expand.
	size_ = new_size;
	returnStatus::OK();
	}
	// callback into java to expand the buffer
	jobject ret =
	env_->CallObjectMethod(jexpander_, vector_expander_method_, vector_idx_, new_size);
	if (env_->ExceptionCheck()) {
	env_->ExceptionDescribe();
	env_->ExceptionClear();
	returnStatus::OutOfMemory("buffer expand failed in java");
	}
	jlong ret_address = env_->GetLongField(ret, vector_expander_ret_address_);
	jlong ret_capacity = env_->GetLongField(ret, vector_expander_ret_capacity_);
	DCHECK_GE(ret_capacity, new_size);
	data_ = reinterpret_cast<uint8_t*>(ret_address);
	size_ = new_size;
	capacity_ = ret_capacity;
	returnStatus::OK();
	}

It seems like we can. I can take a look later today if you don't have time for it.

[pitrou]

@js8544 Are you closing this PR in favor of another one?

[js8544]

@js8544 Are you closing this PR in favor of another one?

@pitrou Yes, I will submit another PR once #14310 is merged since this change requires llvm@14.