[CI][Archery] Archery linking should also check for undefined symbols

[raulcd]

In order to avoid issues like the one referenced below:

arrow/dev/archery/archery/linking.py

Lines 65 to 75 in a0dec7f

	def check_dynamic_library_dependencies(path, allowed, disallowed):
	dylib = DynamicLibrary(path)
	for dep in dylib.list_dependency_names():
	if allowed and dep not in allowed:
	raise DependencyError(
	f"Unexpected shared dependency found in {dylib.path}: `{dep}`"
	)
	if disallowed and dep in disallowed:
	raise DependencyError(
	f"Disallowed shared dependency found in {dylib.path}: `{dep}`"
	)

should also check undefined symbols.

It seems that #39137 is the cause of this. Sorry...

Originally posted by @kou in #39919 (comment)

List of related issues:

• [CI][Archery] Archery linking should also check for undefined symbols Linux #40964
• [CI][Archery] Archery linking should also check for undefined symbols macOS #40965
• [CI][Archery] Archery linking should also check for undefined symbols Windows #40966

[danepitkin]

Hey @kou , by chance is this something you could take on?

[kou]

I didn't have a plan to do this but I may work on this later.

BTW, I think that this is not difficult (at least for .so). We just need check symbols detected by nm XXX.so | grep '^ *U '.

[danepitkin]

@raulcd This should probably be something we add to 15.0.2 as a failsafe.

[vibhatha]

@kou something like this could work right?

# Check for undefined symbols
result = subprocess.run(['nm', '-u', path], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
undefined_symbols = result.stdout.decode('utf-8')
if undefined_symbols:
 raise DependencyError(
 f"Undefined symbols found in {dylib.path}: {undefined_symbols}"
 )

How can we test this though? cc @raulcd

[kou]

Wow! I didn't know the nm's -u option!
But your code will not work because there are some expected undefined symbols. They are symbols provided by linked libraries. See --allow options for linked libraries:

arrow/ci/scripts/java_jni_manylinux_build.sh

Lines 155 to 165 in 0ce7267

	--allow ld-linux-aarch64 \
	--allow ld-linux-x86-64 \
	--allow libc \
	--allow libdl \
	--allow libgcc_s \
	--allow libm \
	--allow libpthread \
	--allow librt \
	--allow libstdc++ \
	--allow libz \
	--allow linux-vdso \

We need to remove expected undefined symbols from undefined_symbols. If there are still one or more undefined symbols, we should report an error.

[kou]

We can test this feature by downloading arrow-dataset-X.Y.Z.jar (see #39919 (comment) for the URLs of them) and run the following command line:

archery linking check-dependencies \
 --allow ld-linux-aarch64 \
 --allow ld-linux-x86-64 \
 --allow libc \
 --allow libdl \
 --allow libgcc_s \
 --allow libm \
 --allow libpthread \
 --allow librt \
 --allow libstdc++ \
 --allow libz \
 --allow linux-vdso \
 arrow-dataset-X.Y.Z/x86_64/libarrow_dataset_jni.so

[vibhatha]

@kou thanks for this wonderful guideline. I will work on this.