ARROW-5082: [Python] Stop exporting copies of shared libraries in wheel

[fsaintjacques]

This reduces the size from 50mb to 28mb. I haven't tested if this breaks the OSX wheel. Note that the fix is brittle since currently (on Linux) it links with the full-versioned shared library binary. This works out due to the -len(x) sorting applied. A proper fix would be to keep the one linked (found via ldd or some other methods). I suspect that auditwheel and subsequent install & test will catch this if the contract were to change.

[wesm]

Is there any documentation about this?

[kou]

pyarrow Debian package doesn't exist.
Source distribution is needed to use pyarrow with libarrow Debian package.

[fsaintjacques]

Could you care to explain, I'm missing some pieces. My comment was regarding uwe comment on zulip.

[pitrou]

In any case, I'm not sure why this needs to be here. Anyone can build a source distribution from scratch (as the name suggests, it just packages source code together).

[xhochy]

This gives us a reproducible and controlled environment to build a source tarball. You still need the right versions of setuptools amd friends to actually support Markdown in the description field of the package information.

[wesm]

If the user wants to link a C++ library to the libarrow that is bundled with wheels, will it still work if there is only libarrow.so.14 there?

[xhochy]

Tensorflow is doing the same. Downstream users might need to add something to their linker commands though.

[fsaintjacques]

Side note, when I extracted the tensorflow packages, I noted they used the versionless path.

[wesm]

@kszucs can we run the Crossbow wheel builds before merging this? Also want to wait for @xhochy to review

[kszucs]

@ursabot crossbow --help

[ursabot]

Usage: @ursabot crossbow [OPTIONS] COMMAND [ARGS]...
Trigger crossbow builds for this pull request
Options:
--help Show this message and exit.
Commands:
package Submit crossbow packaging tasks.
test Submit crossbow testing tasks.

[kszucs]

@ursabot crossbow package wheel

[ursabot]

AMD64 Conda Crossbow (#19707) builder has been succeeded.

Revision: 15932d0

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-10

[kszucs]

Most of the wheel builds will fail mostly because of the missing OpenSSL dependency - so OpenSSL should be turned off.

[nealrichardson]

#4494 may also fix the OpenSSL issue, if we can get that in.

[kszucs]

@nealrichardson You may try to execute the same ursabot command, see whether the PR fixes the wheel builds (execute @ursabot crossbow package wheel conda linux to run everything).

[wesm]

How important is it to have the shared library with the ABI version tag in these wheels? I see two options:

• Ship libarrow.so.$ABI_VERSION
• Ship unversioned libarrow.so

Either of these is OK with me. @xhochy do you have an opinion so we can get this closed out?

[xhochy]

I would prefer the ABI version in the name as this is the more graceful error for the end user than getting a segmentation fault. We should test first if users of the wheel (read turbodbc) need to adjust their build system or whether the ABI named version is still picked up

[wesm]

OK, who wants to do that? We're reaching the critical horizon for 0.14 so need to get this issue closed out and move on to the other backlog items

[fsaintjacques]

I think I've reached my limit of wheels debugging. @xhochy can you pickup the validation of turbobc?

[wesm]

I can take this for a spin tomorrow (Tuesday) if no one else volunteers since I have built turbodbc before

[wesm]

I've tested the wheel with turbodbc locally and turbodbc seems to be fine with linking to the versioned .so file

(note that it's saying "not found" because the pyarrow wheel directory is not in my LD_LIBRARY_PATH)

$ ldd turbodbc_arrow_support.cpython-37m-x86_64-linux-gnu.so linux-vdso.so.1 (0x00007ffe59771000)
libturbodbc.so => /home/wesm/code/turbodbc/build/libturbodbc.so (0x00007f72a07aa000)
libarrow.so.14 => not found
libarrow_python.so.14 => not found
libcpp_odbc.so => /home/wesm/code/turbodbc/build/cpp/cpp_odbc/Library/libcpp_odbc.so (0x00007f72a0724000)
libstdc++.so.6 => /home/wesm/cpp-runtime-toolchain/lib/libstdc++.so.6 (0x00007f72a05e2000)
libgcc_s.so.1 => /home/wesm/cpp-runtime-toolchain/lib/libgcc_s.so.1 (0x00007f72a05cc000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f72a03e1000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f72a0293000)
/lib64/ld-linux-x86-64.so.2 (0x00007f72a0949000)
libodbc.so.2 => /home/wesm/miniconda/envs/turbodbc-dev/lib/libodbc.so.2 (0x00007f72a0212000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f72a020c000)
libiconv.so.2 => /home/wesm/miniconda/envs/turbodbc-dev/lib/./libiconv.so.2 (0x00007f72a0122000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f72a0101000)

[wesm]

+1