SIGSEGV when Python tests use the `ctypes` module

[P403n1x87]

Description:

When using the Python installed with this action I run into SIGSEGV on Linux if my tests use ctypes. This is is the offending workflow

name: Tests
on: push
concurrency: 
 group: ${{ github.head_ref || github.run_id }}
 cancel-in-progress: true
jobs:
 tests-linux:
 runs-on: ubuntu-latest
 
 strategy:
 fail-fast: false
 matrix:
 python-version: ["2.7", "3.5", "3.6", "3.7", "3.8", "3.9", "3.10", "3.11"]
 
 env:
 AUSTIN_TESTS_PYTHON_VERSIONS: ${{ matrix.python-version }}
 
 name: Tests on Linux with Python ${{ matrix.python-version }}
 steps:
 - uses: actions/checkout@v2

 - name: Install build dependencies
 run: |
 sudo apt-get -y install libunwind-dev binutils-dev libiberty-dev

 - name: Install test dependencies
 run: |
 sudo apt-get -y install \
 valgrind \
 gdb

 - name: Install Python
 uses: actions/setup-python@v4
 with:
 python-version: ${{ matrix.python-version }}

 - name: Install Python 3.10
 uses: actions/setup-python@v4
 with:
 python-version: "3.10"

 - name: Compile Austin
 run: |
 autoreconf --install
 ./configure --enable-debug-symbols true
 make

 - name: Run tests
 run: |
 ulimit -c unlimited
 python3.10 -m venv .venv
 source .venv/bin/activate
 pip install --upgrade pip
 pip install -r test/requirements.txt
 sudo -E .venv/bin/pytest --pastebin=failed --no-flaky-report -sr a -n auto
 .venv/bin/pytest --pastebin=failed --no-flaky-report -sr a -n auto
 deactivate

If Python is installed from, e.g., deadsnakes/ppa, then the tests pass without SIGSEGV. I can also get the tests to pass locally.

This is an example of a happy workflow that pulls Python from the PPA: https://github.com/P403n1x87/austin/runs/7046194671?check_suite_focus=true

This is a run with the action: https://github.com/P403n1x87/austin/runs/7045119660?check_suite_focus=true

This was discovered with this PR: https://github.com/P403n1x87/austin/pull/120/files

Action version:

v4

Platform:

• Ubuntu
• macOS
• Windows

Runner type:

• Hosted
• Self-hosted

Tools version:

Repro steps:

See description above

Expected behavior:

No SIGSEGV, like with the Pythons from the PPA.

Actual behavior:

SIGSEGV if ctypes is used

[dsame]

Hello @P403n1x87,

Thanks for your input, but the data you've provided is not enough to discovery the root cause of the issue.

 if result.returncode == -11:
 binary_name = Path(module).stem.replace("test_", "")
> raise SegmentationFault(bt((SRC / binary_name).with_suffix(".so")))
E test.cunit.conftest.SegmentationFault: "/var/crash/_opt_hostedtoolcache_Python_3.10.5_x64_bin_python3.10.0.crash" is not a core dump: file format not recognized
E No stack.

says there's no crash dump and there's nothing to tracebak - i advise you to add the step with `ls -l /var/crash/' to check the exact path of core dump.

Alternatively can you please provide the exact python code that segfaults.

I tried to create a simple repro with typical ctypes use case, but it works as expected.

Ideally it would be good if you changed my minimal repro in order to get the same SEGFAULT error. In this case we will resolve the issue as soon as possible.

[P403n1x87]

@dsame here you can see a run with a backtrace https://github.com/P403n1x87/austin/runs/7045867473?check_suite_focus=true

This seems to point to this line of Python test code

https://github.com/P403n1x87/austin/blob/d1e56c1ec4aa3322d7b8824513d2e01db5c792f9/test/cunit/test_cache.py#L14

which performs a call to

https://github.com/P403n1x87/austin/blob/d1e56c1ec4aa3322d7b8824513d2e01db5c792f9/src/cache.c#L46-L54

To run this locally, you could clone the devel branch of https://github.com/P403n1x87/austin/, create a Python 3.10 virtual environment, activate it, install the test dependencies in test/requirements.txt and then run

pytest test/cunit

This single-line command should do the trick once inside the cloned austin folder

python3.10 -m venv /tmp/austin-venv && source /tmp/austin-venv/bin/activate && pip install -r test/requirements.txt && pytest test/cunit

Note that this requires gcc to be available from PATH

I tried to create a simple repro with typical ctypes use case, but it works as expected.

I would try calling libc.free on a memory allocation in your repro and see what happens.

[dsame]

@P403n1x87 thanks a lot for the backtrace, now i see a SEGFAULT is caused by an instruction of from /lib/x86_64-linux-gnu/libffi.so.7

It is known problem: toolcache python is built against libffi.so.6 which does not exist in ubuntu-20.04

While we are providing the solution, please try to add the step installing libffi.so.6 before running tests as workaround

 - run: |
 curl -LO http://archive.ubuntu.com/ubuntu/pool/main/libf/libffi/libffi6_3.2.1-8_amd64.deb
 sudo dpkg -i libffi6_3.2.1-8_amd64.deb
 sudo ln -sf libffi.so.6.0.4 /usr/lib/x86_64-linux-gnu/libffi.so

[P403n1x87]

@dsame thanks for looking into this. I will give the workaround a try and let you know how it goes!

[dsame]

Hello @P403n1x87,
How it is going with the checking the workaround?
Be aware i am going to close the issue in the next 24 hours due to inactivity, but you still will be able to reopen this issue or create new one in case of a problem still exists.

[P403n1x87]

@dsame sorry I had moved to other things in the meantime. Just opened a draft PR to test the workaround, but it doesn't seem to solve the issue unfortunately 🙁

https://github.com/P403n1x87/austin/runs/7336236967?check_suite_focus=true

[dsame]

Hello @P403n1x87 ,

now the error messages changed
from "E test.cunit.conftest.SegmentationFault"
to "E OSError: [Errno 39] Directory not empty: '_usr_lib_cnf-update-db.0'"

Although i still see the same point of the error origin, can you please double check the new hint?

[P403n1x87]

@dsame

A Segmentation Fault still causes this. Looking at the full traceback from the first failing test you can see the attempt to raise SegmentationFault to report the SIGSEGV and get the backtrace with the bt helper. This helper calls rmdir which now fails because the directory it is trying to delete is probably not empty. I'll try to resolve this so we can get back the backtrace information, but this is definitely still a SIGSEGV.