Get the test suite passing with clang Memory Sanitizer enabled

[gpshead]

BPO	35214
Nosy	@Yhg1s, @gpshead, @benjaminp, @alex, @serhiy-storchaka, @izbyshev, @pablogsal, @miss-islington, @epicfaace
PRs	bpo-35214: Initial clang MemorySanitizer support #10479 [3.7] bpo-35214: Initial clang MemorySanitizer support (GH-10479) #10492 [3.6] bpo-35214: Initial clang MemorySanitizer support (GH-10479) #10493 bpo-35214: Disable getc_unlocked() with MemorySanitizer. #10499 [3.7] bpo-35214: Disable getc_unlocked() with MemorySanitizer. (GH-10499) #10500 [3.6] bpo-35214: Disable getc_unlocked() with MemorySanitizer. (GH-10499) #10501 bpo-35214: Add _Py_ prefix to MEMORY_SANITIZER define #10503 [3.7] bpo-35214: Add _Py_ prefix to MEMORY_SANITIZER def. (GH-10503) #10504 [3.6] bpo-35214: Add _Py_ prefix to MEMORY_SANITIZER def. (GH-10503) #10505 bpo-35214: Fix OOB memory access in unicode escape parser #10506 [3.7] bpo-35214: Fix OOB memory access in unicode escape parser (GH-10506) #10522 [3.6] bpo-35214: Fix OOB memory access in unicode escape parser (GH-10506) #10523 [2.7] bpo-35214: Fix OOB memory access in unicode escape parser (GH-10506) #10538 bpo-35214: MSan workarounds for socket, time, and test_faulthandler. #11375 bpo-35214: MSan workarounds for socket, time, and test_faulthandler. #11375 bpo-35214: MSan workarounds for socket, time, and test_faulthandler. #11375 [3.7] bpo-35214: MSan workarounds for socket, time, and test_faulthandler. (GH-11375) #11378 [3.7] bpo-35214: MSan workarounds for socket, time, and test_faulthandler. (GH-11375) #11378 [3.7] bpo-35214: MSan workarounds for socket, time, and test_faulthandler. (GH-11375) #11378 bpo-35214: Skip test_io tests that'd cause a huge malloc under msan #11385 bpo-35214: Skip test_io tests that'd cause a huge malloc under msan #11385 bpo-35214: Skip test_io tests that'd cause a huge malloc under msan #11385 [3.7] bpo-35214: Skip test_io tests that'd cause a huge malloc under msan (GH-11385) #11388 [3.7] bpo-35214: Skip test_io tests that'd cause a huge malloc under msan (GH-11385) #11388 [3.7] bpo-35214: Skip test_io tests that'd cause a huge malloc under msan (GH-11385) #11388 bpo-35214: Annotate posix calls for clang MSan. #11389 bpo-35214: Annotate posix calls for clang MSan. #11389 bpo-35214: Annotate posix calls for clang MSan. #11389 [3.7] bpo-35214: Annotate posix calls for clang MSan. (GH-11389) #11391 [3.7] bpo-35214: Annotate posix calls for clang MSan. (GH-11389) #11391 [3.7] bpo-35214: Annotate posix calls for clang MSan. (GH-11389) #11391

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = None
created_at = <Date 2018-11-12.07:26:34.927>
labels = ['type-security', 'interpreter-core', '3.7', '3.8', 'extension-modules', 'build', 'tests']
title = 'Get the test suite passing with clang Memory Sanitizer enabled'
updated_at = <Date 2019-08-14.20:02:32.630>
user = 'https://github.com/gpshead'

bugs.python.org fields:

activity = <Date 2019-08-14.20:02:32.630>
actor = 'gregory.p.smith'
assignee = 'gregory.p.smith'
closed = False
closed_date = None
closer = None
components = ['Build', 'Extension Modules', 'Interpreter Core', 'Tests']
creation = <Date 2018-11-12.07:26:34.927>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 35214
keywords = ['patch']
message_count = 28.0
messages = ['329723', '329750', '329759', '329777', '329798', '329803', '329804', '329805', '329806', '329807', '329808', '329809', '329810', '329867', '329870', '329875', '329929', '331639', '331642', '332778', '332784', '332796', '332797', '332799', '332801', '332804', '349639', '349742']
nosy_count = 9.0
nosy_names = ['twouters', 'gregory.p.smith', 'benjamin.peterson', 'alex', 'serhiy.storchaka', 'izbyshev', 'pablogsal', 'miss-islington', 'epicfaace']
pr_nums = ['10479', '10492', '10493', '10499', '10500', '10501', '10503', '10504', '10505', '10506', '10522', '10523', '10538', '11375', '11375', '11375', '11378', '11378', '11378', '11385', '11385', '11385', '11388', '11388', '11388', '11389', '11389', '11389', '11391', '11391', '11391']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue35214'
versions = ['Python 3.7', 'Python 3.8']

[gpshead]

clang's memory sanitizer (-fsanitize=memory) turns up useful problems in code. I'm working on getting a CPython buildbot running it setup but would like our build to be cleaner to start with before I run that.

These are the initial fixes required for most of CPython to pass in an msan build. We've been using these with our interpreters at Google. (PR coming)

[gpshead]

New changeset 1584a00 by Gregory P. Smith in branch 'master':
bpo-35214: Initial clang MemorySanitizer support (GH-10479)
1584a00

[gpshead]

New changeset 5f4d05d by Gregory P. Smith in branch '3.7':
[3.7] bpo-35214: Initial clang MemorySanitizer support (GH-10479) (GH-10492)
5f4d05d

[gpshead]

New changeset 3b5b1c0 by Gregory P. Smith in branch '3.6':
[3.6] bpo-35214: Initial clang MemorySanitizer support (GH-10479) (GH-10493)
3b5b1c0

[benjaminp]

Can we prefix MEMORY_SANITIZER with _Py_?

[pablogsal]

I cannot initialize the interpreter after compiling with --with-memory-sanitizer:

❯ CC=clang ./configure --with-memory-sanitizer && make -j
❯ ./python
Python 3.8.0a0 (heads/master:1584a00815, Nov 13 2018, 03:29:18)
[Clang 7.0.0 (tags/RELEASE_700/final)] on linux
Type "help", "copyright", "credits" or "license" for more information.
==10989==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x5592f18005c3 (/home/pablogsal/cpython/python+0x9a55c3)
#1 0x5592f175c176 (/home/pablogsal/cpython/python+0x901176)
#2 0x5592f17592da (/home/pablogsal/cpython/python+0x8fe2da)
#3 0x5592f1750f82 (/home/pablogsal/cpython/python+0x8f5f82)
#4 0x5592f174a336 (/home/pablogsal/cpython/python+0x8ef336)
#5 0x5592f174c906 (/home/pablogsal/cpython/python+0x8f1906)
#6 0x5592f14ae214 (/home/pablogsal/cpython/python+0x653214)
#7 0x5592f14a6915 (/home/pablogsal/cpython/python+0x64b915)
#8 0x5592f14a293c (/home/pablogsal/cpython/python+0x64793c)
#9 0x5592f0f5ad88 (/home/pablogsal/cpython/python+0xffd88)
#10 0x5592f0f5ce73 (/home/pablogsal/cpython/python+0x101e73)
#11 0x5592f0f4d908 (/home/pablogsal/cpython/python+0xf2908)
#12 0x7fd1a7381222 (/usr/lib/libc.so.6+0x24222)
#13 0x5592f0ed3cdd (/home/pablogsal/cpython/python+0x78cdd)

Uninitialized value was created by a heap allocation
#0 0x5592f0f02a0d (/home/pablogsal/cpython/python+0xa7a0d)
#1 0x7fd1a73cd790 (/usr/lib/libc.so.6+0x70790)

SUMMARY: MemorySanitizer: use-of-uninitialized-value (/home/pablogsal/github/cpython/python+0x9a55c3)
Exiting

❯ clang --version
clang version 7.0.0 (tags/RELEASE_700/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

❯ /lib/libc.so.6
GNU C Library (GNU libc) stable release version 2.28.
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
Compiled by GNU CC version 8.2.1 20180831.
libc ABIs: UNIQUE IFUNC ABSOLUTE
For bug reporting instructions, please see:
<https://bugs.archlinux.org/\>.

I am missing something or is this expected (as there are more PRs to come)?

[alex]

All libraries that are linked against, including libc, need to be compiled with MSAN. MSAN is not for the faint of heart.