Skip to content

fix: fix segfault / memory issue with interpreter in timing mode - #2446

Closed
copybara-service[bot] wants to merge 0 commit into
mainfrom
test_838898360
Closed

fix: fix segfault / memory issue with interpreter in timing mode#2446
copybara-service[bot] wants to merge 0 commit into
mainfrom
test_838898360

Conversation

@copybara-service

Copy link
Copy Markdown
Contributor

fix: fix segfault / memory issue with interpreter in timing mode

If you run

bazel test -c dbg lib/Target/OpenFhePke:InterpreterTest --test_output=all --//openfhe_enable_timing=1 --config=asan

You would notice an issue in the interpreter

==9339==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7b866a11da60 at pc 0x7f8796e27fda bp 0x7ffc01f32950 sp 0x7ffc01f32948
READ of size 8 at 0x7b866a11da60 thread T0
#0 0x7f8796e27fd9 in __root third_party/crosstool/v18/stable/src/libcxx/include/__tree:865:54
#1 0x7f8796e27fd9 in __find_equal third_party/crosstool/v18/stable/src/libcxx/include/__tree:1842:25
#2 0x7f8796e27fd9 in std::__u::pair<std::__u::__tree_iterator<std::__u::__value_type<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator>, mlir::heir::openfhe::Interpreter::TimingData>, std::__u::__tree_node<std::__u::__value_type<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator>, mlir::heir::openfhe::Interpreter::TimingData>, void*>, long>, bool> std::__u::__tree<std::__u::__value_type<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator>, mlir::heir::openfhe::Interpreter::TimingData>, std::__u::__map_value_compare<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator>, std::__u::pair<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator> const, mlir::heir::openfhe::Interpreter::TimingData>, std::__u::less<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator>>>, std::__u::allocator<std::__u::pair<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator> const, mlir::heir::openfhe::Interpreter::TimingData>>>::__emplace_unique<std::__u::piecewise_construct_t const&, std::__u::tuple<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator>&&>, std::__u::tuple<>>(std::__u::piecewise_construct_t const&, std::__u::tuple<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator>&&>&&, std::__u::tuple<>&&)::'lambda'(std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator> const&, std::__u::piecewise_construct_t const&, std::__u::tuple<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator>&&>&&, std::__u::tuple<>&&)::operator()(std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator> const&, std::__u::piecewise_construct_t const&, std::__u::tuple<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator>&&>&&, std::__u::tuple<>&&) const third_party/crosstool/v18/stable/src/libcxx/include/__tree:956:38
#3 0x7f8796dd6e09 in __try_key_extraction_impl<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator >, std::__u::pair<std::__u::__tree_iterator<std::__u::__value_type<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator >, mlir::heir::openfhe::Interpreter::TimingData>, std::__u::__tree_node<std::__u::__value_type<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator >, mlir::heir::openfhe::Interpreter::TimingData>, void > , long>, bool>, (lambda at third_party/crosstool/v18/stable/src/libcxx/include/__tree:955:9), (lambda at third_party/crosstool/v18/stable/src/libcxx/include/__tree:967:9), const std::__u::piecewise_construct_t &, std::__u::tuple<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator > &&>, std::__u::tuple<>, 0> third_party/crosstool/v18/stable/src/libcxx/include/__utility/try_key_extraction.h:91:10
#4 0x7f8796dd6e09 in __try_key_extraction<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator >, (lambda at third_party/crosstool/v18/stable/src/libcxx/include/__tree:955:9), (lambda at third_party/crosstool/v18/stable/src/libcxx/include/__tree:967:9), const std::__u::piecewise_construct_t &, std::__u::tuple<std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator > &&>, std::__u::tuple<> > third_party/crosstool/v18/stable/src/libcxx/include/__utility/try_key_extraction.h:108:10
#5 0x7f8796dd6e09 in __emplace_unique third_party/crosstool/v18/stable/src/libcxx/include/__tree:954:12
#6 0x7f8796dd6e09 in operator[] third_party/crosstool/v18/stable/src/libcxx/include/map:1435:8
#7 0x7f8796dd6e09 in mlir::heir::openfhe::Interpreter::visit(mlir::heir::openfhe::AddOp) third_party/heir/lib/Target/OpenFhePke/Interpreter.cpp:1194:3
#8 0x7f8796dc0d00 in operator()mlir::heir::openfhe::AddOp third_party/heir/lib/Target/OpenFhePke/Interpreter.cpp:196:54
#9 0x7f8796dc0d00 in Case<mlir::heir::openfhe::AddOp, (lambda at third_party/heir/lib/Target/OpenFhePke/Interpreter.cpp:196:39) &> third_party/llvm/llvm-project/llvm/include/llvm/ADT/TypeSwitch.h:172:7
#10 0x7f8796dc0d00 in mlir::heir::openfhe::Interpreter::visit(mlir::Operation
) third_party/heir/lib/Target/OpenFhePke/Interpreter.cpp:180:8
#11 0x7f8796dc02be in mlir::heir::openfhe::Interpreter::interpret(std::__u::basic_string<char, std::__u::char_traits, std::__u::allocator> const&, llvm::ArrayRefmlir::heir::openfhe::TypedCppValue) third_party/heir/lib/Target/OpenFhePke/Interpreter.cpp:171:7
#12 0x7f87974ca92c in mlir::heir::openfhe::InterpreterTest_TestOpenfheAdd_Test::TestBody() third_party/heir/lib/Target/OpenFhePke/InterpreterTest.cpp:546:52
#13 0x7f86e5b6e72d in HandleExceptionsInMethodIfSupported third_party/googletest/googletest/src/gtest.cc
#14 0x7f86e5b6e72d in testing::Test::Run() third_party/googletest/googletest/src/gtest.cc:2755:5
#15 0x7f86e5b6ffc5 in testing::TestInfo::Run() third_party/googletest/googletest/src/gtest.cc:2901:11
#16 0x7f86e5b71b70 in testing::TestSuite::Run() third_party/googletest/googletest/src/gtest.cc:3079:30
#17 0x7f86e5b923b0 in testing::internal::UnitTestImpl::RunAllTests() third_party/googletest/googletest/src/gtest.cc:6074:44
#18 0x7f86e5b9fcb7 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl
, bool (testing::internal::UnitTestImpl::
)(), char const*) third_party/googletest/googletest/src/gtest.cc
#19 0x7f86e5b91505 in testing::UnitTest::Run() third_party/googletest/googletest/src/gtest.cc:5614:10
#20 0x557028e3697c in RUN_ALL_TESTS third_party/googletest/googletest/include/gtest/gtest.h:2352:73
#21 0x557028e3697c in main testing/base/internal/gunit_main.cc:85:10
#22 0x7f878aca2351 in __libc_start_main (/usr/grte/v5/lib64/libc.so.6+0x61351) (BuildId: ca23ec6d935352118622ce674a8bb52d)
#23 0x557028d55469 in _start /usr/grte/v5/debug-src/src/csu/../sysdeps/x86_64/start.S:120

Address 0x7b866a11da60 is located in stack of thread T0 at offset 608 in frame
#0 0x7f87974c9f6f in mlir::heir::openfhe::InterpreterTest_TestOpenfheAdd_Test::TestBody() third_party/heir/lib/Target/OpenFhePke/InterpreterTest.cpp:517

This frame has 33 object(s):
[32, 33) 'ref.tmp'
[48, 96) 'setup' (line 518)
[128, 152) 'vec1' (line 521)
[192, 216) 'vec2' (line 522)
[256, 272) 'ref.tmp' (line 527)
[288, 304) 'ref.tmp' (line 528)
[320, 328) 'context' (line 531)
[352, 376) 'mlirStr' (line 533)
[416, 440) 'ref.tmp' (line 533)
[480, 488) 'module' (line 541)
[512, 600) 'interpreter' (line 543) <== Memory access at offset 608 overflows this variable
[640, 664) 'inputs' (line 544)
[704, 800) 'ref.tmp' (line 544)
[832, 856) 'results' (line 546)
[896, 920) 'ref.tmp' (line 546)
[960, 976) 'gtest_ar' (line 548)
[992, 1000) 'ref.tmp' (line 548)
[1024, 1028) 'ref.tmp' (line 548)
[1040, 1048) 'ref.tmp' (line 548)
[1072, 1080) 'ref.tmp' (line 548)
[1104, 1120) 'resultPt' (line 552)
[1136, 1152) 'ref.tmp' (line 553)
[1168, 1184) 'tmp' (line 553)
[1200, 1224) 'resultVec' (line 556)
[1264, 1280) 'gtest_ar' (line 557)
[1296, 1304) 'ref.tmp' (line 557)
[1328, 1336) 'ref.tmp' (line 557)
[1360, 1368) 'ref.tmp' (line 557)
[1392, 1400) 'ref.tmp' (line 557)
[1424, 1440) 'gtest_ar' (line 559)
[1456, 1464) 'ref.tmp' (line 559)
[1488, 1496) 'ref.tmp' (line 559)
[1520, 1528) 'ref.tmp' (line 559)
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions are supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow third_party/crosstool/v18/stable/src/libcxx/include/__tree:865:54 in __root (//third_party/heir/lib/Target/OpenFhePke:InterpreterTest)
Shadow bytes around the buggy address:
0x7b866a11d780: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
0x7b866a11d800: f1 f1 f1 f1 f8 f2 00 00 00 00 00 00 f2 f2 f2 f2
0x7b866a11d880: 00 00 00 f2 f2 f2 f2 f2 00 00 00 f2 f2 f2 f2 f2
0x7b866a11d900: f8 f8 f2 f2 f8 f8 f2 f2 00 f2 f2 f2 00 00 00 f2
0x7b866a11d980: f2 f2 f2 f2 f8 f8 f8 f2 f2 f2 f2 f2 00 f2 f2 f2
=>0x7b866a11da00: 00 00 00 00 00 00 00 00 00 00 00 f2[f2]f2 f2 f2
0x7b866a11da80: 00 00 00 f2 f2 f2 f2 f2 f8 f8 f8 f8 f8 f8 f8 f8
0x7b866a11db00: f8 f8 f8 f8 f2 f2 f2 f2 00 00 00 f2 f2 f2 f2 f2
0x7b866a11db80: 00 00 00 f2 f2 f2 f2 f2 f8 f8 f2 f2 f8 f2 f2 f2
0x7b866a11dc00: f8 f2 f8 f2 f2 f2 f8 f2 f2 f2 f8 f8 f2 f2 f8 f8
0x7b866a11dc80: f2 f2 f8 f8 f2 f2 f8 f8 f8 f2 f2 f2 f2 f2 f8 f8
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==9339==ABORTING
-- 2025-12-01 12:56:55 PST Forge runner: Test failed wi

Without asan, we just get a segfault at the line timingResults[op_name]. I don't know if there's a better fix since all I can guess is there is some memory corruption when instantiating a map in the define...

@copybara-service
copybara-serviceBotforce-pushed the test_838898360 branch 2 times, most recently from 93262ac to 5341d78CompareJanuary 5, 2026 17:02
@copybara-service
copybara-serviceBot deleted the test_838898360 branch January 5, 2026 17:42
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants