malloc error when executing make (debian11, nginx)

[janis-mueller]

Describe the bug
When executing make on debian 11 I get an malloc.c.2379 error

Logs and dumps
bash: malloc.c:2379: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.

To Reproduce

Steps to reproduce the behavior:
./build.sh
git submodule init
git submodule update
./configure
make
make install

Expected behavior
make executes without errors

Server (please complete the following information):

• ModSecurity versionv 3.0.12 [Latest]
• WebServer: nginx 1.18.0-6.1+deb11u3
• OS (and distro): debian 11 5.10.0-28-amd64

[airween]

Hi @janis-mueller,

thanks for reporting.

Just for sure I tried to reproduce this issue on an updated Debian 11 (amd64), but I wasn't able to do that.

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye

I copied the commands you listed above, but finally make finished the build process successfully:

$ make
...
/bin/bash ../libtool  --tag=CXX   --mode=link g++  -g -O2 -L../src/.libs/  -lmodsecurity -lpthread -lm -lstdc++   -L/usr/lib/x86_64-linux-gnu/    -o rules_optimization optimization/rules_optimization-optimization.o -lcurl  -lmaxminddb -lrt   -lxml2  -llua5.1 -lpcre -lpcre -lpcre2-8  -lyajl 
libtool: link: g++ -g -O2 -o .libs/rules_optimization optimization/rules_optimization-optimization.o  -L../src/.libs/ /home/airween/src/ModSecurity_3091/ModSecurity/src/.libs/libmodsecurity.so -lpthread -lm -lstdc++ -L/usr/lib/x86_64-linux-gnu/ -lcurl -lmaxminddb -lrt -lxml2 -llua5.1 -lpcre -lpcre2-8 -lyajl -Wl,-rpath -Wl,/usr/local/modsecurity/lib
make[2]: Leaving directory '/home/airween/src/ModSecurity_3091/ModSecurity/test'
make[1]: Leaving directory '/home/airween/src/ModSecurity_3091/ModSecurity/test'
make[1]: Entering directory '/home/airween/src/ModSecurity_3091/ModSecurity'
make[1]: Nothing to be done for 'all-am'.
make[1]: Leaving directory '/home/airween/src/ModSecurity_3091/ModSecurity'

Based on the error message above (bash: malloc.c:2379: sysmalloc: Assertion '(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.) this looks more like a bash issue than a ModSecurity source code issue to me. Perhaps you don't have enough memory...?

[janis-mueller]

Hi @airween

Thanks for the fast reply. I have 8GB of memory and monitored memory usage during the make process and it never exceeded 1GB but still exits with

bash: malloc.c:2379: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.
make[3]: *** [Makefile:1837: libmodsecurity.la] Aborted
make[2]: *** [Makefile:3528: all-recursive] Error 1
make[1]: *** [Makefile:1241: all] Error 2
make: *** [Makefile:1047: all-recursive] Error 1

any tips on how I can proceed to run make successfully (pretty sure it is no memory issue)

[janis-mueller]

make.txt
I included the make.txt before the error appears if it helps

@airween

[airween]

@janis-mueller - thanks, unfortunately there is nothing that could help to figure out why can't you build the library.

Probably you should try:

sudo apt install script
script -f build.log
make clean
./configure
make

and press a CTRL-D at the end (no matter what's the result).

Please share that file - may be I can find something there.

[janis-mueller]

@airween thanks for the further help. I followed your steps and here is the resultating build.log
build.log

[airween]

Thanks.

Does this happen always after the building of xml.cc file? Or it occurred different places?

I still think that this is some memory issue - found just this opinion, because the line starts with bash: malloc.c.

[airween]

Note, that you can find a precompiled package for Debian 11 here.

[janis-mueller]

Thanks.

Does this happen always after the building of xml.cc file? Or it occurred different places?

I still think that this is some memory issue - found just this opinion, because the line starts with bash: malloc.c.

Not sure where I can find the info regarding xml.cc, but I think it is likely a corrupted RAM is the issue here right?

[janis-mueller]

Note, that you can find a precompiled package for Debian 11 here.

Thanks, I will try that as a quick fix

[airween]

Not sure where I can find the info regarding xml.cc,

Sorry, that was my mistake. After building the source file xml.cc, the linker starts to link:

libtool: compile:  g++ -DHAVE_CONFIG_H ... -c variables/xml.cc  -fPIC -DPIC -o variables/.libs/libmodsecurity_la-xml.o
libtool: compile:  g++ -DHAVE_CONFIG_H ... -c variables/xml.cc -o variables/libmodsecurity_la-xml.o >/dev/null 2>&1
mv -f variables/.deps/libmodsecurity_la-xml.Tpo variables/.deps/libmodsecurity_la-xml.Plo
/bin/bash ../libtool  --tag=CXX   --mode=link g++  -g -O2             -version-info 3:12:0  -o libmodsecurity.la -rpath /usr/local/modsecurity/lib parser/libmodsecurity_la-seclang-parser.lo ... ... variables/libmodsecurity_la-time_year.lo variables/libmodsecurity_la-tx.lo variables/libmodsecurity_la-variable.lo variables/libmodsecurity_la-xml.lo   -lrt      ../others/libinjection.la ../others/libmbedtls.la -lpcre -lpcre     
bash: malloc.c:2379: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.

and then comes the malloc error.

but I think it is likely a corrupted RAM is the issue here right?

Yes, that's my guess. Perhaps a memtest would help to clarify that?

[janis-mueller]

Not sure where I can find the info regarding xml.cc,

Sorry, that was my mistake. After building the source file xml.cc, the linker starts to link:

libtool: compile:  g++ -DHAVE_CONFIG_H ... -c variables/xml.cc  -fPIC -DPIC -o variables/.libs/libmodsecurity_la-xml.o
libtool: compile:  g++ -DHAVE_CONFIG_H ... -c variables/xml.cc -o variables/libmodsecurity_la-xml.o >/dev/null 2>&1
mv -f variables/.deps/libmodsecurity_la-xml.Tpo variables/.deps/libmodsecurity_la-xml.Plo
/bin/bash ../libtool  --tag=CXX   --mode=link g++  -g -O2             -version-info 3:12:0  -o libmodsecurity.la -rpath /usr/local/modsecurity/lib parser/libmodsecurity_la-seclang-parser.lo ... ... variables/libmodsecurity_la-time_year.lo variables/libmodsecurity_la-tx.lo variables/libmodsecurity_la-variable.lo variables/libmodsecurity_la-xml.lo   -lrt      ../others/libinjection.la ../others/libmbedtls.la -lpcre -lpcre     
bash: malloc.c:2379: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.

and then comes the malloc error.

but I think it is likely a corrupted RAM is the issue here right?

Yes, that's my guess. Perhaps a memtest would help to clarify that?

I will definitely do a memtest. Will close this issue for now. Thanks A LOT for your time!