Nginx 1.5.10 + ModSecurity 2.7.7: Process killed by segmentation fault at 10

[ThraaxSession]

The process is killed by Signal 11. The software tries to allocate unassigned memory area.
kernel: [6062665.616763] nginx[8497]: segfault at 10 ip 00007fab654c2581 sp 00007fff401c4db0 error 4 in libapr-1.so.0.4.6[7fab654aa000+30000]

[ThraaxSession]

Hey guys! :)
I'd like to know if something new about the issue. Any feedback please.

Regards
Gino

[zimmerle]

Hi @GinoHereIam we are working on a refactoring of our nginx versions. Beta code is available at:

https://github.com/SpiderLabs/ModSecurity/tree/nginx_refactoring

Do you mind to test it?

[ThraaxSession]

I'll give feedback in the near future.
Hopefully it works.

[DOSarrest]

We are also seeing this problem. We have tested the nginx_refactoring branch and still observe the same problem...
kernel: nginx[7045]: segfault at 10 ip 00007f6aebb9f1c1 sp 00007fffe1b0f130 error 4 in libapr-1.so.0.4.8[7f6aebb86000+2f000]
This also causes problems with nginx...
[alert] 21527#0: worker process 5999 exited on signal 11
Making modsecurity for nginx VERY unstable. Our current testing is with nginx/1.4.7

[zimmerle]

Hi @DOSarrest, it would be great if you can share more details about this segfault.

Can you send me GDB output? while running the nginx_refactoring branch?

There are instructions on how to do it here:
https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual

This GDB output will be very valuable to fix the problem.

Br.,
F.

[DOSarrest]

Hello @zimmerle, Yes, we are willing to do what ever we can to help resolve this.

We located the information on how to do it here...

https://github.com/SpiderLabs/ModSecurity/wiki/Debugging-ModSecurity

and we are going to go try and get you the information right now and will reply shortly.

[zimmerle]

Ops, Sorry I have placed the wrong link.

Thanks this GDB information will be very helpful.

[DOSarrest]

Np problem, Glad to help. If you don't mind, could you also check into #713 we have not had a response on that yet. It's related to nginx refactorying.

[DOSarrest]

Here is what I am getting currently...

# gdb nginx
GNU gdb (Gentoo 7.6.2 p1) 7.6.2
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>...
Reading symbols from /usr/sbin/nginx...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/sbin/nginx 
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/nginx.conf:56

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff703b1c1 in apr_global_mutex_lock () from /usr/lib64/libapr-1.so.0
(gdb) bt full
#0  0x00007ffff703b1c1 in apr_global_mutex_lock () from /usr/lib64/libapr-1.so.0
No symbol table info available.
#1  0x000000000056403e in ?? ()
No symbol table info available.
#2  0x0000000000562a0d in ?? ()
No symbol table info available.
#3  0x00000000005607a2 in ?? ()
No symbol table info available.
#4  0x000000000055b17a in modsecFinishRequest ()
No symbol table info available.
#5  0x0000000000532839 in ?? ()
No symbol table info available.
#6  0x0000000000423d09 in ngx_destroy_pool ()
No symbol table info available.
#7  0x0000000000476b31 in ngx_http_free_request ()
No symbol table info available.
#8  0x00000000004756a3 in ?? ()
No symbol table info available.
#9  0x0000000000474bc7 in ?? ()
No symbol table info available.
#10 0x000000000047482f in ngx_http_finalize_request ()
No symbol table info available.
#11 0x00000000004922e8 in ?? ()
No symbol table info available.
#12 0x000000000049153a in ?? ()
No symbol table info available.
#13 0x00000000004912a4 in ?? ()
No symbol table info available.
#14 0x000000000048fc63 in ?? ()
No symbol table info available.
#15 0x000000000048e133 in ?? ()
No symbol table info available.
#16 0x000000000048c9fb in ?? ()
No symbol table info available.
#17 0x0000000000455e9e in ?? ()
No symbol table info available.
#18 0x000000000044653c in ngx_process_events_and_timers ()
No symbol table info available.
#19 0x00000000004529c1 in ngx_single_process_cycle ()
No symbol table info available.
#20 0x0000000000420b20 in main ()
No symbol table info available.

[DOSarrest]

I'm trying to figure out how to get debugging symbols working

[zimmerle]

Did you had a change to re-compile, both, ModSec and nginx with its respective "debug" enablers?

If so, can you try a "make clean" (or start from a fresh directory) before: "./configure" ... "make install"

[DOSarrest]

I am using gentoo package management to handle the compilation process, and the main problem was that gentoo was stripping the binaries before installing them. I had modified the build process to stop stripping the binaries and have a better dump for you...

# gdb nginx
GNU gdb (Gentoo 7.6.2 p1) 7.6.2
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>...
Reading symbols from /usr/sbin/nginx...done.
(gdb) run
Starting program: /usr/sbin/nginx 
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
nginx: [warn] duplicate MIME type "text/html" in /etc/nginx/nginx.conf:58

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff703b1c1 in apr_global_mutex_lock () from /usr/lib64/libapr-1.so.0
(gdb) bt full
#0  0x00007ffff703b1c1 in apr_global_mutex_lock () from /usr/lib64/libapr-1.so.0
No symbol table info available.
#1  0x00000000005688cf in sec_audit_logger (msr=0x98ba8d0) at ../apache2/msc_logging.c:587
        arr = 0x0
        te = 0x0
        tarr_pattern = 0x0
        telts_pattern = 0x0
        str1 = 0x0
        str2 = 0x0
        text = 0x0
        rule = 0x0
        next_rule = 0x0
        nbytes = 160309048
        nbytes_written = 6053746
        md5hash = "\000\000\000\000\000\000\000\000\240\"\216\t\000\000\000"
        was_limited = 0
        present = 0
        wrote_response_body = 0
        entry_filename = 0x7fffffffcc00 "\020\004B"
        entry_basename = 0x7fffffffe250 "\001"
        rc = 0
        i = 0
        limit = 558
        k = 32767
        sanitized_partial = -173670503
        j = 32767
        buf = 0x0
        pat = 0x0
        mparm = 0x0
        arg_min = 32767
        arg_max = -170072608
        sanitize_matched = 32767
#2  0x0000000000565de1 in modsecurity_process_phase_logging (msr=0x98ba8d0) at ../apache2/modsecurity.c:701
        time_before = 1400194077820059
        time_after = 1400194077820082
#3  0x000000000056604c in modsecurity_process_phase (msr=0x98ba8d0, phase=5) at ../apache2/modsecurity.c:807
No locals.
#4  0x0000000000563be2 in hook_log_transaction (r=0x98b9b30) at ../apache2/mod_security2.c:1239
        arr = 0x98ba420
        origr = 0x98b9b30
        msr = 0x98ba8d0
#5  0x000000000055b8e5 in modsecFinishRequest (r=0x98b9b30) at api.c:656
No locals.
#6  0x00000000005327d9 in ngx_http_modsecurity_cleanup (data=0x93e1820) at /var/tmp/portage/www-servers/nginx-1.4.7/work/ModSecurity-nginx_refactoring/nginx/modsecurity/ngx_http_modsecurity.c:1535
        ctx = 0x93e1820
#7  0x0000000000423ca9 in ngx_destroy_pool (pool=0x93dfbf0) at src/core/ngx_palloc.c:55
        p = 0x2
        n = 0x8914c8
        l = 0x9914528
        c = 0x93e0bd8
#8  0x0000000000476ad1 in ngx_http_free_request (r=0x93dfc40, rc=0) at src/http/ngx_http_request.c:3391
        log = 0x98303d0
        pool = 0x93dfbf0
        linger = {l_onoff = 0, l_linger = 0}
        cln = 0x0
        ctx = 0x98304a0
        clcf = 0x0
#9  0x000000000047683c in ngx_http_close_request (r=0x93dfc40, rc=0) at src/http/ngx_http_request.c:3307
        c = 0x7ff9fef17390
#10 0x00000000004763a7 in ngx_http_lingering_close_handler (rev=0x7ff9fe922aa0) at src/http/ngx_http_request.c:3167
        n = 0
        timer = 30
        c = 0x7ff9fef17390
        r = 0x93dfc40
        clcf = 0x85e4b8 <cached_time+696>
        buffer = '\000' <repeats 64 times>, "\200\317\377\377\377\177", '\000' <repeats 14 times>, "\002\000\000\000\022\311\213\t", '\000' <repeats 396 times>...
#11 0x0000000000455e3e in ngx_epoll_process_events (cycle=0x88fac0, timer=425, flags=1) at src/event/modules/ngx_epoll_module.c:683
        events = 1
        revents = 21
        instance = 0
        i = 0
        level = 9021472
        err = 0
        rev = 0x7ff9fe922aa0
        wev = 0x7ff9fe32eaa0
        queue = 0x860340 <ngx_event_timer_sentinel>
        c = 0x7ff9fef17390
#12 0x00000000004464dc in ngx_process_events_and_timers (cycle=0x88fac0) at src/event/ngx_event.c:249
        flags = 1
        timer = 425
        delta = 1400194077819
#13 0x0000000000452961 in ngx_single_process_cycle (cycle=0x88fac0) at src/os/unix/ngx_process_cycle.c:315
        i = 70
#14 0x0000000000420ac0 in main (argc=1, argv=0x7fffffffe258) at src/core/nginx.c:409
        i = 70
        log = 0x85e120 <ngx_log>
        cycle = 0x88fac0
        init_cycle = {conf_ctx = 0x0, pool = 0x88eb10, log = 0x85e120 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, handler = 0x0, data = 0x0, action = 0x0, priority = 0, facility = 0, syslog_on = 0, syslog_set = 0}, files = 0x0, 
          free_connections = 0x0, free_connection_n = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, open_files = {
            last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0}, connection_n = 0, files_n = 0, connections = 0x0, 
          read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 21, data = 0x5bcc9a "/etc/nginx/nginx.conf"}, conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 11, data = 0x5bcc9a "/etc/nginx/nginx.conf"}, prefix = {len = 5, 
            data = 0x5bcc94 "/usr/"}, lock_file = {len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
        ccf = 0x890b70
(gdb)