"Database is not open. Use: SecGeoLookupDb directive." after reload

[jptosso]

Describe the bug

I'm using mmdb to block countries from my website and it works fine until I reload my nginx server and I start getting the message "Database is not open. Use: SecGeoLookupDb directive." in the debug log and no country would be blocked anymore.

Logs and dumps

Before reload:

[...] [/] [4] (Rule: 20) Executing operator "GeoLookup against REMOTE_ADDR.
[...] [/] [9] Target value: "111.111.111.111" (Variable: REMOTE_ADDR)
[...] [/] [9] Matched vars updated.
[...] [/] [9] Saving msg: Block China IP address
[...] [/] [4] Rule returned 1.
[...] [/] [4] Executing chained rule.
[...] [/] [4] (Rule: 0) Executing operator "Within" with param "CN CL" Was: "" against GEO:COUNTRY_CODE.
[...] [/] [9] Target value: "CN" (Variable: GEO:COUNTRY_CODE)
[...] [/] [9] Matched vars updated.
[...] [/] [4] Rule returned 1.

After reload:

[...] [...] [4] (Rule: 20) Executing operator "GeoLookup against REMOTE_ADDR.
[...] [...] [9] Target value: "111.111.111.111" (Variable: REMOTE_ADDR)
[...] [...] [4] Database is not open. Use: SecGeoLookupDb directive.
[...] [...] [4] Rule returned 0.

ModSecurity config log:

ModSecurity - v3.0.3-56-g6d5198b1 for Linux
Mandatory dependencies

• libInjection ....v3.0.3-56-g6d5198b1
• SecLang tests ....6d5198b

Optional dependencies

• GeoIP/MaxMind ....found
  • (MaxMind) v1.3.2
    -lmaxminddb, -DWITH_MAXMIND -I/usr/local/include
  • (GeoIP) v1.6.12
    -lGeoIP, -I/usr/include/
• LibCURL ....found v7.58.0
  -lcurl, -DWITH_CURL_SSLVERSION_TLSv1_2 -DWITH_CURL
• YAJL ....found v2.1.0
  -lyajl, -DWITH_YAJL -I/usr/include/yajl
• LMDB ....not found
• LibXML2 ....found v2.9.4
  -lxml2, -I/usr/include/libxml2 -DWITH_LIBXML2
• SSDEEP ....not found
• LUA ....found v501
  -llua5.1 -L/usr/lib/x86_64-linux-gnu/, -DWITH_LUA -DWITH_LUA_5_1 -I/usr/include/lua5.1

Other Options

• Test Utilities ....enabled
• SecDebugLog ....enabled
• afl fuzzer ....disabled
• library examples ....enabled
• Building parser ....disabled
• Treating pm operations as critical section ....disabled

To Reproduce

Use the following configuration with your own country code and restart nginx.

SecGeoLookupDb /mnt/nginx/defaults/geoip/geoip.mmdb

SecAction
"id:19,
phase:1,
nolog,
pass,
t:none,
setvar:'tx.high_risk_country_codes=CN CL'"
SecRule REMOTE_ADDR "@geoLookup" "chain,id:20,drop,msg:'Block China IP address',phase:2"
SecRule GEO:COUNTRY_CODE "@Within %{tx.high_risk_country_codes}"

Test with your own country code and you should get a 403 forbidden error code, then remove your country from the collection, reload and try again, you should receive 200. Add your country code again to the collection and you should get a 200 status and the previous reported message in the logs as the geoip database was unloaded.
Or you can just create a disruptive rule with geoip and after testing it works, you can check if it works after a reload.

Expected behavior

Allow me to reload the geoip database without restarting the server.

Server:

• ModSecurity version (and connector): ModSecurity master (6d5198b), Connector master (d7101e1)
• WebServer: 1.15.19
• OS: 4.15.0-1021-aws Fixed: nginx INCS config on fedora #21-Ubuntu x86_64
• libmaxminddb: 1.3.2 (from source)
• Nginx flags: ./configure
  --with-http_ssl_module
  --add-module=/tmp/ModSecurity-nginx/
  --with-http_ssl_module
  --with-http_sub_module
  --with-http_realip_module
  --with-http_sub_module
  --with-http_gunzip_module
  --with-http_gzip_static_module

More info

It seems like Utils::GeoLookup::getInstance().cleanUp(); is being called and Utils::GeoLookup::setDataBase is not being called again, I'm trying to find out the problem over there.

Small workaround

You can "reload" nginx by renewing the process:

create new master and workers

pid=$(cat /usr/local/nginx/logs/nginx.pid)
sudo kill -USR2 $pid

gracefully kill the old workers

sudo kill -WINCH $pid

gracefully kill the old master

sudo kill -QUIT $pid

That would reload the whole engine and nginx without any downtime but using many resources.
Benchmarks for 1000 reloads and winchs in a 8 core 16gb memory server with just 100 rules:

#https://gist.github.com/jptosso/aa72ff932d3fad39614923bccb2d175a
ubuntu@server:~$ sudo bash /etc/test.sh
winch: 17 seconds
reload: 35 seconds

[victorhora]

I believe the issue is probably due to the fact that the call to GeoLookup::setDataBase is made only at the parser here: https://github.com/SpiderLabs/ModSecurity/blob/v3/master/src/parser/seclang-parser.cc#L2706-L2723

The leads me to believe that on a reload, when GeoLookup::evaluate eventually is called, GeoLookup::lookup is called directly without any prior call to GeoLookup::setDataBase. We might be missing an init() at the geo_lookup.cc operator.

That being said, I could only reproduce this bug when I had BOTH MaxMind and the legacy GeoIP libraries/db installed.
So I believe that a valid temporary workaround is having only one of the libraries.

[jptosso]

Tested your workaround and it does work:

ModSecurity - v3.0.3-56-g6d5198b1 for Linux

Mandatory dependencies

• libInjection ....v3.0.3-56-g6d5198b1
• SecLang tests ....6d5198b

Optional dependencies

• GeoIP/MaxMind ....found
  • (MaxMind) v1.3.2
    -lmaxminddb, -DWITH_MAXMIND -I/usr/local/include
• LibCURL ....found v7.58.0
  -lcurl, -DWITH_CURL_SSLVERSION_TLSv1_2 -DWITH_CURL
• YAJL ....found v2.1.0
  -lyajl, -DWITH_YAJL -I/usr/include/yajl
• LMDB ....not found
• LibXML2 ....found v2.9.4
  -lxml2, -I/usr/include/libxml2 -DWITH_LIBXML2
• SSDEEP ....not found
• LUA ....found v502
  -lluajit-5.1 -L/usr/lib/x86_64-linux-gnu/, -DWITH_LUA -DWITH_LUA_5_2 -I/usr/include/lua5.2

[victorhora]

Thanks for the feedback @asdfuken. Good to know that it works as intended this way :)

Regardless, we'll try to get it fixed prior to 3.0.4 release. Thanks for reporting!

[welljsjs]

This is still an issue for us, we're on e9dce44.

Sorry what exactly is the workaround @jptosso ?

[zimmerle]

Fix under QA here - https://travis-ci.org/github/SpiderLabs/ModSecurity/builds/743954613?utm_source=github_status&utm_medium=notification