LMDB installed perfectly but its not showing up in working

[nasirbas1]

Hi @zimmerle ,

For persistant collections, we recompiled modsecurity with lmdb flag after installing lmdb,
and the installation worked perfectly fine and lmdb showed up in the installed packages after
running the
./configure --with-lmdb script

However lmdb is not working. Its not showing up in logs, nor are there any errors anywhere. Is there
any way to verify the installation .

Also for SMDB installation , given the fact that every worker process has its own memory which they sync up with smdb at intervals, if any ip is blocked (say I set IP.BLOCKED=1 and deny it after checking this BLOCKED flag) , why does it even after hours altogether(considering that all process are in sync by then) let some of the requests from that blocked IP to go through ??

OS Version : CentOS Linux 7
Nginx Version : 1.16.1
Modsecurity version : 3.03

Mandatory dependencies

• libInjection ....v3.9.2-30-gbf234eb
• SecLang tests ....e6b03e4

Optional dependencies

• GeoIP/MaxMind ....found
  • (GeoIP) v1.5.0
    -lGeoIP , -I/usr/include/
• LibCURL ....found v7.29.0
  -lcurl , -DWITH_CURL
• YAJL ....found v2.0.4
  -lyajl , -DWITH_YAJL
  + LMDB ....found v0.9.22
  -llmdb , -DWITH_LMDB
• LibXML2 ....found v2.9.1
  -lxml2 -lz -lm -ldl, -I/usr/include/libxml2 -DWITH_LIBXML2

[zimmerle]

Hi @nasirbas1,

Please make sure that nignx/connector is using the library that you have compiled with lmdb support. It is very likely that your webserver/connector is using the old library.

[nasirbas1]

@zimmerle for the leakage part will requests from blocked IP's stop leaking through with lmdb because with smdb even after hours of blocking some of the request still go through.

[zimmerle]

@nasirbas1 the manner to block the request (timeout, number of attempts, block time, etc...) should consider that the counter increment is not atomic. There may be a difference from multiprocess sync. Regardless, from my experience, the lmdb sync is quite fast. 1 hour without a sync, sounds to me that you are still using the memory collection still.

Regardless, if your intention is to block a given IP address, there are different manners to achieve such functionality in an atomic manner, such us: rbl

[nasirbas1]

@zimmerle
Regarding your earlier comment
"Please make sure that nignx/connector is using the library that you have compiled with lmdb support. It is very likely that your webserver/connector is using the old library."

I don't thinks this is the case , I removed all the symlinks and folders , basically recompiled everything and its still not working . I think it has taken up the storage mode , because earlier with smdb, atleast it used to set the variable on IP collection but now setvar does happen, but when I try to retrieve the value in next request its blank . There seems to be no place to view exactly whats going wrong ?
Is there any way to debug this?

[zimmerle]

you can enable ModSecurity debug logs. You will see the actual value of each variable upon resolution time. you can also set the numbers of workers to 1 to test if the problem is within the workers sync.

[nasirbas1]

@zimmerle So I have already turned the debug logs at level 9. Its correctly setting the variable , but I think its not able to persist it to the lmdb because on the very next request the value previously set in the variable is blank .... The same works perfectly, when I compile it with smdb . There seems to be no entry in the logs related to the persistance of values to lmdb

[zimmerle]

if you don't have multiple workers nor restart the webserver the value should be persistent regardless of the collection backend. Can you share the rules that you are using?

[nasirbas1]

@zimmerle I do have multiple workers Please find below the rules

Initialize IP Collection using the IP address which comes as True-IP from load-balancer

SecRule REQUEST_HEADERS:True-IP "^\b(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\b" "id:660001,phase:1,t:none,pass,log,capture,setvar:tx.client_ip=%{tx.1}"
SecAction "phase:1,nolog,pass,initcol:ip=%{tx.client_ip}"

logic to block any ip basically sets IP.blocked = 2

SecRule REQUEST_URI "@contains /ip/blacklist"
"id:440001,chain,phase:1,t:none,deny,log,status:200"
SecRule REMOTE_ADDR "@ipMatch 127.0.0.1" "chain,t:none"
SecRule ARGS:ip "^\b(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\b"
"t:none,setvar:ip.allowed=2"

Checks if allowed variable on IP collections for this IP is 2 then drops this request

SecRule IP:ALLOWED "@eq 2" "id:880001,phase:1,t:none,drop,log,logdata:'Dynamic Blacklist'"

[zimmerle]

What is the patch to your lmdb file? is it consistent? can you keep only a single worker for testing?

[nasirbas1]

SecDataDir /etc/nginx/modsec/storage/

Have added all the permissions , so the user has all the permissions to write to this folder

If you meant version its 2.9.1

+ LMDB ....found v2.9.1
-llmdb , -DWITH_LMDB

P.S. Apart from ./configure --with-lmdb & SecDataDir /etc/nginx/modsec/storage/ , do we need to do anything else to turn on lmdb support?

[zimmerle]

please check the content and size of the files....

[nasirbas1]

@zimmerle which files ... the ones which lmdb is supposed to create ?
like ip.pag and ip.dir ? If so no such file exists