Skip to content

Limit resources used by regex_remap to prevent crashes on stack overflow #5762

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SolidWallOfCode merged 2 commits into from
Aug 15, 2019
Merged

Limit resources used by regex_remap to prevent crashes on stack overflow #5762

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
070b4b9
Limit resources used by regex_remap to prevent crashes on resource ex…
SolidWallOfCode Jul 25, 2019
926c550
fix.
SolidWallOfCode Aug 6, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions plugins/regex_remap/regex_remap.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -400,11 +400,14 @@ RemapRegex::compile(const char *&error, int &erroffset)
return -1;
}

_extra = pcre_study(_rex, 0, &error);
if ((_extra == nullptr) && (error != nullptr)) {
_extra = pcre_study(_rex, PCRE_STUDY_EXTRA_NEEDED, &error);
if (error != nullptr) {

@shinrich shinrich Aug 14, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should you still have a test somewhere to return if _extra is nullptr before you dereference it?

@SolidWallOfCode SolidWallOfCode Aug 15, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. Because of the PCRE_STUDY_EXTRA_NEEDED, the returned value will only be nullptr if memory is exhausted, in which case we're already over the cliff.

return -1;
}

_extra->match_limit_recursion = 2047;
_extra->flags |= PCRE_EXTRA_MATCH_LIMIT_RECURSION;

if (pcre_fullinfo(_rex, _extra, PCRE_INFO_CAPTURECOUNT, &ccount) != 0) {
error = "call to pcre_fullinfo() failed";
return -1;
Expand Down Expand Up @@ -640,6 +643,7 @@ struct RemapInstance {
bool host = false;
int hits = 0;
int misses = 0;
int failures = 0;
std::string filename;
};

Expand Down Expand Up @@ -874,6 +878,7 @@ TSRemapDeleteInstance(void *ih)
fprintf(stderr, "[%s]: Profiling information for regex_remap file `%s':\n", now, (ri->filename).c_str());
fprintf(stderr, "[%s]: Total hits (matches): %d\n", now, ri->hits);
fprintf(stderr, "[%s]: Total missed (no regex matches): %d\n", now, ri->misses);
fprintf(stderr, "[%s]: Total regex internal errors: %d\n", now, ri->failures);

if (ri->hits > 0) { // Avoid divide by zeros...
int ix = 1;
Expand Down Expand Up @@ -981,7 +986,8 @@ TSRemapDoRemap(void *ih, TSHttpTxn txnp, TSRemapRequestInfo *rri)
// Apply the regular expressions, in order. First one wins.
while (re) {
// Since we check substitutions on parse time, we don't need to reset ovector
if (re->match(match_buf, match_len, ovector) != -1) {
auto match_result = re->match(match_buf, match_len, ovector);
if (match_result >= 0) {
int new_len = re->get_lengths(ovector, lengths, rri, &req_url);

// Set timeouts
Expand Down Expand Up @@ -1072,6 +1078,10 @@ TSRemapDoRemap(void *ih, TSHttpTxn txnp, TSRemapRequestInfo *rri)
}
break;
}
} else if (match_result != -1) {
ink_atomic_increment(&(ri->failures), 1);
TSError("[%s] Bad regular expression result %d from \"%s\" in file \"%s\".", PLUGIN_NAME, match_result, re->regex(),
ri->filename.c_str());
}

// Try the next regex
Expand Down
4 changes: 4 additions & 0 deletions tests/gold_tests/pluginTest/regex_remap/gold/regex_remap_crash.gold
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
HTTP/1.1 200 OK
``
uuid: 180
``
4 changes: 4 additions & 0 deletions tests/gold_tests/pluginTest/regex_remap/gold/regex_remap_smoke.gold
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
HTTP/1.1 200 OK
``
uuid: smoke
``
92 changes: 92 additions & 0 deletions tests/gold_tests/pluginTest/regex_remap/regex_remap.test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import os
import time
import json
Test.Summary = '''
Test regex_remap
'''

@x-jesse-zhang x-jesse-zhang Aug 5, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can add Test.Variables.Autest.NormalizeKill = 0 to work around the -9 sigkill return code on microserver while I try to work up a fix.

## Test description:
# Load up cache, ensure fresh
# Create regex reval rule, config reload:
# ensure item is staled only once.
# Add a new rule, config reload:
# ensure item isn't restaled again, but rule still in effect.
#
# If the rule disappears from regex_revalidate.conf its still loaded!!
# A rule's expiry can't be changed after the fact!

Test.SkipUnless(
Condition.PluginExists('regex_remap.so'),
)
Test.ContinueOnFail = False

# configure origin server
server = Test.MakeOriginServer("server", lookup_key="{%uuid}")
server.addSessionFromFiles("replay")
replay = {}
with open(os.path.join(Test.TestDirectory, 'replay/yts-2819.replay.json')) as src:
replay = json.load(src)

replay_txns = replay["sessions"][0]["transactions"]

# Define ATS and configure
ts = Test.MakeATSProcess("ts", command="traffic_server")

testName = "regex_remap"

regex_remap_conf_path = os.path.join(ts.Variables.CONFIGDIR, 'regex_remap.conf')
curl_and_args = 'curl -s -D - -v --proxy localhost:{} '.format(ts.Variables.port)

path1_rule = 'path1 {}\n'.format(int(time.time()) + 600)

ts.Disk.File(regex_remap_conf_path, typename="ats:config").AddLines([
"# regex_remap configuration\n"
"^/alpha/bravo/[?]((?!action=(newsfeed|calendar|contacts|notepad)).)*$ http://example.one @status=301\n"
])

ts.Disk.remap_config.AddLine(
"map http://example.one/ http://localhost:{}/ @plugin=regex_remap.so @pparam=regex_remap.conf\n".format(server.Variables.Port)
)

# minimal configuration
ts.Disk.records_config.update({
'proxy.config.diags.debug.enabled': 1,
'proxy.config.diags.debug.tags': 'http|regex_remap',
'proxy.config.http.cache.http': 0,
'proxy.config.http.server_ports': '{}'.format(ts.Variables.port),
})

# 0 Test - Load cache (miss) (path1)
tr = Test.AddTestRun("smoke test")
tr.Processes.Default.StartBefore(server)
tr.Processes.Default.StartBefore(Test.Processes.ts)
creq=replay_txns[0]['client-request']
tr.Processes.Default.Command = curl_and_args + '--header "uuid: {}" '.format(creq["headers"]["fields"][1][1]) + creq["url"]
tr.Processes.Default.ReturnCode = 0
tr.Processes.Default.Streams.stdout = "gold/regex_remap_smoke.gold"
tr.StillRunningAfter = ts

# Crash test.

@x-jesse-zhang x-jesse-zhang Aug 5, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To address the TS failure in Autest, you need to this to test your diags.log
ts.Disk.diags_log.Content = Testers.ContainsExpression(<expression you want>, <description>)

tr = Test.AddTestRun("crash test")
creq=replay_txns[1]['client-request']
tr.Processes.Default.Command = curl_and_args + '--header "uuid: {}" '.format(creq["headers"]["fields"][1][1]) + '"{}"'.format(creq["url"])
tr.Processes.Default.ReturnCode = 0
tr.Processes.Default.Streams.stdout = "gold/regex_remap_crash.gold"
ts.Disk.diags_log.Content = Testers.ContainsExpression('ERROR: .regex_remap. Bad regular expression result -21', "Resource limit exceeded")
tr.StillRunningAfter = ts
155 changes: 155 additions & 0 deletions tests/gold_tests/pluginTest/regex_remap/replay/yts-2819.replay.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
{
"meta": {
"version": "1.0"
},
"sessions": [
{
"protocols": [
"tcp",
"ipv6"
],
"transactions": [
{
"uuid": "smoke",
"client-request": {
"version": "1.1",
"scheme": "http",
"method": "GET",
"url": "http://example.one/",
"headers": {
"fields": [
[
"Host",
"example.one"
],
[
"uuid",
"smoke"
]
]
}
},
"proxy-request": {
"version": "1.1",
"scheme": "http",
"method": "GET",
"url": "http://example.one/",
"headers": {
"fields": [
[
"uuid",
"smoke"
]
]
}
},
"server-response": {
"status": 200,
"reason": "OK",
"content": {
"size": 6128
},
"headers": {
"fields": [
[
"Host",
"example.one"
],
[
"uuid",
"smoke"
],
[
"Content-Length",
"6128"
]
]
}
},
"proxy-response": {
"status": 200,
"reason": "OK",
"content": {
"size": 6128
},
"headers": {
"fields": [
[ "Content-Length", 6128 ]
]
}
}
},
{
"uuid": "180",
"client-request": {
"version": "1.1",
"scheme": "http",
"method": "GET",
"url": "http://example.one/alpha/bravo/?action=newsfed;param0001=00003E;param0002=00004E;param0003=00005E;param0004=00006E;param0005=00007E;param0006=00008E;param0007=00009E;param0008=0000AE;param0009=0000BE;param0010=0000CE;param0011=0000DE;param0012=0000EE;param0013=0000FE;param0014=00010E;param0015=00011E;param0016=00012E;param0017=00013E;param0018=00014E;param0019=00015E;param0020=00016E;param0021=00017E;param0022=00018E;param0023=00019E;param0024=0001AE;param0025=0001BE;param0026=0001CE;param0027=0001DE;param0028=0001EE;param0029=0001FE;param0030=00020E;param0031=00021E;param0032=00022E;param0033=00023E;param0034=00024E;param0035=00025E;param0036=00026E;param0037=00027E;param0038=00028E;param0039=00029E;param0040=0002AE;param0041=0002BE;param0042=0002CE;param0043=0002DE;param0044=0002EE;param0045=0002FE;param0046=00030E;param0047=00031E;param0048=00032E;param0049=00033E;param0050=00034E;param0051=00035E;param0052=00036E;param0053=00037E;param0054=00038E;param0055=00039E;param0056=0003AE;param0057=0003BE;param0058=0003CE;param0059=0003DE;param0060=0003EE;param0061=0003FE;param0062=00040E;param0063=00041E;param0064=00042E;param0065=00043E;param0066=00044E;param0067=00045E;param0068=00046E;param0069=00047E;param0070=00048E;param0071=00049E;param0072=0004AE;param0073=0004BE;param0074=0004CE;param0075=0004DE;param0076=0004EE;param0077=0004FE;param0078=00050E;param0079=00051E;param0080=00052E;param0081=00053E;param0082=00054E;param0083=00055E;param0084=00056E;param0085=00057E;param0086=00058E;param0087=00059E;param0088=0005AE;param0089=0005BE;param0090=0005CE;param0091=0005DE;param0092=0005EE;param0093=0005FE;param0094=00060E;param0095=00061E;param0096=00062E;param0097=00063E;param0098=00064E;param0099=00065E;param0100=00066E;param0101=00067E;param0102=00068E;param0103=00069E;param0104=0006AE;param0105=0006BE;param0106=0006CE;param0107=0006DE;param0108=0006EE;param0109=0006FE;param0110=00070E;param0111=00071E;param0112=00072E;param0113=00073E;param0114=00074E;param0115=00075E;param0116=00076E;param0117=00077E;param0118=00078E;param0119=00079E;param0120=0007AE;param0121=0007BE;param0122=0007CE;param0123=0007DE;param0124=0007EE;param0125=0007FE;param0126=00080E;param0127=00081E;param0128=00082E;param0129=00083E;param0130=00084E;param0131=00085E;param0132=00086E;param0133=00087E;param0134=00088E;param0135=00089E;param0136=0008AE;param0137=0008BE;param0138=0008CE;param0139=0008DE;param0140=0008EE;param0141=0008FE;param0142=00090E;param0143=00091E;param0144=00092E;param0145=00093E;param0146=00094E;param0147=00095E;param0148=00096E;param0149=00097E;param0150=00098E;param0151=00099E;param0152=0009AE;param0153=0009BE;param0154=0009CE;param0155=0009DE;param0156=0009EE;param0157=0009FE;param0158=000A0E;param0159=000A1E;param0160=000A2E;param0161=000A3E;param0162=000A4E;param0163=000A5E;param0164=000A6E;param0165=000A7E;param0166=000A8E;param0167=000A9E;param0168=000AAE;param0169=000ABE;param0170=000ACE;param0171=000ADE;param0172=000AEE;param0173=000AFE;param0174=000B0E;param0175=000B1E;param0176=000B2E;param0177=000B3E;param0178=000B4E;param0179=000B5E",
"headers": {
"fields": [
[
"Host",
"example.one"
],
[
"uuid",
"180"
]
]
}
},
"proxy-request": {
"version": "1.1",
"scheme": "http",
"method": "GET",
"url": "http://example.one/alpha/bravo/?action=newsfed;param0001=00003E;param0002=00004E;param0003=00005E;param0004=00006E;param0005=00007E;param0006=00008E;param0007=00009E;param0008=0000AE;param0009=0000BE;param0010=0000CE;param0011=0000DE;param0012=0000EE;param0013=0000FE;param0014=00010E;param0015=00011E;param0016=00012E;param0017=00013E;param0018=00014E;param0019=00015E;param0020=00016E;param0021=00017E;param0022=00018E;param0023=00019E;param0024=0001AE;param0025=0001BE;param0026=0001CE;param0027=0001DE;param0028=0001EE;param0029=0001FE;param0030=00020E;param0031=00021E;param0032=00022E;param0033=00023E;param0034=00024E;param0035=00025E;param0036=00026E;param0037=00027E;param0038=00028E;param0039=00029E;param0040=0002AE;param0041=0002BE;param0042=0002CE;param0043=0002DE;param0044=0002EE;param0045=0002FE;param0046=00030E;param0047=00031E;param0048=00032E;param0049=00033E;param0050=00034E;param0051=00035E;param0052=00036E;param0053=00037E;param0054=00038E;param0055=00039E;param0056=0003AE;param0057=0003BE;param0058=0003CE;param0059=0003DE;param0060=0003EE;param0061=0003FE;param0062=00040E;param0063=00041E;param0064=00042E;param0065=00043E;param0066=00044E;param0067=00045E;param0068=00046E;param0069=00047E;param0070=00048E;param0071=00049E;param0072=0004AE;param0073=0004BE;param0074=0004CE;param0075=0004DE;param0076=0004EE;param0077=0004FE;param0078=00050E;param0079=00051E;param0080=00052E;param0081=00053E;param0082=00054E;param0083=00055E;param0084=00056E;param0085=00057E;param0086=00058E;param0087=00059E;param0088=0005AE;param0089=0005BE;param0090=0005CE;param0091=0005DE;param0092=0005EE;param0093=0005FE;param0094=00060E;param0095=00061E;param0096=00062E;param0097=00063E;param0098=00064E;param0099=00065E;param0100=00066E;param0101=00067E;param0102=00068E;param0103=00069E;param0104=0006AE;param0105=0006BE;param0106=0006CE;param0107=0006DE;param0108=0006EE;param0109=0006FE;param0110=00070E;param0111=00071E;param0112=00072E;param0113=00073E;param0114=00074E;param0115=00075E;param0116=00076E;param0117=00077E;param0118=00078E;param0119=00079E;param0120=0007AE;param0121=0007BE;param0122=0007CE;param0123=0007DE;param0124=0007EE;param0125=0007FE;param0126=00080E;param0127=00081E;param0128=00082E;param0129=00083E;param0130=00084E;param0131=00085E;param0132=00086E;param0133=00087E;param0134=00088E;param0135=00089E;param0136=0008AE;param0137=0008BE;param0138=0008CE;param0139=0008DE;param0140=0008EE;param0141=0008FE;param0142=00090E;param0143=00091E;param0144=00092E;param0145=00093E;param0146=00094E;param0147=00095E;param0148=00096E;param0149=00097E;param0150=00098E;param0151=00099E;param0152=0009AE;param0153=0009BE;param0154=0009CE;param0155=0009DE;param0156=0009EE;param0157=0009FE;param0158=000A0E;param0159=000A1E;param0160=000A2E;param0161=000A3E;param0162=000A4E;param0163=000A5E;param0164=000A6E;param0165=000A7E;param0166=000A8E;param0167=000A9E;param0168=000AAE;param0169=000ABE;param0170=000ACE;param0171=000ADE;param0172=000AEE;param0173=000AFE;param0174=000B0E;param0175=000B1E;param0176=000B2E;param0177=000B3E;param0178=000B4E;param0179=000B5E",
"headers": {
"fields": [
[
"uuid",
"180"
]
]
}
},
"server-response": {
"status": 200,
"reason": "OK",
"content": {
"size": 6128
},
"headers": {
"fields": [
[
"Host",
"example.one"
],
[
"uuid",
"180"
],
[
"Content-Length",
"6128"
]
]
}
},
"proxy-response": {
"status": 200,
"reason": "OK",
"content": {
"size": 6128
},
"headers": {
"fields": [
[ "Content-Length", 6128 ]
]
}
}
}
]
}
]
}