Fix handling of backtracking stack with some loops by stephentoub · Pull Request #79353 · dotnet/runtime

stephentoub · 2022-12-07T17:52:42Z

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes#79298, but I think we should backport this to release/7.0.

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with - a minimum bound of at least 2 - no child constructs that backtrack - and a child that's more than a one/notone/set (aka things that match a single character) are possibly leaving state on the backtracking stack when: - at least one iteration of the loop successfully matches - but not enough iterations match to make the loop successful such that matching the loop fails In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack. We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

ghost · 2022-12-07T17:52:58Z

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes #79298, but I think we should backport this.

Author:	stephentoub
Assignees:	-
Labels:	`area-System.Text.RegularExpressions`
Milestone:	8.0.0

stephentoub · 2022-12-07T20:34:01Z

/backport to release/7.0

github-actions · 2022-12-07T20:34:15Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3642612581

stephentoub · 2022-12-07T17:52:42Z

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes#79298, but I think we should backport this to release/7.0.

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with - a minimum bound of at least 2 - no child constructs that backtrack - and a child that's more than a one/notone/set (aka things that match a single character) are possibly leaving state on the backtracking stack when: - at least one iteration of the loop successfully matches - but not enough iterations match to make the loop successful such that matching the loop fails In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack. We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

ghost · 2022-12-07T17:52:58Z

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes #79298, but I think we should backport this.

Author:	stephentoub
Assignees:	-
Labels:	`area-System.Text.RegularExpressions`
Milestone:	8.0.0

stephentoub · 2022-12-07T20:34:01Z

/backport to release/7.0

github-actions · 2022-12-07T20:34:15Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3642612581

stephentoub · 2022-12-07T17:52:42Z

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes#79298, but I think we should backport this to release/7.0.

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with - a minimum bound of at least 2 - no child constructs that backtrack - and a child that's more than a one/notone/set (aka things that match a single character) are possibly leaving state on the backtracking stack when: - at least one iteration of the loop successfully matches - but not enough iterations match to make the loop successful such that matching the loop fails In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack. We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

ghost · 2022-12-07T17:52:58Z

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes #79298, but I think we should backport this.

Author:	stephentoub
Assignees:	-
Labels:	`area-System.Text.RegularExpressions`
Milestone:	8.0.0

stephentoub · 2022-12-07T20:34:01Z

/backport to release/7.0

github-actions · 2022-12-07T20:34:15Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3642612581

stephentoub · 2022-12-07T17:52:42Z

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes#79298, but I think we should backport this to release/7.0.

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with - a minimum bound of at least 2 - no child constructs that backtrack - and a child that's more than a one/notone/set (aka things that match a single character) are possibly leaving state on the backtracking stack when: - at least one iteration of the loop successfully matches - but not enough iterations match to make the loop successful such that matching the loop fails In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack. We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

ghost · 2022-12-07T17:52:58Z

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes #79298, but I think we should backport this.

Author:	stephentoub
Assignees:	-
Labels:	`area-System.Text.RegularExpressions`
Milestone:	8.0.0

stephentoub · 2022-12-07T20:34:01Z

/backport to release/7.0

github-actions · 2022-12-07T20:34:15Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3642612581

stephentoub · 2022-12-07T17:52:42Z

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes#79298, but I think we should backport this to release/7.0.

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with - a minimum bound of at least 2 - no child constructs that backtrack - and a child that's more than a one/notone/set (aka things that match a single character) are possibly leaving state on the backtracking stack when: - at least one iteration of the loop successfully matches - but not enough iterations match to make the loop successful such that matching the loop fails In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack. We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

ghost · 2022-12-07T17:52:58Z

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes #79298, but I think we should backport this.

Author:	stephentoub
Assignees:	-
Labels:	`area-System.Text.RegularExpressions`
Milestone:	8.0.0

stephentoub · 2022-12-07T20:34:01Z

/backport to release/7.0

github-actions · 2022-12-07T20:34:15Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3642612581

stephentoub · 2022-12-07T17:52:42Z

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes#79298, but I think we should backport this to release/7.0.

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with - a minimum bound of at least 2 - no child constructs that backtrack - and a child that's more than a one/notone/set (aka things that match a single character) are possibly leaving state on the backtracking stack when: - at least one iteration of the loop successfully matches - but not enough iterations match to make the loop successful such that matching the loop fails In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack. We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

ghost · 2022-12-07T17:52:58Z

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes #79298, but I think we should backport this.

Author:	stephentoub
Assignees:	-
Labels:	`area-System.Text.RegularExpressions`
Milestone:	8.0.0

stephentoub · 2022-12-07T20:34:01Z

/backport to release/7.0

github-actions · 2022-12-07T20:34:15Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3642612581

stephentoub · 2022-12-07T17:52:42Z

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes#79298, but I think we should backport this to release/7.0.

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with - a minimum bound of at least 2 - no child constructs that backtrack - and a child that's more than a one/notone/set (aka things that match a single character) are possibly leaving state on the backtracking stack when: - at least one iteration of the loop successfully matches - but not enough iterations match to make the loop successful such that matching the loop fails In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack. We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

ghost · 2022-12-07T17:52:58Z

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes #79298, but I think we should backport this.

Author:	stephentoub
Assignees:	-
Labels:	`area-System.Text.RegularExpressions`
Milestone:	8.0.0

stephentoub · 2022-12-07T20:34:01Z

/backport to release/7.0

github-actions · 2022-12-07T20:34:15Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3642612581

stephentoub · 2022-12-07T17:52:42Z

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes#79298, but I think we should backport this to release/7.0.

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with - a minimum bound of at least 2 - no child constructs that backtrack - and a child that's more than a one/notone/set (aka things that match a single character) are possibly leaving state on the backtracking stack when: - at least one iteration of the loop successfully matches - but not enough iterations match to make the loop successful such that matching the loop fails In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack. We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

ghost · 2022-12-07T17:52:58Z

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

With both RegexOptions.Compiled and the Regex source generator, Regex greedy loops with

a minimum bound of at least 2
no child constructs that backtrack
and a child that's more than a one/notone/set (aka things that match a single character)

are possibly leaving state on the backtracking stack when:

at least one iteration of the loop successfully matches
but not enough iterations match to make the loop successful such that matching the loop fails

In that case, if a previous construct in the pattern pushed any state onto the backtracking stack such that it expects to be able to pop off and use that state upon backtracking to it, it will potentially pop the erroneously leftover state. This can then cause execution to go awry, as it's getting back an unexpected value. That can lead to false positives, false negatives, or exceptions such as an IndexOutOfRangeException due to trying to pop too much from the backtracking stack.

We already have the ability to remember the backtracking stack position when we initially enter the loop so that we can reset to that position later on. The fix is simply to extend that to also perform that reset when failing the match of such a loop in such circumstances.

Fixes #79298, but I think we should backport this.

Author:	stephentoub
Assignees:	-
Labels:	`area-System.Text.RegularExpressions`
Milestone:	8.0.0

stephentoub · 2022-12-07T20:34:01Z

/backport to release/7.0

github-actions · 2022-12-07T20:34:15Z

Started backporting to release/7.0: https://github.com/dotnet/runtime/actions/runs/3642612581

stephentoub added the area-System.Text.RegularExpressions label Dec 7, 2022

stephentoub added this to the 8.0.0 milestone Dec 7, 2022

stephentoub requested a review from joperezr December 7, 2022 17:52

ghost assigned stephentoub Dec 7, 2022

stephentoub mentioned this pull request Dec 7, 2022
IndexOutOfRangeException when using compiled and ignorecase regex in .NET 7.0 #79298
Closed

joperezr approved these changes Dec 7, 2022
View reviewed changes

stephentoub merged commit 13a9a3c into dotnet:mainDec 7, 2022

stephentoub deleted the fixregexloopstackreset branch December 7, 2022 20:33

github-actionsBot mentioned this pull request Dec 7, 2022
[release/7.0] Fix handling of backtracking stack with some loops #79361
Merged

stephentoub mentioned this pull request Dec 8, 2022
Fix startingStackpos tracking for loop in loop #79382
Merged

ghost locked as resolved and limited conversation to collaborators Jan 7, 2023