[API Proposal]: An API to optimally encapsulate several searching algorithms and techniques, using one simple class

[panost]

Background and motivation

Too many overloads of searching in ReadOnlySpan<T> and yet they cover only very simple search cases. In addition to those methods, in each invocation, they calculate or examine several cases before actually find the best course of how to optimally perform the search. See for example SpanHelpers.IndexOfAny or in the same class what IndexOf does when searching for a sequence.

We need a class where not only all those cases will be calculated or examined once and used thousand or even million of times, but also a class that will support more complex searching scenarios.

API Proposal

A. Base class

public abstract class Scanner<T> {
 public abstract bool Contain( T value );
 // Actually a placeholder. This should be overridden by all implementations
 public virtual int IndexOf( ReadOnlySpan<T> span ) {
 for ( int i = 0; i < span.Length; i++ ) {
 if ( Contain( span[ i ] ) )
 return i;
 }
 return -1;
 }

 public virtual int LastIndexOf( ReadOnlySpan<T> span ) {
 for ( int i = span.Length-1; i >=0 ; i-- ) {
 if ( Contain( span[ i ] ) )
 return i;
 }

 return -1;
 }
 
 public int IndexOf( ReadOnlySpan<T> span, int index ) {
 int idx = IndexOf( span.Slice( index ) );
 if ( idx >= 0 )
 return idx + index;
 return -1;
 }
 //..More IndexOf, LastIndexOf and others follow, that actually call the virtual methods above
 
 protected virtual Scanner<T> GetInverse(){
 return new ScannerInverse( this );
 }
 
 protected Scanner<T> _inverse;
 /// <summary>
 /// Get the scanner that performs the inverse search
 /// </summary>
 public Scanner<T> Inverse {
 get => _inverse ??= GetInverse();
 protected internal set => _inverse = value;
 }
}

B. An example of a Scanner that is searching for a simple value. The example is incomplete

public class ScanOne<T> : Scanner<T> where T : struct, IEquatable<T> {
 private readonly T _value;

 public ScanOne( T value ) {
 _value = value;
 }

 public override bool Contain( T value ) {
 return _value.Equals( value );
 }

 public override int IndexOf( ReadOnlySpan<T> span ) {
 return span.IndexOf( _value );
 }

 public override int LastIndexOf( ReadOnlySpan<T> span ) {
 return span.LastIndexOf( _value );
 }

 protected override Scanner<T> GetInverse() {
 return new Inv<T>(this);
 }

 class Inv<T> : Scanner<T> where T : struct, IEquatable<T> {
 private readonly T _value;

 public override bool Contain( T value ) {
 return !_value.Equals( value );
 }
 public Inv( ScanOne<T> parent ) {
 this.Inverse = parent;
 _value = parent._value;
 }
 public override int IndexOf( ReadOnlySpan<T> span ) {
 T value=_value;
 for ( int i = 0; i < span.Length; i++ ) {
 if ( !span[ i ].Equals( value ) )
 return i;
 }

 return -1;
 }
 }
}

C. An example of a Scanner that is using an ASCII lookup table. The example is incomplete

public class Table16CharScanner : Scanner<char> {
 private readonly ushort _mask;
 private ushort[] _table;

 public Table16CharScanner( ushort mask, ushort[] table ) {
 _mask = mask;
 _table = table;
 }
 /// <summary>
 /// Check for characters that are above 0x7F
 /// </summary>
 /// <param name="ch">character to check</param>
 /// <returns>true if is found;false otherwise</returns>
 protected internal virtual bool ContainExt( char ch ) {
 return false;
 }

 protected override Scanner<char> GetInverse() {
 return new Table16CharScannerInverse( this, _mask, _table );
 }

 public sealed override bool Contain( char value ) {
 if ( value < _table.Length ) {
 return ( _table[ value ] & _mask ) != 0;
 }
 return ContainExt( value );
 }

 public override int IndexOf( ReadOnlySpan<char> span ) {
 var mask = _mask;
 var tableLength = _table.Length;
 var table = _table;
 for ( int i = 0; i < span.Length; i++ ) {
 var ch = span[i];
 if ( ch < tableLength ) {
 if ( ( table[ ch ] & mask ) != 0 ) {
 return i;
 }
 } else if ( ContainExt( ch ) ) {
 return i;
 }
 }
 return -1;
 }

D. ReadOnlySpan extensions

public static class ScannerExtensions {
 public int IndexOf<T>( this ReadOnlySpan<T> span, Scanner<T> scanner );
 public int LastIndexOf<T>( this ReadOnlySpan<T> span, Scanner<T> scanner );
 public ReadOnlySpan<T> Trim<T>( this ReadOnlySpan<T> span, Scanner<T> trim );
 public ReadOnlySpan<T> Skip<T>( this ReadOnlySpan<T> span, Scanner<T> scanner );
 public ReadOnlySpan<T> Remain<T>( this ReadOnlySpan<T> span, Scanner<T> scanner );
 public ReadOnlySpan<T> Trim<T>( this ReadOnlySpan<T> span, Scanner<T> trimStart, Scanner<T> trimEnd );
 public ReadOnlySpan<T> TrimStart<T>( this ReadOnlySpan<T> span, Scanner<T> trimStart );
 public ReadOnlySpan<T> TrimEnd<T>( this ReadOnlySpan<T> span, Scanner<T> trimEnd );
 public SplitIterator<T> Split<T>( this ReadOnlySpan<T> span, Scanner<T> search );
 public SplitIterator<T> Split<T>( this ReadOnlySpan<T> span, Scanner<T> search, Scanner<T> trim, bool skipEmpty );
 public int Count( this ReadOnlySpan<char> span, Scanner<T> search );
}

API Usage

ReadOnlySpan<char> span="..";
// trim Unicode whitespace
span = span.Trim( WhiteSpaceScanner );
// trim what JS spec considers white spaces from the start and WhiteSpace or ';' or ',' from the end
span = span.Trim( JavascriptWhitespace, WhiteSpaceSemicolonOrComma ); 
// skip invalid id chars, find id until first invalid
span = span.Skip(HtmlIdScanner ).Remain( HtmlIdScanner.Inverse ); 
// split on ';' or ',' and trim whitespaces for each entry
var iterator = span.Split(SemicolonOrCommaScanner,WhiteSpaceScanner); 

Risks

The current API of Regular expressions can't be optimally encapsulated by a Scanner because it requires a string as input

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

Too many overloads of searching in ReadOnlySpan<T> and yet they cover only very simple search cases. In addition to those methods, in each invocation, they calculate or examine several cases before actually find the best course of how to optimally perform the search. See for example SpanHelpers.IndexOfAny or in the same class what IndexOf does when searching for a sequence.

We need a class where not only all those cases will be calculated or examined once and used thousand or even million of times, but also a class that will support more complex searching scenarios.

API Proposal

A. Base class

public abstract class Scanner<T> {
 public abstract bool Contain( T value );
 // Actually a placeholder. This should be overridden by all implementations
 public virtual int IndexOf( ReadOnlySpan<T> span ) {
 for ( int i = 0; i < span.Length; i++ ) {
 if ( Contain( span[ i ] ) )
 return i;
 }
 return -1;
 }

 public virtual int LastIndexOf( ReadOnlySpan<T> span ) {
 for ( int i = span.Length-1; i >=0 ; i-- ) {
 if ( Contain( span[ i ] ) )
 return i;
 }

 return -1;
 }
 
 public int IndexOf( ReadOnlySpan<T> span, int index ) {
 int idx = IndexOf( span.Slice( index ) );
 if ( idx >= 0 )
 return idx + index;
 return -1;
 }
 //..More IndexOf, LastIndexOf and others follow, that actually call the virtual methods above
 
 protected virtual Scanner<T> GetInverse(){
 return new ScannerInverse( this );
 }
 
 protected Scanner<T> _inverse;
 /// <summary>
 /// Get the scanner that performs the inverse search
 /// </summary>
 public Scanner<T> Inverse {
 get => _inverse ??= GetInverse();
 protected internal set => _inverse = value;
 }
}

B. An example of a Scanner that is searching for a simple value. The example is incomplete

public class ScanOne<T> : Scanner<T> where T : struct, IEquatable<T> {
 private readonly T _value;

 public ScanOne( T value ) {
 _value = value;
 }

 public override bool Contain( T value ) {
 return _value.Equals( value );
 }

 public override int IndexOf( ReadOnlySpan<T> span ) {
 return span.IndexOf( _value );
 }

 public override int LastIndexOf( ReadOnlySpan<T> span ) {
 return span.LastIndexOf( _value );
 }

 protected override Scanner<T> GetInverse() {
 return new Inv<T>(this);
 }

 class Inv<T> : Scanner<T> where T : struct, IEquatable<T> {
 private readonly T _value;

 public override bool Contain( T value ) {
 return !_value.Equals( value );
 }
 public Inv( ScanOne<T> parent ) {
 this.Inverse = parent;
 _value = parent._value;
 }
 public override int IndexOf( ReadOnlySpan<T> span ) {
 T value=_value;
 for ( int i = 0; i < span.Length; i++ ) {
 if ( !span[ i ].Equals( value ) )
 return i;
 }

 return -1;
 }
 }
}

C. An example of a Scanner that is using an ASCII lookup table. The example is incomplete

public class Table16CharScanner : Scanner<char> {
 private readonly ushort _mask;
 private ushort[] _table;

 public Table16CharScanner( ushort mask, ushort[] table ) {
 _mask = mask;
 _table = table;
 }
 /// <summary>
 /// Check for characters that are above 0x7F
 /// </summary>
 /// <param name="ch">character to check</param>
 /// <returns>true if is found;false otherwise</returns>
 protected internal virtual bool ContainExt( char ch ) {
 return false;
 }

 protected override Scanner<char> GetInverse() {
 return new Table16CharScannerInverse( this, _mask, _table );
 }

 public sealed override bool Contain( char value ) {
 if ( value < _table.Length ) {
 return ( _table[ value ] & _mask ) != 0;
 }
 return ContainExt( value );
 }

 public override int IndexOf( ReadOnlySpan<char> span ) {
 var mask = _mask;
 var tableLength = _table.Length;
 var table = _table;
 for ( int i = 0; i < span.Length; i++ ) {
 var ch = span[i];
 if ( ch < tableLength ) {
 if ( ( table[ ch ] & mask ) != 0 ) {
 return i;
 }
 } else if ( ContainExt( ch ) ) {
 return i;
 }
 }
 return -1;
 }

D. ReadOnlySpan extensions

public static class ScannerExtensions {
 public int IndexOf<T>( this ReadOnlySpan<T> span, Scanner<T> scanner );
 public int LastIndexOf<T>( this ReadOnlySpan<T> span, Scanner<T> scanner );
 public ReadOnlySpan<T> Trim<T>( this ReadOnlySpan<T> span, Scanner<T> trim );
 public ReadOnlySpan<T> Skip<T>( this ReadOnlySpan<T> span, Scanner<T> scanner );
 public ReadOnlySpan<T> Remain<T>( this ReadOnlySpan<T> span, Scanner<T> scanner );
 public ReadOnlySpan<T> Trim<T>( this ReadOnlySpan<T> span, Scanner<T> trimStart, Scanner<T> trimEnd );
 public ReadOnlySpan<T> TrimStart<T>( this ReadOnlySpan<T> span, Scanner<T> trimStart );
 public ReadOnlySpan<T> TrimEnd<T>( this ReadOnlySpan<T> span, Scanner<T> trimEnd );
 public SplitIterator<T> Split<T>( this ReadOnlySpan<T> span, Scanner<T> search );
 public SplitIterator<T> Split<T>( this ReadOnlySpan<T> span, Scanner<T> search, Scanner<T> trim, bool skipEmpty );
 public int Count( this ReadOnlySpan<char> span, Scanner<T> search );
}

API Usage

ReadOnlySpan<char> span="..";
// trim Unicode whitespace
span = span.Trim( WhiteSpaceScanner );
// trim what JS spec considers white spaces from the start and WhiteSpace or ';' or ',' from the end
span = span.Trim( JavascriptWhitespace, WhiteSpaceSemicolonOrComma ); 
// skip invalid id chars, find id until first invalid
span = span.Skip(HtmlIdScanner ).Remain( HtmlIdScanner.Inverse ); 
// split on ';' or ',' and trim whitespaces for each entry
var iterator = span.Split(SemicolonOrCommaScanner,WhiteSpaceScanner); 

Risks

The current API of Regular expressions can't be optimally encapsulated by a Scanner because it requires a string as input

Author:	panost
Assignees:	-
Labels:	api-suggestion, area-System.Runtime, untriaged
Milestone:	-

[Joe4evr]

The current API of Regular expressions can't be optimally encapsulated by a Scanner because it requires a string as input

Clearly you missed the efforts of span-ifying the Regex API.

[panost]

It seems that is partially implemented under a new name IndexOfAnyValues here #68328. As I can see, it can only have internal implementations, so no custom optimizations such as lookup tables, are supported. I do not know how can be effectively used in cases like those in #78204

[MihaZupan]

As I can see, it can only have internal implementations, so no custom optimizations such as lookup tables, are supported.

What sort of sets of values were you hoping to see optimized? The IndexOfAnyValues implementation can do much better than a scalar loop around a table lookup in most cases.

I do not know how can be effectively used in cases like those in #78204

Can you elaborate? The issue links to a dozen PRs doing just that

[panost]

Lexer just found a letter and wants to execute an IndexOf first non-valid identifier to locate the end of the identifier. In this case invalid identifier characters are all except A-Z, a-z, _ in the ASCII range. In Unicode range all except letters and digits.
All those rules for the ASCII range are encoded in a lookup table. The same table includes other scan rules, such as valid number characters or white space characters.

If the parser is SQL then use a table that includes the '$' as a valid identifier character, if it is CSS use a lookup table that includes '-'.

Which PR covers such a usage ?
To be clear I want three instances of IndexOfAnyValues to cover identifier scanning for Js, SQL and CSS, so that i can use selectively in a lexer

[MihaZupan]

You could do something like

private static readonly IndexOfAnyValues<char> s_validJsIdentifierChars = IndexOfAnyValues.Create("abc");
private static readonly IndexOfAnyValues<char> s_validSQLIdentifierChars = IndexOfAnyValues.Create("def");
private static readonly IndexOfAnyValues<char> s_validCSSIdentifierChars = IndexOfAnyValues.Create("ghi");

private static int JsIdentifierLength(ReadOnlySpan<char> text)
{
 int i = text.IndexOfAnyExcept(s_validJsIdentifierChars);
 return i < 0 ? text.Length : i;
}

private static int SQLIdentifierLength(ReadOnlySpan<char> text)
{
 int i = text.IndexOfAnyExcept(s_validSQLIdentifierChars);
 return i < 0 ? text.Length : i;
}

// or a shared thing like
private static int IdentifierLength(ReadOnlySpan<char> text, IndexOfAnyValues<char> identifierChars)
{
 int i = text.IndexOfAnyExcept(identifierChars);
 return i < 0 ? text.Length : i;
}

You can essentially treat the IndexOfAnyValues instance as your lookup table.

[panost]

1. Where is the support for the cases outside of the ASCII range? I'll admit it is rare case, but still you have to be consistent. To create a string with length > 60,000, so you can use the IndexOfAnyValues.Create is out of the question!
2. The IndexOfAnyValues.Create when more than 5 chars are used (in my case around 50 chars, just for the ASCII) is finally using an instance of the IndexOfAnyCharValuesProbabilistic class which is a generic Bloom filter and far more slower than a lookup in a table

[MihaZupan]

We're open to adding more specialized implementations as scenarios come up, within reason.
For example, if you have 50 ascii chars, the implementation will end up using this.
For the probabilistic map (what you'd end up getting when using a mix of ascii and non-ascii values), we can vectorize the non-except case - #80963.

It may make sense to add another implementation that really is just a lookup table to handle the -Except case of the probabilistic map, or for large sets of values where the bloom filter isn't as effective.
How many characters per set are you looking at in practice?

To create a string with length > 60,000, so you can use the IndexOfAnyValues.Create is out of the question!

If you have over 60k characters, you can invert the condition and create the lookup with 65536 - 60000 characters to make it a bit more manageable. Keep in mind that you don't have to use a constant there, you can generate the set at runtime.

cc: @stephentoub