Musser-Nishanov generic sequence-matching algorithm(s)

[jeremy-murphy]

Introduction

In 1997, David R. Musser and Gor V. Nishanov wrote a hitherto unpublished paper and accompanying code, A Fast Generic Sequence Matching Algorithm.

It struck me as odd that this algorithm had not been widely adopted, as Musser's introsort had been.
So I contacted the authors and with their permission am attempting to bring it to a wider audience.

I provide the abstract from their paper here for completeness:

A string matching—and more generally, sequence matching—algorithm is presented
that has a linear worst-case computing time bound, a low worst-case bound on the
number of comparisons (2n), and sublinear average-case behavior that is better than
that of the fastest versions of the Boyer-Moore algorithm. The algorithm retains its
efficiency advantages in a wide variety of sequence matching problems of practical
interest, including traditional string matching; large-alphabet problems (as in Unicode
strings); and small-alphabet, long-pattern problems (as in DNA searches). Since it is
expressed as a generic algorithm for searching in sequences over an arbitrary type T , it
is well suited for use in generic software libraries such as the C ++ Standard Template
Library. The algorithm was obtained by adding to the Knuth-Morris-Pratt algorithm
one of the pattern-shifting techniques from the Boyer-Moore algorithm, with provision
for use of hashing in this technique. In situations in which a hash function or random
access to the sequences is not available, the algorithm falls back to an optimized version
of the Knuth-Morris-Pratt algorithm.

In the benchmarks that I have run, it is faster than Boyer-Moore, but equivalent to Boyer-Moore-Horspool, on 8-bit text.

On '2-bit' text (DNA sequence) however, the custom search traits boosts the performance to 5-10x faster than the nearest rival. I have demonstrated this in a new search test, no. 5.

What is also interesting is that a simpler, fallback algorithm is provided for corpus iterators that are not random access.

Design

I have not changed the fundamental algorithms in a material way, they should be recognizable from the original code apart from some simplifications.

What I have done personally is to restructure the interface to match the existing Boost.Algorithm searchers. The Musser-Nishanov search strategy is complicated by having two algorithms, accelerated linear (AL) and hashed accelerated linear (HAL), that are mostly but not entirely independent. If the corpus iterator is not random-access or the (static) suffix size is zero, then AL is selected at compile time. Otherwise, a HAL search object is created, but this does not guarantee that HAL will be used. If the pattern size (known at run time) is smaller than the suffix size, then the HAL search object must fall back to AL.

The user-facing API is a two-part musser_nishanov class that does the initial static choice between AL and HAL. Since HAL uses the same data structures as AL plus one more it is easy to implement HAL in terms of AL. This justified making AL a distinct class that is inherited publicly by the non-random-access musser_nishanov class, and inherited privately by the HAL class so that it could be used as the fallback algorithm. The HAL class currently is the musser_nishanov class for random access iterators, as opposed to being a distinct class.

The following is a crude UML diagram of the class hierarchy, showing the private inheritance in brown and public inheritance in blue, following Doxygen style.

Since the HAL search object makes its final decision about algorithm at run time, I had to represent this choice somehow. The simplest, though not necessarily most efficient, way is with bind() and function<>. It also chooses a null searcher if the pattern is empty. This method of storing which
algorithm to call might be causing a slight performance penalty, but it requires more detailed tests.

Selected Benchmarks

I whittled the benchmark output down for the purpose of displaying here. These values should be taken with a grain of 5-10% salt: Musser-Nishanov is not consistently faster than Boyer-Moore-Horspool for example.

8-bit random characters

This is an edited search_test2 output:

Corpus is 2756252 entries long
---- Middle -----
Pattern is 105 entries long
std::search 0.4848 seconds 100% 484781
boyer_moore_search 0.1171 seconds 24.16% 117117
boyer_moore_horspool_search 0.1076 seconds 22.2% 107602
musser_nishanov_search 0.1033 seconds 21.31% 103296
------ End ------
Pattern is 43 entries long
std::search 0.6206 seconds 100% 620587
boyer_moore_search 0.2075 seconds 33.44% 207515
boyer_moore_horspool_search 0.1742 seconds 28.07% 174218
musser_nishanov_search 0.1686 seconds 27.17% 168633
--- Not found ---
Pattern is 91 entries long
std::search 1.028 seconds 100% 1028228
boyer_moore_search 0.1971 seconds 19.17% 197079
boyer_moore_horspool_search 0.1825 seconds 17.75% 182502
musser_nishanov_search 0.1702 seconds 16.56% 170235

DNA sequences

And this is an edited search_test5 output, which is '2-bit' (DNA) sequence data, for which Musser-Nishanov has a few customized search traits, but I included just one here. The number --- here --- is the pattern size, search is timed for finding all matches (which may be zero -- this benchmark needs some work).
Just to be clear, there is no musser_nishanov_dna class, that is just shorthand for specialization.

Corpus is 997532 entries long
--- 10 ---
matches: 2
std::search 2.239 seconds 100% 2239480
boyer_moore object 0.6368 seconds 28.43% 636794
boyer_moore_horspool object 0.664 seconds 29.65% 663994
musser_nishanov object 0.7033 seconds 31.4% 703286
musser_nishanov_dna object 0.3932 seconds 17.56% 393151
--- 20 ---
matches: 0
std::search 1.99 seconds 100% 1989855
boyer_moore object 0.9167 seconds 46.07% 916750
boyer_moore_horspool object 1.006 seconds 50.54% 1005750
musser_nishanov object 0.9499 seconds 47.74% 949912
musser_nishanov_dna object 0.1749 seconds 8.791% 174933
--- 40 ---
matches: 0
std::search 1.881 seconds 100% 1880573
boyer_moore object 0.7201 seconds 38.29% 720095
boyer_moore_horspool object 0.8477 seconds 45.08% 847738
musser_nishanov object 0.8151 seconds 43.34% 815128
musser_nishanov_dna object 0.09294 seconds 4.942% 92943
--- 150 ---
matches: 11
std::search 1.983 seconds 100% 1982931
boyer_moore object 0.6318 seconds 31.86% 631754
boyer_moore_horspool object 0.6806 seconds 34.32% 680592
musser_nishanov object 0.6594 seconds 33.25% 659393
musser_nishanov_dna object 0.04976 seconds 2.509% 49759

Caveats (or, what is unfinished)

I did not want to let perfect be the enemy of good, but more importantly I wanted to get a conversation about this code underway as soon as possible, so I have opened this PR well before everything is complete.

• Documentation.
• 'wide char' benchmark.
• Non-random-access corpus benchmark.
• Worst-case benchmarks.
• The benchmark suite for search test 5 needs to be either trimmed or better utilized.

[zamazan4ik]

@mclow can you check this PR please?

[jeremy-murphy]

@zamazan4ik it's OK, I'll call for Marshall's attention when it's finished. The list of caveats/unfinished things is real; I'll tick them off or remove them when they're done or remove them completely if I change my mind about them. No need to keep drawing his attention until then, but I'm always ready to engage in a conversation about what is currently here.

[mclow]

Very, very odd. I just got an email about three new commits. But when I come here, I see they were made 16 days ago.

[jeremy-murphy]

Very, very odd. I just got an email about three new commits. But when I come here, I see they were made 16 days ago.

Some of the commits might be that old, I only just pushed for the first time in ages.