Add scan and search benchmark by naitoh · Pull Request #111 · ruby/strscan

naitoh · 2024-10-21T12:17:13Z

Why?

To improve the parsing process, I would like to add benchmarks for all parsing processes.

scan

scan_full(regexp, false, true) == StringScanner#check
scan_full(regexp, false, false) == StringScanner#match?

CRuby

$ benchmark-driver benchmark/scan.yaml
Warming up --------------------------------------
check(reg) 10.558M i/s - 10.848M times in 1.027445s (94.71ns/i)
check(str) 13.368M i/s - 13.782M times in 1.030978s (74.80ns/i)
match?(reg) 16.080M i/s - 16.247M times in 1.010340s (62.19ns/i)
match?(str) 23.336M i/s - 23.501M times in 1.007088s (42.85ns/i)
Calculating -------------------------------------
check(reg) 11.601M i/s - 31.675M times in 2.730287s (86.20ns/i)
check(str) 15.217M i/s - 40.104M times in 2.635475s (65.72ns/i)
match?(reg) 18.781M i/s - 48.241M times in 2.568662s (53.25ns/i)
match?(str) 29.441M i/s - 70.007M times in 2.377840s (33.97ns/i)
Comparison:
match?(str): 29441324.5 i/s
match?(reg): 18780543.7 i/s - 1.57x slower
check(str): 15217130.1 i/s - 1.93x slower
check(reg): 11601371.2 i/s - 2.54x slower

JRuby

$ benchmark-driver benchmark/scan.yaml
Warming up --------------------------------------
check(reg) 8.129M i/s - 8.090M times in 0.995222s (123.02ns/i)
check(str) 16.691M i/s - 16.616M times in 0.995519s (59.91ns/i)
match?(reg) 8.979M i/s - 9.001M times in 1.002440s (111.37ns/i)
match?(str) 26.138M i/s - 26.011M times in 0.995150s (38.26ns/i)
Calculating -------------------------------------
check(reg) 11.808M i/s - 24.387M times in 2.065238s (84.69ns/i)
check(str) 31.762M i/s - 50.072M times in 1.576495s (31.48ns/i)
match?(reg) 13.944M i/s - 26.936M times in 1.931719s (71.71ns/i)
match?(str) 50.872M i/s - 78.414M times in 1.541392s (19.66ns/i)
Comparison:
match?(str): 50872250.2 i/s
check(str): 31761544.3 i/s - 1.60x slower
match?(reg): 13944219.6 i/s - 3.65x slower
check(reg): 11808244.1 i/s - 4.31x slower

search

search_full(regexp, false, true) == StringScanner#check_until
search_full(regexp, false, false) == StringScanner#exist?

$ benchmark-driver benchmark/search.yaml
Warming up --------------------------------------
check_until(reg) 9.338M i/s - 9.456M times in 1.012573s (107.09ns/i)
check_until(str) 11.385M i/s - 11.979M times in 1.052173s (87.83ns/i)
exist?(reg) 13.416M i/s - 13.517M times in 1.007532s (74.54ns/i)
exist?(str) 17.976M i/s - 18.677M times in 1.038981s (55.63ns/i)
Calculating -------------------------------------
check_until(reg) 10.297M i/s - 28.015M times in 2.720634s (97.11ns/i)
check_until(str) 12.684M i/s - 34.156M times in 2.692853s (78.84ns/i)
exist?(reg) 15.184M i/s - 40.249M times in 2.650786s (65.86ns/i)
exist?(str) 21.426M i/s - 53.928M times in 2.517008s (46.67ns/i)
Comparison:
exist?(str): 21425527.1 i/s
exist?(reg): 15183679.9 i/s - 1.41x slower
check_until(str): 12684053.7 i/s - 1.69x slower
check_until(reg): 10297134.8 i/s - 2.08x slower

JRuby

$ benchmark-driver benchmark/search.yaml
Warming up --------------------------------------
check_until(reg) 7.646M i/s - 7.649M times in 1.000381s (130.78ns/i)
check_until(str) 13.075M i/s - 13.010M times in 0.995048s (76.48ns/i)
exist?(reg) 8.728M i/s - 8.684M times in 0.994921s (114.57ns/i)
exist?(str) 20.609M i/s - 20.514M times in 0.995399s (48.52ns/i)
Calculating -------------------------------------
check_until(reg) 9.371M i/s - 22.939M times in 2.447900s (106.71ns/i)
check_until(str) 22.760M i/s - 39.225M times in 1.723414s (43.94ns/i)
exist?(reg) 11.758M i/s - 26.185M times in 2.226997s (85.05ns/i)
exist?(str) 34.564M i/s - 61.827M times in 1.788749s (28.93ns/i)
Comparison:
exist?(str): 34564306.2 i/s
check_until(str): 22759878.4 i/s - 1.52x slower
exist?(reg): 11757927.4 i/s - 2.94x slower
check_until(reg): 9371009.3 i/s - 3.69x slower

## Why? `StringScanner#match?()` is faster than `StringScanner#check()`. See: ruby/strscan#111 ## Benchmark ``` RUBYLIB= BUNDLER_ORIG_RUBYLIB= /Users/naitoh/.rbenv/versions/3.3.4/bin/ruby -v -S benchmark-driver /Users/naitoh/ghq/github.com/naitoh/rexml/benchmark/parse.yaml ruby 3.3.4 (2024-07-09 revision be1089c8ec) [arm64-darwin22] Calculating ------------------------------------- before after before(YJIT) after(YJIT) dom 18.849 19.403 32.551 34.728 i/s - 100.000 times in 5.305314s 5.153743s 3.072111s 2.879488s sax 27.706 29.435 48.126 52.247 i/s - 100.000 times in 3.609376s 3.397367s 2.077880s 1.913973s pull 31.817 33.907 56.941 58.925 i/s - 100.000 times in 3.142961s 2.949250s 1.756193s 1.697082s stream 31.120 33.186 52.530 55.816 i/s - 100.000 times in 3.213334s 3.013325s 1.903689s 1.791600s Comparison: dom after(YJIT): 34.7 i/s before(YJIT): 32.6 i/s - 1.07x slower after: 19.4 i/s - 1.79x slower before: 18.8 i/s - 1.84x slower sax after(YJIT): 52.2 i/s before(YJIT): 48.1 i/s - 1.09x slower after: 29.4 i/s - 1.78x slower before: 27.7 i/s - 1.89x slower pull after(YJIT): 58.9 i/s before(YJIT): 56.9 i/s - 1.03x slower after: 33.9 i/s - 1.74x slower before: 31.8 i/s - 1.85x slower stream after(YJIT): 55.8 i/s before(YJIT): 52.5 i/s - 1.06x slower after: 33.2 i/s - 1.68x slower before: 31.1 i/s - 1.79x slower ``` - YJIT=ON : 1.03x - 1.09x faster - YJIT=OFF : 1.03x - 1.06x faster

kou · 2024-10-22T04:47:56Z

I agree that we should add this benchmark but can we use more meaningful file name than full?
full is too implementation specific and meaningless for this case. (scan_full()/search_full() mean that they accept "full" options. They don't mean any features.)

naitoh · 2024-10-22T05:04:21Z

@kou

but can we use more meaningful file name than full?

How about the following file name?

scan_search.yaml

kou · 2024-10-22T05:44:41Z

It's better than "full".

Do we need to use one file for all cases?
If we use 2 files for check/match? and check_until/exist?, can we use better name than "scan_search"?

naitoh · 2024-10-22T23:46:22Z

I have renamed it to scan_and_search.yaml.

Do we need to use one file for all cases?

In JRuby, there are differences in results for each execution, so I would like to execution everything in one file.

JRuby

$ benchmark-driver benchmark/scan_and_search.yaml
Warming up --------------------------------------
check(reg1) 7.249M i/s - 7.207M times in 0.994265s (137.95ns/i)
check(str1) 19.899M i/s - 19.847M times in 0.997392s (50.25ns/i)
check_until(reg2) 7.918M i/s - 7.918M times in 0.999911s (126.29ns/i)
check_until(str2) 15.581M i/s - 15.463M times in 0.992425s (64.18ns/i)
match?(reg1) 8.717M i/s - 8.742M times in 1.002787s (114.71ns/i)
match?(str1) 25.952M i/s - 25.689M times in 0.989893s (38.53ns/i)
exist?(reg2) 8.715M i/s - 8.700M times in 0.998353s (114.75ns/i)
exist?(str2) 20.071M i/s - 19.938M times in 0.993373s (49.82ns/i)
Calculating -------------------------------------
check(reg1) 12.048M i/s - 21.747M times in 1.804991s (83.00ns/i)
check(str1) 30.335M i/s - 59.697M times in 1.967916s (32.97ns/i)
check_until(reg2) 9.803M i/s - 23.755M times in 2.423189s (102.01ns/i)
check_until(str2) 24.277M i/s - 46.743M times in 1.925398s (41.19ns/i)
match?(reg1) 13.778M i/s - 26.152M times in 1.898151s (72.58ns/i)
match?(str1) 51.994M i/s - 77.855M times in 1.497389s (19.23ns/i)
exist?(reg2) 11.644M i/s - 26.144M times in 2.245393s (85.88ns/i)
exist?(str2) 32.540M i/s - 60.213M times in 1.850455s (30.73ns/i)
Comparison:
match?(str1): 51994015.6 i/s exist?(str2): 32539799.8 i/s - 1.60x slower
check(str1): 30335149.7 i/s - 1.71x slower
check_until(str2): 24277229.9 i/s - 2.14x slower
match?(reg1): 13777722.9 i/s - 3.77x slower
check(reg1): 12048062.8 i/s - 4.32x slower
exist?(reg2): 11643598.9 i/s - 4.47x slower
check_until(reg2): 9803153.3 i/s - 5.30x slower
$ benchmark-driver benchmark/scan_and_search.yaml
Warming up --------------------------------------
check(reg1) 8.691M i/s - 8.643M times in 0.994453s (115.06ns/i)
check(str1) 19.273M i/s - 19.116M times in 0.991856s (51.89ns/i)
check_until(reg2) 7.308M i/s - 7.276M times in 0.995534s (136.83ns/i)
check_until(str2) 15.596M i/s - 15.458M times in 0.991141s (64.12ns/i)
match?(reg1) 8.627M i/s - 8.608M times in 0.997741s (115.91ns/i)
match?(str1) 25.974M i/s - 25.865M times in 0.995829s (38.50ns/i)
exist?(reg2) 8.417M i/s - 8.475M times in 1.006861s (118.80ns/i)
exist?(str2) 9.009M i/s - 8.976M times in 0.996368s (111.00ns/i)
Calculating -------------------------------------
check(reg1) 11.344M i/s - 26.073M times in 2.298316s (88.15ns/i)
check(str1) 35.082M i/s - 57.819M times in 1.648119s (28.50ns/i)
check_until(reg2) 9.956M i/s - 21.925M times in 2.202211s (100.44ns/i)
check_until(str2) 23.878M i/s - 46.788M times in 1.959452s (41.88ns/i)
match?(reg1) 11.412M i/s - 25.882M times in 2.267979s (87.63ns/i)
match?(str1) 49.088M i/s - 77.921M times in 1.587374s (20.37ns/i)
exist?(reg2) 11.628M i/s - 25.252M times in 2.171684s (86.00ns/i)
exist?(str2) 32.934M i/s - 27.027M times in 0.820653s (30.36ns/i)
Comparison:
match?(str1): 49087994.1 i/s check(str1): 35081736.9 i/s - 1.40x slower
exist?(str2): 32933597.9 i/s - 1.49x slower
check_until(str2): 23878238.9 i/s - 2.06x slower
exist?(reg2): 11627809.5 i/s - 4.22x slower
match?(reg1): 11411983.2 i/s - 4.30x slower
check(reg1): 11344492.2 i/s - 4.33x slower
check_until(reg2): 9955951.4 i/s - 4.93x slower

kou · 2024-10-23T01:57:09Z

Do we need to compare check and check_until (match? and exist?)?
They are different operations and use cases. (check/match? check only at the current scan pointer and check_until/exist? check from the current scan pointer.)

I think that JRuby's unstable results show a different problem. We may need to use more long target string (and/or pattern) to make the target operations the main operation in the benchmark.

## Why? `StringScanner#match?()` is faster than `StringScanner#check()`. See: ruby/strscan#111 ## Benchmark ``` RUBYLIB= BUNDLER_ORIG_RUBYLIB= /Users/naitoh/.rbenv/versions/3.3.4/bin/ruby -v -S benchmark-driver /Users/naitoh/ghq/github.com/naitoh/rexml/benchmark/parse.yaml ruby 3.3.4 (2024-07-09 revision be1089c8ec) [arm64-darwin22] Calculating ------------------------------------- before after before(YJIT) after(YJIT) dom 18.849 19.403 32.551 34.728 i/s - 100.000 times in 5.305314s 5.153743s 3.072111s 2.879488s sax 27.706 29.435 48.126 52.247 i/s - 100.000 times in 3.609376s 3.397367s 2.077880s 1.913973s pull 31.817 33.907 56.941 58.925 i/s - 100.000 times in 3.142961s 2.949250s 1.756193s 1.697082s stream 31.120 33.186 52.530 55.816 i/s - 100.000 times in 3.213334s 3.013325s 1.903689s 1.791600s Comparison: dom after(YJIT): 34.7 i/s before(YJIT): 32.6 i/s - 1.07x slower after: 19.4 i/s - 1.79x slower before: 18.8 i/s - 1.84x slower sax after(YJIT): 52.2 i/s before(YJIT): 48.1 i/s - 1.09x slower after: 29.4 i/s - 1.78x slower before: 27.7 i/s - 1.89x slower pull after(YJIT): 58.9 i/s before(YJIT): 56.9 i/s - 1.03x slower after: 33.9 i/s - 1.74x slower before: 31.8 i/s - 1.85x slower stream after(YJIT): 55.8 i/s before(YJIT): 52.5 i/s - 1.06x slower after: 33.2 i/s - 1.68x slower before: 31.1 i/s - 1.79x slower ``` - YJIT=ON : 1.03x - 1.09x faster - YJIT=OFF : 1.03x - 1.06x faster

## Why? `StringScanner#match?()` is faster than `StringScanner#check()`. See: ruby/strscan#111 ## Benchmark ``` RUBYLIB= BUNDLER_ORIG_RUBYLIB= /Users/naitoh/.rbenv/versions/3.3.4/bin/ruby -v -S benchmark-driver /Users/naitoh/ghq/github.com/naitoh/rexml/benchmark/parse.yaml ruby 3.3.4 (2024-07-09 revision be1089c8ec) [arm64-darwin22] Calculating ------------------------------------- before after before(YJIT) after(YJIT) dom 18.819 19.362 32.846 34.708 i/s - 100.000 times in 5.313905s 5.164791s 3.044500s 2.881200s sax 28.188 29.982 48.386 52.554 i/s - 100.000 times in 3.547597s 3.335304s 2.066732s 1.902809s pull 31.962 33.902 57.868 60.662 i/s - 100.000 times in 3.128689s 2.949690s 1.728071s 1.648467s stream 31.436 33.030 52.808 56.647 i/s - 100.000 times in 3.181095s 3.027574s 1.893635s 1.765304s Comparison: dom after(YJIT): 34.7 i/s before(YJIT): 32.8 i/s - 1.06x slower after: 19.4 i/s - 1.79x slower before: 18.8 i/s - 1.84x slower sax after(YJIT): 52.6 i/s before(YJIT): 48.4 i/s - 1.09x slower after: 30.0 i/s - 1.75x slower before: 28.2 i/s - 1.86x slower pull after(YJIT): 60.7 i/s before(YJIT): 57.9 i/s - 1.05x slower after: 33.9 i/s - 1.79x slower before: 32.0 i/s - 1.90x slower stream after(YJIT): 56.6 i/s before(YJIT): 52.8 i/s - 1.07x slower after: 33.0 i/s - 1.72x slower before: 31.4 i/s - 1.80x slower ``` - YJIT=ON : 1.06x - 1.09x faster - YJIT=OFF : 1.02x - 1.06x faster Co-authored-by: Sutou Kouhei <kou@clear-code.com>

## Why? `StringScanner#match?` is faster than `StringScanner#check`. See: ruby/strscan#111 ## Benchmark ``` RUBYLIB= BUNDLER_ORIG_RUBYLIB= /Users/naitoh/.rbenv/versions/3.3.4/bin/ruby -v -S benchmark-driver /Users/naitoh/ghq/github.com/naitoh/rexml/benchmark/parse.yaml ruby 3.3.4 (2024-07-09 revision be1089c8ec) [arm64-darwin22] Calculating ------------------------------------- before after before(YJIT) after(YJIT) dom 18.819 19.362 32.846 34.708 i/s - 100.000 times in 5.313905s 5.164791s 3.044500s 2.881200s sax 28.188 29.982 48.386 52.554 i/s - 100.000 times in 3.547597s 3.335304s 2.066732s 1.902809s pull 31.962 33.902 57.868 60.662 i/s - 100.000 times in 3.128689s 2.949690s 1.728071s 1.648467s stream 31.436 33.030 52.808 56.647 i/s - 100.000 times in 3.181095s 3.027574s 1.893635s 1.765304s Comparison: dom after(YJIT): 34.7 i/s before(YJIT): 32.8 i/s - 1.06x slower after: 19.4 i/s - 1.79x slower before: 18.8 i/s - 1.84x slower sax after(YJIT): 52.6 i/s before(YJIT): 48.4 i/s - 1.09x slower after: 30.0 i/s - 1.75x slower before: 28.2 i/s - 1.86x slower pull after(YJIT): 60.7 i/s before(YJIT): 57.9 i/s - 1.05x slower after: 33.9 i/s - 1.79x slower before: 32.0 i/s - 1.90x slower stream after(YJIT): 56.6 i/s before(YJIT): 52.8 i/s - 1.07x slower after: 33.0 i/s - 1.72x slower before: 31.4 i/s - 1.80x slower ``` - YJIT=ON : 1.05x - 1.09x faster - YJIT=OFF : 1.02x - 1.06x faster --------- Co-authored-by: Sutou Kouhei <kou@clear-code.com>

# Why? To improve the parsing process, I would like to add benchmarks for all parsing processes. ## scan - scan_full(regexp, false, true) == StringScanner#check - scan_full(regexp, false, false) == StringScanner#match? ### CRuby ``` $ benchmark-driver benchmark/scan.yaml Warming up -------------------------------------- check(reg) 10.558M i/s - 10.848M times in 1.027445s (94.71ns/i) check(str) 13.368M i/s - 13.782M times in 1.030978s (74.80ns/i) match?(reg) 16.080M i/s - 16.247M times in 1.010340s (62.19ns/i) match?(str) 23.336M i/s - 23.501M times in 1.007088s (42.85ns/i) Calculating ------------------------------------- check(reg) 11.601M i/s - 31.675M times in 2.730287s (86.20ns/i) check(str) 15.217M i/s - 40.104M times in 2.635475s (65.72ns/i) match?(reg) 18.781M i/s - 48.241M times in 2.568662s (53.25ns/i) match?(str) 29.441M i/s - 70.007M times in 2.377840s (33.97ns/i) Comparison: match?(str): 29441324.5 i/s match?(reg): 18780543.7 i/s - 1.57x slower check(str): 15217130.1 i/s - 1.93x slower check(reg): 11601371.2 i/s - 2.54x slower ``` ### JRuby ``` $ benchmark-driver benchmark/scan.yaml Warming up -------------------------------------- check(reg) 8.129M i/s - 8.090M times in 0.995222s (123.02ns/i) check(str) 16.691M i/s - 16.616M times in 0.995519s (59.91ns/i) match?(reg) 8.979M i/s - 9.001M times in 1.002440s (111.37ns/i) match?(str) 26.138M i/s - 26.011M times in 0.995150s (38.26ns/i) Calculating ------------------------------------- check(reg) 11.808M i/s - 24.387M times in 2.065238s (84.69ns/i) check(str) 31.762M i/s - 50.072M times in 1.576495s (31.48ns/i) match?(reg) 13.944M i/s - 26.936M times in 1.931719s (71.71ns/i) match?(str) 50.872M i/s - 78.414M times in 1.541392s (19.66ns/i) Comparison: match?(str): 50872250.2 i/s check(str): 31761544.3 i/s - 1.60x slower match?(reg): 13944219.6 i/s - 3.65x slower check(reg): 11808244.1 i/s - 4.31x slower ``` ## search - search_full(regexp, false, true) == StringScanner#check_until - search_full(regexp, false, false) == StringScanner#exist? ``` $ benchmark-driver benchmark/search.yaml Warming up -------------------------------------- check_until(reg) 9.338M i/s - 9.456M times in 1.012573s (107.09ns/i) check_until(str) 11.385M i/s - 11.979M times in 1.052173s (87.83ns/i) exist?(reg) 13.416M i/s - 13.517M times in 1.007532s (74.54ns/i) exist?(str) 17.976M i/s - 18.677M times in 1.038981s (55.63ns/i) Calculating ------------------------------------- check_until(reg) 10.297M i/s - 28.015M times in 2.720634s (97.11ns/i) check_until(str) 12.684M i/s - 34.156M times in 2.692853s (78.84ns/i) exist?(reg) 15.184M i/s - 40.249M times in 2.650786s (65.86ns/i) exist?(str) 21.426M i/s - 53.928M times in 2.517008s (46.67ns/i) Comparison: exist?(str): 21425527.1 i/s exist?(reg): 15183679.9 i/s - 1.41x slower check_until(str): 12684053.7 i/s - 1.69x slower check_until(reg): 10297134.8 i/s - 2.08x slower ``` ### JRuby ``` $ benchmark-driver benchmark/search.yaml Warming up -------------------------------------- check_until(reg) 7.646M i/s - 7.649M times in 1.000381s (130.78ns/i) check_until(str) 13.075M i/s - 13.010M times in 0.995048s (76.48ns/i) exist?(reg) 8.728M i/s - 8.684M times in 0.994921s (114.57ns/i) exist?(str) 20.609M i/s - 20.514M times in 0.995399s (48.52ns/i) Calculating ------------------------------------- check_until(reg) 9.371M i/s - 22.939M times in 2.447900s (106.71ns/i) check_until(str) 22.760M i/s - 39.225M times in 1.723414s (43.94ns/i) exist?(reg) 11.758M i/s - 26.185M times in 2.226997s (85.05ns/i) exist?(str) 34.564M i/s - 61.827M times in 1.788749s (28.93ns/i) Comparison: exist?(str): 34564306.2 i/s check_until(str): 22759878.4 i/s - 1.52x slower exist?(reg): 11757927.4 i/s - 2.94x slower check_until(reg): 9371009.3 i/s - 3.69x slower ```

naitoh · 2024-11-03T14:27:02Z

Do we need to compare check and check_until (match? and exist?)?

It does not seem necessary to compare check and check_until.
I split it into two files.

I think that JRuby's unstable results show a different problem. We may need to use more long target string (and/or pattern) to make the target operations the main operation in the benchmark.

This one has been left as is because I could not find a good solution.

kou · 2024-11-03T21:21:15Z

Thanks.

(ruby/strscan#111) # Why? To improve the parsing process, I would like to add benchmarks for all parsing processes. ## scan - scan_full(regexp, false, true) == StringScanner#check - scan_full(regexp, false, false) == StringScanner#match? ### CRuby ``` $ benchmark-driver benchmark/scan.yaml Warming up -------------------------------------- check(reg) 10.558M i/s - 10.848M times in 1.027445s (94.71ns/i) check(str) 13.368M i/s - 13.782M times in 1.030978s (74.80ns/i) match?(reg) 16.080M i/s - 16.247M times in 1.010340s (62.19ns/i) match?(str) 23.336M i/s - 23.501M times in 1.007088s (42.85ns/i) Calculating ------------------------------------- check(reg) 11.601M i/s - 31.675M times in 2.730287s (86.20ns/i) check(str) 15.217M i/s - 40.104M times in 2.635475s (65.72ns/i) match?(reg) 18.781M i/s - 48.241M times in 2.568662s (53.25ns/i) match?(str) 29.441M i/s - 70.007M times in 2.377840s (33.97ns/i) Comparison: match?(str): 29441324.5 i/s match?(reg): 18780543.7 i/s - 1.57x slower check(str): 15217130.1 i/s - 1.93x slower check(reg): 11601371.2 i/s - 2.54x slower ``` ### JRuby ``` $ benchmark-driver benchmark/scan.yaml Warming up -------------------------------------- check(reg) 8.129M i/s - 8.090M times in 0.995222s (123.02ns/i) check(str) 16.691M i/s - 16.616M times in 0.995519s (59.91ns/i) match?(reg) 8.979M i/s - 9.001M times in 1.002440s (111.37ns/i) match?(str) 26.138M i/s - 26.011M times in 0.995150s (38.26ns/i) Calculating ------------------------------------- check(reg) 11.808M i/s - 24.387M times in 2.065238s (84.69ns/i) check(str) 31.762M i/s - 50.072M times in 1.576495s (31.48ns/i) match?(reg) 13.944M i/s - 26.936M times in 1.931719s (71.71ns/i) match?(str) 50.872M i/s - 78.414M times in 1.541392s (19.66ns/i) Comparison: match?(str): 50872250.2 i/s check(str): 31761544.3 i/s - 1.60x slower match?(reg): 13944219.6 i/s - 3.65x slower check(reg): 11808244.1 i/s - 4.31x slower ``` ## search - search_full(regexp, false, true) == StringScanner#check_until - search_full(regexp, false, false) == StringScanner#exist? ``` $ benchmark-driver benchmark/search.yaml Warming up -------------------------------------- check_until(reg) 9.338M i/s - 9.456M times in 1.012573s (107.09ns/i) check_until(str) 11.385M i/s - 11.979M times in 1.052173s (87.83ns/i) exist?(reg) 13.416M i/s - 13.517M times in 1.007532s (74.54ns/i) exist?(str) 17.976M i/s - 18.677M times in 1.038981s (55.63ns/i) Calculating ------------------------------------- check_until(reg) 10.297M i/s - 28.015M times in 2.720634s (97.11ns/i) check_until(str) 12.684M i/s - 34.156M times in 2.692853s (78.84ns/i) exist?(reg) 15.184M i/s - 40.249M times in 2.650786s (65.86ns/i) exist?(str) 21.426M i/s - 53.928M times in 2.517008s (46.67ns/i) Comparison: exist?(str): 21425527.1 i/s exist?(reg): 15183679.9 i/s - 1.41x slower check_until(str): 12684053.7 i/s - 1.69x slower check_until(reg): 10297134.8 i/s - 2.08x slower ``` ### JRuby ``` $ benchmark-driver benchmark/search.yaml Warming up -------------------------------------- check_until(reg) 7.646M i/s - 7.649M times in 1.000381s (130.78ns/i) check_until(str) 13.075M i/s - 13.010M times in 0.995048s (76.48ns/i) exist?(reg) 8.728M i/s - 8.684M times in 0.994921s (114.57ns/i) exist?(str) 20.609M i/s - 20.514M times in 0.995399s (48.52ns/i) Calculating ------------------------------------- check_until(reg) 9.371M i/s - 22.939M times in 2.447900s (106.71ns/i) check_until(str) 22.760M i/s - 39.225M times in 1.723414s (43.94ns/i) exist?(reg) 11.758M i/s - 26.185M times in 2.226997s (85.05ns/i) exist?(str) 34.564M i/s - 61.827M times in 1.788749s (28.93ns/i) Comparison: exist?(str): 34564306.2 i/s check_until(str): 22759878.4 i/s - 1.52x slower exist?(reg): 11757927.4 i/s - 2.94x slower check_until(reg): 9371009.3 i/s - 3.69x slower ``` ruby/strscan@81a80a176b

…ntil(string)` (#226) ## Why? `StringScanner#check_until(string)` is faster than `StringScanner#check_until(regex)`. See: - ruby/strscan#106 - ruby/strscan#111 ## Benchmark ``` RUBYLIB= BUNDLER_ORIG_RUBYLIB= /Users/naitoh/.rbenv/versions/3.3.4/bin/ruby -v -S benchmark-driver /Users/naitoh/ghq/github.com/naitoh/rexml/benchmark/parse.yaml ruby 3.3.4 (2024-07-09 revision be1089c8ec) [arm64-darwin22] Calculating ------------------------------------- before after before(YJIT) after(YJIT) dom 19.459 19.840 35.035 35.786 i/s - 100.000 times in 5.139034s 5.040369s 2.854304s 2.794367s sax 30.057 30.026 52.986 53.716 i/s - 100.000 times in 3.326998s 3.330499s 1.887303s 1.861652s pull 33.777 34.415 62.294 64.020 i/s - 100.000 times in 2.960622s 2.905668s 1.605284s 1.562002s stream 33.789 34.003 60.174 60.411 i/s - 100.000 times in 2.959521s 2.940916s 1.661845s 1.655334s Comparison: dom after(YJIT): 35.8 i/s before(YJIT): 35.0 i/s - 1.02x slower after: 19.8 i/s - 1.80x slower before: 19.5 i/s - 1.84x slower sax after(YJIT): 53.7 i/s before(YJIT): 53.0 i/s - 1.01x slower before: 30.1 i/s - 1.79x slower after: 30.0 i/s - 1.79x slower pull after(YJIT): 64.0 i/s before(YJIT): 62.3 i/s - 1.03x slower after: 34.4 i/s - 1.86x slower before: 33.8 i/s - 1.90x slower stream after(YJIT): 60.4 i/s before(YJIT): 60.2 i/s - 1.00x slower after: 34.0 i/s - 1.78x slower before: 33.8 i/s - 1.79x slower ``` - YJIT=ON : 1.00x - 1.03x faster - YJIT=OFF : 1.00x - 1.02x faster

naitoh mentioned this pull request Oct 21, 2024
Add IOSource#match? method ruby/rexml#216
Merged

naitoh force-pushed the add_full_benchmark branch from abb447d to 056c258Compare October 22, 2024 23:38

naitoh changed the title ~~Add full benchmark~~Add scan and search benchmarkOct 22, 2024

naitoh force-pushed the add_full_benchmark branch from 056c258 to 085505dCompare November 3, 2024 14:21

kou merged commit 81a80a1 into ruby:masterNov 3, 2024

naitoh deleted the add_full_benchmark branch November 4, 2024 06:49

naitoh mentioned this pull request Dec 15, 2024
Optimize IOSource#read_until method by using StringScanner#check_until(string)ruby/rexml#226
Merged

Navigation Menu

Uh oh!

Add scan and search benchmark - #111

Add scan and search benchmark#111
kou merged 1 commit into
ruby:masterfrom
naitoh:add_full_benchmark

naitoh commented Oct 21, 2024•
edited
Loading

Uh oh!

kou commented Oct 22, 2024

Uh oh!

naitoh commented Oct 22, 2024

Uh oh!

kou commented Oct 22, 2024

Uh oh!

naitoh commented Oct 22, 2024

Uh oh!

kou commented Oct 23, 2024

Uh oh!

naitoh commented Nov 3, 2024•
edited
Loading

Uh oh!

kou commented Nov 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

naitoh commented Oct 21, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why?

scan

CRuby

JRuby

search

JRuby

Uh oh!

kou commented Oct 22, 2024

Uh oh!

naitoh commented Oct 22, 2024

Uh oh!

kou commented Oct 22, 2024

Uh oh!

naitoh commented Oct 22, 2024

JRuby

Uh oh!

kou commented Oct 23, 2024

Uh oh!

naitoh commented Nov 3, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou commented Nov 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

naitoh commented Oct 21, 2024•
edited
Loading

naitoh commented Nov 3, 2024•
edited
Loading