Traverse syntax eagerly in syntax-search-everything

[jackfirth]

syntax-search-everything builds a lazy stream, layering stream-lazy, stream-append and a generated syntax-parse search class over every node. It runs on every syntax object the expander visits, and two of its three callers consume the whole traversal, so the suspension machinery costs far more than the list it avoids allocating.

Building the list directly makes the traversal itself 3–5x faster at every input size, measured in isolation over all 9,262 subforms of xrepl.rkt, grouped by node count:

nodes per call	lazy stream	eager list	generator
<=3 (6918 calls)	1.25 us	0.57 us	1.18 us
4-10 (1422 calls)	9.84 us	2.57 us	4.42 us
11-50 (730 calls)	40.92 us	8.79 us	14.42 us
>50 (192 calls)	501.91 us	90.43 us	142.36 us

I can't resolve the end-to-end effect above noise, and I'd rather say so than quote a number. Analysing xrepl.rkt measures ~67-69s on this branch against ~71-72s on master, but master itself has ranged 67.9-72.0s across this session on identical code. Interleaved A/B pairs favoured this branch both times, but the effect and the noise are the same order of magnitude. An earlier version of this description claimed ~78s → ~74s; that was a single hot-machine reading and I don't stand behind it.

I also tried a generator instead of a list (third column). It's much better than the lazy stream and keeps O(1) early exit, but it's ~1.6x slower than the list everywhere and roughly a wash with the stream on tiny forms — which are 75% of the calls — so it showed no end-to-end benefit.

One open question worth your call:syntax-completely-original? in syntax-replacement.rkt uses for/and, which short-circuits on the first non-original subform. Going eager removes that early exit, and I haven't measured how often it fires. If it fires often on large syntax, a generator would be the better trade there even though it's worse for the hot analysis.rkt caller.

raco test -p resyntax passes. Given the unresolved end-to-end picture, this one is much weaker than #840/#842 and I won't be offended if it's not worth the churn.

[github-actions]

Resyntax analyzed 1 file in this pull request and found no issues.

[github-actions]

Docs preview: https://resyntax.notjack.space/preview/843/

[coveralls]

Coverage is 96.589% — perf/eager-search-everything into master. No base build found for master.

[jackfirth]

syntax-search-everything builds a lazy stream, layering stream-lazy, stream-append and a generated syntax-parse search class over every node. It runs on every syntax object the expander visits, and two of its three callers consume the whole traversal, so the suspension machinery costs far more than the list it avoids allocating.

Building the list directly makes the traversal itself 3–5x faster at every input size, measured in isolation over all 9,262 subforms of xrepl.rkt, grouped by node count:

nodes per call	lazy stream	eager list	generator
<=3 (6918 calls)	1.25 us	0.57 us	1.18 us
4-10 (1422 calls)	9.84 us	2.57 us	4.42 us
11-50 (730 calls)	40.92 us	8.79 us	14.42 us
>50 (192 calls)	501.91 us	90.43 us	142.36 us

I can't resolve the end-to-end effect above noise, and I'd rather say so than quote a number. Analysing xrepl.rkt measures ~67-69s on this branch against ~71-72s on master, but master itself has ranged 67.9-72.0s across this session on identical code. Interleaved A/B pairs favoured this branch both times, but the effect and the noise are the same order of magnitude. An earlier version of this description claimed ~78s → ~74s; that was a single hot-machine reading and I don't stand behind it.

I also tried a generator instead of a list (third column). It's much better than the lazy stream and keeps O(1) early exit, but it's ~1.6x slower than the list everywhere and roughly a wash with the stream on tiny forms — which are 75% of the calls — so it showed no end-to-end benefit.

One open question worth your call:syntax-completely-original? in syntax-replacement.rkt uses for/and, which short-circuits on the first non-original subform. Going eager removes that early exit, and I haven't measured how often it fires. If it fires often on large syntax, a generator would be the better trade there even though it's worse for the hot analysis.rkt caller.

raco test -p resyntax passes. Given the unresolved end-to-end picture, this one is much weaker than #840/#842 and I won't be offended if it's not worth the churn.

[github-actions]

Resyntax analyzed 1 file in this pull request and found no issues.

[github-actions]

Docs preview: https://resyntax.notjack.space/preview/843/

[coveralls]

Coverage is 96.589% — perf/eager-search-everything into master. No base build found for master.

[jackfirth]

syntax-search-everything builds a lazy stream, layering stream-lazy, stream-append and a generated syntax-parse search class over every node. It runs on every syntax object the expander visits, and two of its three callers consume the whole traversal, so the suspension machinery costs far more than the list it avoids allocating.

Building the list directly makes the traversal itself 3–5x faster at every input size, measured in isolation over all 9,262 subforms of xrepl.rkt, grouped by node count:

nodes per call	lazy stream	eager list	generator
<=3 (6918 calls)	1.25 us	0.57 us	1.18 us
4-10 (1422 calls)	9.84 us	2.57 us	4.42 us
11-50 (730 calls)	40.92 us	8.79 us	14.42 us
>50 (192 calls)	501.91 us	90.43 us	142.36 us

I can't resolve the end-to-end effect above noise, and I'd rather say so than quote a number. Analysing xrepl.rkt measures ~67-69s on this branch against ~71-72s on master, but master itself has ranged 67.9-72.0s across this session on identical code. Interleaved A/B pairs favoured this branch both times, but the effect and the noise are the same order of magnitude. An earlier version of this description claimed ~78s → ~74s; that was a single hot-machine reading and I don't stand behind it.

I also tried a generator instead of a list (third column). It's much better than the lazy stream and keeps O(1) early exit, but it's ~1.6x slower than the list everywhere and roughly a wash with the stream on tiny forms — which are 75% of the calls — so it showed no end-to-end benefit.

One open question worth your call:syntax-completely-original? in syntax-replacement.rkt uses for/and, which short-circuits on the first non-original subform. Going eager removes that early exit, and I haven't measured how often it fires. If it fires often on large syntax, a generator would be the better trade there even though it's worse for the hot analysis.rkt caller.

raco test -p resyntax passes. Given the unresolved end-to-end picture, this one is much weaker than #840/#842 and I won't be offended if it's not worth the churn.

[github-actions]

Resyntax analyzed 1 file in this pull request and found no issues.

[github-actions]

Docs preview: https://resyntax.notjack.space/preview/843/

[coveralls]

Coverage is 96.589% — perf/eager-search-everything into master. No base build found for master.

[jackfirth]

syntax-search-everything builds a lazy stream, layering stream-lazy, stream-append and a generated syntax-parse search class over every node. It runs on every syntax object the expander visits, and two of its three callers consume the whole traversal, so the suspension machinery costs far more than the list it avoids allocating.

Building the list directly makes the traversal itself 3–5x faster at every input size, measured in isolation over all 9,262 subforms of xrepl.rkt, grouped by node count:

nodes per call	lazy stream	eager list	generator
<=3 (6918 calls)	1.25 us	0.57 us	1.18 us
4-10 (1422 calls)	9.84 us	2.57 us	4.42 us
11-50 (730 calls)	40.92 us	8.79 us	14.42 us
>50 (192 calls)	501.91 us	90.43 us	142.36 us

I can't resolve the end-to-end effect above noise, and I'd rather say so than quote a number. Analysing xrepl.rkt measures ~67-69s on this branch against ~71-72s on master, but master itself has ranged 67.9-72.0s across this session on identical code. Interleaved A/B pairs favoured this branch both times, but the effect and the noise are the same order of magnitude. An earlier version of this description claimed ~78s → ~74s; that was a single hot-machine reading and I don't stand behind it.

I also tried a generator instead of a list (third column). It's much better than the lazy stream and keeps O(1) early exit, but it's ~1.6x slower than the list everywhere and roughly a wash with the stream on tiny forms — which are 75% of the calls — so it showed no end-to-end benefit.

One open question worth your call:syntax-completely-original? in syntax-replacement.rkt uses for/and, which short-circuits on the first non-original subform. Going eager removes that early exit, and I haven't measured how often it fires. If it fires often on large syntax, a generator would be the better trade there even though it's worse for the hot analysis.rkt caller.

raco test -p resyntax passes. Given the unresolved end-to-end picture, this one is much weaker than #840/#842 and I won't be offended if it's not worth the churn.

[github-actions]

Resyntax analyzed 1 file in this pull request and found no issues.

[github-actions]

Docs preview: https://resyntax.notjack.space/preview/843/

[coveralls]

Coverage is 96.589% — perf/eager-search-everything into master. No base build found for master.

[jackfirth]

syntax-search-everything builds a lazy stream, layering stream-lazy, stream-append and a generated syntax-parse search class over every node. It runs on every syntax object the expander visits, and two of its three callers consume the whole traversal, so the suspension machinery costs far more than the list it avoids allocating.

Building the list directly makes the traversal itself 3–5x faster at every input size, measured in isolation over all 9,262 subforms of xrepl.rkt, grouped by node count:

nodes per call	lazy stream	eager list	generator
<=3 (6918 calls)	1.25 us	0.57 us	1.18 us
4-10 (1422 calls)	9.84 us	2.57 us	4.42 us
11-50 (730 calls)	40.92 us	8.79 us	14.42 us
>50 (192 calls)	501.91 us	90.43 us	142.36 us

I can't resolve the end-to-end effect above noise, and I'd rather say so than quote a number. Analysing xrepl.rkt measures ~67-69s on this branch against ~71-72s on master, but master itself has ranged 67.9-72.0s across this session on identical code. Interleaved A/B pairs favoured this branch both times, but the effect and the noise are the same order of magnitude. An earlier version of this description claimed ~78s → ~74s; that was a single hot-machine reading and I don't stand behind it.

I also tried a generator instead of a list (third column). It's much better than the lazy stream and keeps O(1) early exit, but it's ~1.6x slower than the list everywhere and roughly a wash with the stream on tiny forms — which are 75% of the calls — so it showed no end-to-end benefit.

One open question worth your call:syntax-completely-original? in syntax-replacement.rkt uses for/and, which short-circuits on the first non-original subform. Going eager removes that early exit, and I haven't measured how often it fires. If it fires often on large syntax, a generator would be the better trade there even though it's worse for the hot analysis.rkt caller.

raco test -p resyntax passes. Given the unresolved end-to-end picture, this one is much weaker than #840/#842 and I won't be offended if it's not worth the churn.

[github-actions]

Resyntax analyzed 1 file in this pull request and found no issues.

[github-actions]

Docs preview: https://resyntax.notjack.space/preview/843/

[coveralls]

Coverage is 96.589% — perf/eager-search-everything into master. No base build found for master.

[jackfirth]

syntax-search-everything builds a lazy stream, layering stream-lazy, stream-append and a generated syntax-parse search class over every node. It runs on every syntax object the expander visits, and two of its three callers consume the whole traversal, so the suspension machinery costs far more than the list it avoids allocating.

Building the list directly makes the traversal itself 3–5x faster at every input size, measured in isolation over all 9,262 subforms of xrepl.rkt, grouped by node count:

nodes per call	lazy stream	eager list	generator
<=3 (6918 calls)	1.25 us	0.57 us	1.18 us
4-10 (1422 calls)	9.84 us	2.57 us	4.42 us
11-50 (730 calls)	40.92 us	8.79 us	14.42 us
>50 (192 calls)	501.91 us	90.43 us	142.36 us

I can't resolve the end-to-end effect above noise, and I'd rather say so than quote a number. Analysing xrepl.rkt measures ~67-69s on this branch against ~71-72s on master, but master itself has ranged 67.9-72.0s across this session on identical code. Interleaved A/B pairs favoured this branch both times, but the effect and the noise are the same order of magnitude. An earlier version of this description claimed ~78s → ~74s; that was a single hot-machine reading and I don't stand behind it.

I also tried a generator instead of a list (third column). It's much better than the lazy stream and keeps O(1) early exit, but it's ~1.6x slower than the list everywhere and roughly a wash with the stream on tiny forms — which are 75% of the calls — so it showed no end-to-end benefit.

One open question worth your call:syntax-completely-original? in syntax-replacement.rkt uses for/and, which short-circuits on the first non-original subform. Going eager removes that early exit, and I haven't measured how often it fires. If it fires often on large syntax, a generator would be the better trade there even though it's worse for the hot analysis.rkt caller.

raco test -p resyntax passes. Given the unresolved end-to-end picture, this one is much weaker than #840/#842 and I won't be offended if it's not worth the churn.

[github-actions]

Resyntax analyzed 1 file in this pull request and found no issues.

[github-actions]

Docs preview: https://resyntax.notjack.space/preview/843/

[coveralls]

Coverage is 96.589% — perf/eager-search-everything into master. No base build found for master.

[jackfirth]

syntax-search-everything builds a lazy stream, layering stream-lazy, stream-append and a generated syntax-parse search class over every node. It runs on every syntax object the expander visits, and two of its three callers consume the whole traversal, so the suspension machinery costs far more than the list it avoids allocating.

Building the list directly makes the traversal itself 3–5x faster at every input size, measured in isolation over all 9,262 subforms of xrepl.rkt, grouped by node count:

nodes per call	lazy stream	eager list	generator
<=3 (6918 calls)	1.25 us	0.57 us	1.18 us
4-10 (1422 calls)	9.84 us	2.57 us	4.42 us
11-50 (730 calls)	40.92 us	8.79 us	14.42 us
>50 (192 calls)	501.91 us	90.43 us	142.36 us

I can't resolve the end-to-end effect above noise, and I'd rather say so than quote a number. Analysing xrepl.rkt measures ~67-69s on this branch against ~71-72s on master, but master itself has ranged 67.9-72.0s across this session on identical code. Interleaved A/B pairs favoured this branch both times, but the effect and the noise are the same order of magnitude. An earlier version of this description claimed ~78s → ~74s; that was a single hot-machine reading and I don't stand behind it.

I also tried a generator instead of a list (third column). It's much better than the lazy stream and keeps O(1) early exit, but it's ~1.6x slower than the list everywhere and roughly a wash with the stream on tiny forms — which are 75% of the calls — so it showed no end-to-end benefit.

One open question worth your call:syntax-completely-original? in syntax-replacement.rkt uses for/and, which short-circuits on the first non-original subform. Going eager removes that early exit, and I haven't measured how often it fires. If it fires often on large syntax, a generator would be the better trade there even though it's worse for the hot analysis.rkt caller.

raco test -p resyntax passes. Given the unresolved end-to-end picture, this one is much weaker than #840/#842 and I won't be offended if it's not worth the churn.

[github-actions]

Resyntax analyzed 1 file in this pull request and found no issues.

[github-actions]

Docs preview: https://resyntax.notjack.space/preview/843/

[coveralls]

Coverage is 96.589% — perf/eager-search-everything into master. No base build found for master.

[jackfirth]

syntax-search-everything builds a lazy stream, layering stream-lazy, stream-append and a generated syntax-parse search class over every node. It runs on every syntax object the expander visits, and two of its three callers consume the whole traversal, so the suspension machinery costs far more than the list it avoids allocating.

Building the list directly makes the traversal itself 3–5x faster at every input size, measured in isolation over all 9,262 subforms of xrepl.rkt, grouped by node count:

nodes per call	lazy stream	eager list	generator
<=3 (6918 calls)	1.25 us	0.57 us	1.18 us
4-10 (1422 calls)	9.84 us	2.57 us	4.42 us
11-50 (730 calls)	40.92 us	8.79 us	14.42 us
>50 (192 calls)	501.91 us	90.43 us	142.36 us

I can't resolve the end-to-end effect above noise, and I'd rather say so than quote a number. Analysing xrepl.rkt measures ~67-69s on this branch against ~71-72s on master, but master itself has ranged 67.9-72.0s across this session on identical code. Interleaved A/B pairs favoured this branch both times, but the effect and the noise are the same order of magnitude. An earlier version of this description claimed ~78s → ~74s; that was a single hot-machine reading and I don't stand behind it.

I also tried a generator instead of a list (third column). It's much better than the lazy stream and keeps O(1) early exit, but it's ~1.6x slower than the list everywhere and roughly a wash with the stream on tiny forms — which are 75% of the calls — so it showed no end-to-end benefit.

One open question worth your call:syntax-completely-original? in syntax-replacement.rkt uses for/and, which short-circuits on the first non-original subform. Going eager removes that early exit, and I haven't measured how often it fires. If it fires often on large syntax, a generator would be the better trade there even though it's worse for the hot analysis.rkt caller.

raco test -p resyntax passes. Given the unresolved end-to-end picture, this one is much weaker than #840/#842 and I won't be offended if it's not worth the churn.

[github-actions]

Resyntax analyzed 1 file in this pull request and found no issues.

[github-actions]

Docs preview: https://resyntax.notjack.space/preview/843/

[coveralls]

Coverage is 96.589% — perf/eager-search-everything into master. No base build found for master.