Skip to content

Block Parser: Explore a streaming lazy interface - #5705

Draft
dmsnell wants to merge 2 commits into
WordPress:trunkfrom
dmsnell:dmsnell-streaming-lazy-block-parser
Draft

Block Parser: Explore a streaming lazy interface#5705
dmsnell wants to merge 2 commits into
WordPress:trunkfrom
dmsnell:dmsnell-streaming-lazy-block-parser

Conversation

@dmsnell

@dmsnelldmsnell commented Nov 26, 2023

Copy link
Copy Markdown
Member

Augmented but not replaced entirely by the Unified Block Parser in #6381
Alternatively provided by the block-delimiter-finder in #6760

For a 3 MB document which took 5 seconds and 14 GB to parse, this version of the parser parsed it in 27 ms and 20 MB.

Initial testing This version is slower for the home page render of `twentytwentyfour`. While unfortunate, this is not entirely surprising as this was designed to fix the catastrophically bad cases.

lazy-block-parser-slower-for-tt4-home-page

However, in catastrophic cases it's wildly better than trunk. The following was tested for a 15 KB / 400 line chunk of the 3 MB post mentioned above.

lazy-block-parser-faster-for-large-page

The algorithm has wild complexity too. For the same post, including the first 599 lines (only 23 KB of HTML), trunk consumes 520 MB or memory while this branch only consumes 10 MB. With no more than 15 samples the data is extremely significant.

lazy-block-parser-faster-for-very-large-page

Testing Results

This may be slightly slower for a number of normal posts. For the home page render of twentytwentyfour it rendered 3.7 ms slower than trunk. However, for my catastrophically-broken test post, the impact of the lazy parsing is dramatic and significant after only a single request.

The lazy parser is still slow for really pathological cases, but unlike trunk it runs within a mostly bounded memory footprint. The more pathological the post, the more dramatic the improvement in both runtime and memory use becomes. Below is a chart comparing slices of my test file against both parsers. Between each test run the database is reset. The number of lines reported is the count of how many of the original 3 MB document lines were extracted as the test post.

Of particular note is that this lazy parser allows for more control over the performance threshold still. Further expansion would allow setting a time limit, an upper bound on m emory usage, and a content length threshold after which the parser could pause and/or collapse the remainder of the post into a single unparsed block, essentially turning everything after the limit into a chunk of raw HTML (the static fallback render).

Linesmax depthKBtrunk msbranch msΔspeeduptrunk MBbranch MB
(tt4)2592.696.1+3.78%x0.96
400124151,170624-47%x1.88
600187234.82 s968-80%x5.0052010
7922483016.7 s3.53 s-79%x4.731.8 GB14
100031638118 s10.1 s-91%x11.76.5 GB23
1200380468.01 min22 s-95%x2213.9 GB32
79k (all)256833 MB30

With memory_limit=55G on a 60 GB system I was unable to create the post via wp_insert_post() and it failed after some number of tens of minutes.

on this branch the post inserted after a few seconds and used a peak memory of 64 MB

@dmsnell
dmsnellforce-pushed the dmsnell-streaming-lazy-block-parser branch from 2585f14 to 20e8c0bCompareNovember 26, 2023 07:48
@dmsnell
dmsnellforce-pushed the dmsnell-streaming-lazy-block-parser branch from d1671bd to 13c1fb1CompareDecember 7, 2023 18:38
@joemcgill
joemcgill self-requested a review December 8, 2023 15:23
@dmsnell
dmsnellforce-pushed the dmsnell-streaming-lazy-block-parser branch from ab3978b to 7d95983CompareDecember 12, 2023 17:05
@dmsnell
dmsnellforce-pushed the dmsnell-streaming-lazy-block-parser branch 2 times, most recently from 3b38369 to 3e55ebeCompareJanuary 13, 2024 13:09
@dmsnell
dmsnellforce-pushed the dmsnell-streaming-lazy-block-parser branch 3 times, most recently from fed1bab to 2aa6d3eCompareJanuary 30, 2024 21:49
dmsnell added 2 commits June 8, 2024 17:22
For a 3 MB document which took 5 seconds and 14 GB to parse, this version of the parser parsed it in 27 ms and 20 MB.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@dmsnell