Skip to content

Fill pooled databases in the background, and disable page verification - #1044

Merged
SimonCropp merged 4 commits into
mainfrom
pooled-background-fill
Sep 1, 2026
Merged

Fill pooled databases in the background, and disable page verification#1044
SimonCropp merged 4 commits into
mainfrom
pooled-background-fill

Conversation

@SimonCropp

Copy link
Copy Markdown
Owner

Two changes to the pooled database path, plus three measurements that argued against changing anything else.

Fill the pooled databases in the background

CreatePool built all PoolSize databases before returning, so the first pooled test paid every file copy and attach up front, and any test starting concurrently queued behind poolLock for the whole build. PoolSize defaults to ProcessorCount, so that is 16-32 serial database creations before the first test runs.

Only the first database is now awaited. The rest fill in on a background task while tests run, so the wait overlaps with useful work. The lease semaphore starts empty and gains a permit as each database lands, which changes a lease wait from "every pooled database is leased" to "every database built so far is leased".

The fill stays serial deliberately: it now runs alongside live tests, and PoolSize concurrent file copies would just compete with them for the same disk.

A background failure cannot hang the pool. FillPool records the cause and releases one permit per unbuilt database, so every parked waiter wakes, fails the dequeue, puts its permit back for the next caller, and rethrows the cause. Leases arriving afterwards fail fast on the same check. Teardown waits for the fill, since DeleteInstance deletes the directory a background copy may still be writing into.

PooledFillFailureTests covers that failure path by faulting the fill while a second lease is parked on the semaphore. It is mutation checked: with the wake-up release commented out it fails at the 30s timeout, and passes in 8s with it restored.

Disable page verification on the template

page_verify defaulted to checksum, so the engine computed a checksum over every 8KB page as it was written to disk, and re-verified it on read, to detect storage corrupting a page underneath it. These databases are built from a template and deleted with the run, so that signal is never read.

Set on the template, which persists it through detach/copy/attach into pooled, per-test and shared databases alike. Unlike auto_close, which the attach resets to the model default, page_verify survives - verified against all three database kinds.

PageVerifyBenchmarks measures it: a checkpoint-heavy insert of ~7500 data pages is ~5% faster on average, and none won all five trials, though between-run variance on the test machine is comparable to the effect. Bytes written are identical either way, so the saving is CPU, not I/O. Worth taking mainly because the thing given up is worthless here, not because the win is large.

Measured and deliberately not applied

Three further settings were investigated. None resulted in a change, and the benchmarks are kept as the record so they are not re-litigated from scratch.

read_committed_snapshot off for pooled - the biggest win of the lot, and refused. A pooled database is leased exclusively, so its row versioning buys nothing: turning it off is worth ~11% wall clock, ~29% less write I/O and ~38% less read I/O, consistently across three runs. But RCSI is what lets a second connection to a leased database read while the lease holds an uncommitted write, and that is public API - OpenNewConnection, NewConnectionOwnedDbContext and IDbContextFactory.CreateDbContext all open one. Probed both ways against a real pooled lease mid-transaction: with RCSI on the read returns the last committed state, with it off it blocks until the command timeout. Not worth trading a documented API for 11%, especially as it would surface as a mysterious timeout.

accelerated_database_recovery - unusable. LocalDB is Express edition (EngineEdition 4) and enabling it fails with error 12128, regardless of syntax or connection.

Substitutes for ADR - none worth having, and none needed. RollbackCostBenchmarks sizes the gap ADR would close: a ~85us floor plus ~0.94us per modified row, so a test touching 10-100 rows spends under 0.2ms on rollback. Database snapshots do work on LocalDB and are the one available mechanism with ADR's shape, but ResetStrategyBenchmarks shows a revert is 370x slower than the rollback at 100 rows and 11x slower at 10000 - its cost is almost entirely fixed, so the gap narrows with write size but never closes at any volume a test reaches.

Verification

Full suite green, 0 failures:

ProjectTests
LocalDb.Tests126 passed, 2 skipped
EfLocalDb.Tests112
EfLocalDb.Xunit.V3.Tests32
EfLocalDb.NUnit.Tests32
EfLocalDb.TUnit.Tests32
EfLocalDb.MSTest.Tests31
EfClassicLocalDb.Tests20

Solution builds clean across net8.0/net9.0/net10.0/net48 with TreatWarningsAsErrors. Pooled tests additionally run green at LocalDBPoolSize of 1, 2 and 8 - size 1 exercising the case where the fill loop never runs, size 2 putting four concurrent tests against two databases while the fill is still in flight.

CreatePool built all PoolSize databases before returning, so the first
pooled test paid every file copy and attach up front, and any test that
started concurrently queued behind poolLock for the whole build. PoolSize
defaults to ProcessorCount, so that is 16-32 serial database creations
before the first test runs.
Only the first database is now awaited. The rest fill in on a background
task while tests run, so the wait overlaps with useful work. The lease
semaphore starts empty and gains a permit as each database lands, which
changes a lease wait from "every pooled database is leased" to "every
database built so far is leased".
The fill stays serial: it now runs alongside live tests, and PoolSize
concurrent file copies would just compete with them for the same disk.
A background failure cannot hang the pool. FillPool records the cause and
releases one permit per unbuilt database, so every parked waiter wakes,
fails the dequeue, puts its permit back for the next caller, and rethrows
the cause. Leases arriving afterwards fail fast on the same check. Also
dispose the connection in AddPooled on the failure path, which only began
to matter once a failure could happen off the caller's stack.
Teardown waits for the fill, since DeleteInstance deletes the directory a
background copy may still be writing into.
page_verify defaulted to checksum, so the engine computed a checksum over
every 8KB page as it was written to disk and re-verified it on read, to
detect storage corrupting a page underneath it. These databases are built
from the template and deleted with the run, so that signal is never read.
Set on the template, which persists it through detach/copy/attach into
pooled, per-test and shared databases alike. Unlike auto_close, which the
attach resets to the model default, page_verify survives.
PageVerifyBenchmarks measures it: a checkpoint-heavy insert of ~7500 data
pages is ~5% faster on average, and none won all five trials, though the
between-run variance on the test machine is comparable to the effect.
Bytes written are identical either way, so the saving is CPU, not I/O.
A pooled database is leased exclusively, so the row versioning RCSI pays
for cannot buy anything there. Measured against the write-then-roll-back
shape a pooled test has, turning it off is worth ~11% wall clock, ~29%
less write I/O, ~38% less read I/O and ~44% less memory, consistently
across three runs.
Not applied. RCSI is what lets a second connection to a leased database
read while the lease holds an uncommitted write, and that is public API:
OpenNewConnection, NewConnectionOwnedDbContext and IDbContextFactory
.CreateDbContext all open one. Probed both ways against a pooled lease
mid-transaction: with RCSI on the read returns the last committed state,
with it off it blocks on the lease's X locks until the command timeout.
Trading a documented API for 11% is not worth it, so the benchmark is
kept as the record of the trade rather than the change.
accelerated_database_recovery was the other candidate, since every lease
ends in a rollback. It cannot be used at all: LocalDB is Express edition
and enabling it fails with error 12128.
ADR is unavailable on LocalDB, so measure what its absence actually costs
before looking for a substitute. RollbackCostBenchmarks isolates the
rollback from the write: a fixed floor of ~85us plus ~0.94us per modified
row, linear as expected for log-walking undo. At the volumes a test
writes that is nothing - ~93us for 10 rows, ~161us for 100 - and only
reaches ~9.5ms at 10000 rows.
Database snapshots, unlike ADR, do work on LocalDB, and a revert is the
one available mechanism with ADR's shape: page-level undo against a
sparse file rather than a walk backwards through the log. The snapshot
also survives the revert, so one baseline can be reverted to repeatedly.
ResetStrategyBenchmarks compares it against the transaction rollback end
to end, including the reconnect a revert needs for exclusive access.
Revert loses badly: 172ms against 0.47ms at 100 rows, 307ms against 27ms
at 10000. Its cost is almost entirely fixed, so the gap narrows as the
write grows but never closes at any volume a test reaches. Rolling back
the lease transaction stays the right mechanism; both benchmarks are kept
as the record.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@SimonCropp