Uh oh!
There was an error while loading. Please reload this page.
feat(examples): a complete optimization method for a game - #39
Merged
Conversation
`compareOptimizationMethods` takes methods, not loops. It runs each on disjoint
train, selection and final-test scenarios and reports every one's lift over the
baseline with a simultaneous, Bonferroni-adjusted bootstrap interval, so the
question it answers is whether one way of SEARCHING beats another by more than
the game's own noise.
`bestOfN` is the simplest honest method. It authors N candidate programs,
scores each on the selection partition, and keeps the best. The final-test
partition is never read, which is the one rule a method may not break: choosing
a winner on the test data reports a lift it did not earn.
Spend is stated through `CostProvenance`, which is a union that refuses to call
a partial sum observed. One candidate that billed a plan and reported no
per-request figure makes the whole total `{kind: 'uncaptured', usd: null}` with
the reason recorded, rather than summing it as zero and making the arm nobody
could meter the cheapest one.
Authoring a candidate is supplied by the caller, because building a player is
the expensive half that carries credentials, and a method should not decide how
an agent is run.Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
compareOptimizationMethodstakes methods, not loops. It runs each on disjoint train, selection and final-test scenarios and reports every method's lift over the baseline with a simultaneous, Bonferroni-adjusted bootstrap interval. The question it answers is whether one way of searching beats another by more than the game's own noise.bestOfNis the simplest honest method: author N candidate programs, score each on the selection partition, keep the best. The final-test partition is never read — choosing a winner on test data reports a lift it did not earn, and it is why the comparator hands a method two partitions instead of one.Cost is stated, not assumed
CostProvenanceis a union that structurally refuses to call a partial sum observed. One candidate that billed a plan and reported no per-request figure makes the whole total{kind: 'uncaptured', usd: null}, with the reason recorded. Summing it as zero would make the arm nobody could meter the cheapest one.That is the same rule this repo enforces on a cell's
usd, now enforced by a type rather than by discipline.What running the framework's own statistics revealed
Re-running the earlier three-profile 2048 study through
pairedBootstrap:All three returned
gateEligible: false.BOOTSTRAP_GATE_MIN_Nis 20, and every study in this repo so far has run 1 to 3 replicates. The framework declines to let them gate a decision, which is the correct answer.One methodological note for whoever uses this next:
pairedBootstrappairs by index. The replicates here are not matched — sonnet rep1 and haiku rep1 share nothing but a subscript — so pairing them manufactures precision. That is why its sonnet-vs-haiku interval excludes zero while an unpaired resample of the same numbers does not. UseseedGroupto make replicates genuinely paired before relying on it.Local: 0 typecheck errors, boundary check passes.