Skip to content

A lease amendment chain answers as-of questions - #683

Merged
WaylandYang merged 2 commits into
devfrom
bench/a-lease-chain-answers-as-of-questions
Sep 13, 2026
Merged

WaylandYang merged 2 commits into
devfrom
bench/a-lease-chain-answers-as-of-questions

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

Adds the lease-chain bench used to find and measure #679 and #680. It is scripts only; no filings are committed, the same as the NVIDIA corpus.

The recall bench (recall.mjs) asks whether a fact reached the graph. This one asks whether, after a contract is rewritten by a chain of amendments, its terms answer correctly at a moment. The Blackbaud HQ lease's Phase 2 exercise deadline was extended five times in 2020 before the tenth amendment tied it to a trigger date. The property was sold to a Blackbaud affiliate that became the landlord, and the price was cut after inspection. There are 18 questions about those values on particular dates.

What is added, under scripts/bench/corpora/contracts/

  • edgar-manifest.json lists 7 contract chains and 62 EDGAR filings. For each it records CIK, accession, filing date, role, and a title read from the document's opening.
  • fetch.py fetches them into edgar/, as HTML plus plain text. It uses curl for the local proxy and needs SEC_USER_AGENT. The redacted CSG/Comcast chain is fetched only with --with-redacted.
  • lease_bench.py setup creates a fresh base and turns off the switches that edit the graph after extraction, as recall.mjs does. It declares a generic lease and property-sale ontology, with single-valued terms marked functional, and ingests the 13 amendments (the 145-chunk base lease only with --with-base). It logs in like recall.mjs: BENCH_BASE, and BENCH_TOKEN or BENCH_EMAIL/BENCH_PASSWORD.
  • lease_bench.py score answers the questions the way the product reads. It takes each lease and purchase-agreement entity's facts and keeps those whose server-computed holds_from/holds_to contains the moment. A question is right when the values holding then are exactly the true one.
  • README.md covers how to run it, the truth table, and the 2026-09-13 runs.
  • .gitignore ignores the fetched edgar/, cuad/ and the per-run lease-kb-*.json.

A run is one fresh base and takes about 10 minutes. Runs vary by about 3 questions, so a build needs at least two.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: WaylandYang <wayland0916@gmail.com>
@WaylandYang
WaylandYang merged commit 2250383 into dev Sep 13, 2026
4 checks passed
@WaylandYang
WaylandYang deleted the bench/a-lease-chain-answers-as-of-questions branch September 13, 2026 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant