Skip to content

OpenClaw backfill reads message fields one level too high, projecting 0 rows - #552

Merged
philcunliffe merged 3 commits into
masterfrom
fix/issue-543
Aug 2, 2026
Merged

OpenClaw backfill reads message fields one level too high, projecting 0 rows#552
philcunliffe merged 3 commits into
masterfrom
fix/issue-543

Conversation

@philcunliffe

@philcunliffephilcunliffe commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Root cause

A real OpenClaw v3 type: "message" record is two levels deep. The record line
states only ['id', 'message', 'parentId', 'timestamp', 'type']; the message
itself (role, content, and on an assistant turn api, model, provider,
stopReason, usage) is nested under message.

parseOpenclawSessionMessage (session_file.js) read model, provider,
api, stopReason, and usage off the record line, so all of them came back
undefined. effectiveProviders then had no stated provider to propagate,
every record resolved to unknown, and PROJECTABLE_PROVIDERS excluded all of
them fail-closed. That is why a real ~/.openclaw scanned files_seen: 7, sessions_projected: 0, records_excluded: 18 and hyp status reported
backfill @hypaware/openclaw [done] (0 rows): a parse miss and an intended
exclusion are indistinguishable at that seam.

The same defect sat one stage later at both consumers, so fixing provider
alone would only have moved the drop:

  • backfill.js projectedMessageFromRecord read record.role / record.content
  • settle.js read message.record.role (rawRole) and message.record.content
    for the LLP 0159 content match key, so a real session settled nothing either

The fix

One envelope-shape change in the LLP 0158 reader, not a per-field patch and not
a loosened exclusion filter:

  • openclawMessageEnvelope(row) states the address once: the nested message
    object is the message envelope, with the record line as the fallback for a
    record that nests none. A field the envelope states is never overridden by a
    same-named field on the line.
  • role and content become normalized fields on OpenclawSessionMessage.
    Their address is the thing that was easy to get wrong, so they belong to the
    one reader rather than to each consumer's own reach into record. record
    stays exposed for genuinely caller-specific fields (parentId, toolCallId).
  • Both consumers now read the normalized fields, so neither can drift a level
    again.

usage needed no change: usageAttributes already accepts OpenClaw's
input / output / cacheRead / cacheWrite spelling.

Fixtures

All three OpenClaw suites now write the real two-level record shape through one
helper each (messageLine in the backfill suite, sessionFileLine in the
settlement suite, literal nested records in the reader suite). The old fixtures
asserted a flat envelope OpenClaw never writes, which is exactly why CI stayed
green through this bug. A new test pins the on-disk bytes against the shape
verified on a live install, so a fixture cannot quietly re-invent the flat one.

LLP 0158 is updated in the same commit: the verified record shape, read rule 5
(the envelope address and why it fails quietly), the "a field two callers must
locate identically is the reader's" consequence, and the path-faithful-fixture
consequence.

Test evidence

The added regression test is
test/plugins/openclaw-backfill.test.js ->
a mixed real-shape session partially projects: anthropic turns land, claude-cli turns stay excluded,
the mixed-session case from the issue: an anthropic turn and a claude-cli
turn in one real-shape file must partially project.

Before (on the fixed fixtures, unfixed source):

TAP version 13
# Subtest: a mixed real-shape session partially projects: anthropic turns land, claude-cli turns stay excluded
not ok 1 - a mixed real-shape session partially projects: anthropic turns land, claude-cli turns stay excluded
---
duration_ms: 7.13412
type: 'test'
location: '.../test/plugins/openclaw-backfill.test.js:544:1'
failureType: 'testCodeFailure'
error: |-
the anthropic half of the session must project
0 !== 1
code: 'ERR_ASSERTION'
name: 'AssertionError'
expected: 1
actual: 0
operator: 'strictEqual'
...
1..1
# tests 1
# pass 0
# fail 1

After:

TAP version 13
# Subtest: a mixed real-shape session partially projects: anthropic turns land, claude-cli turns stay excluded
ok 1 - a mixed real-shape session partially projects: anthropic turns land, claude-cli turns stay excluded
---
duration_ms: 8.39832
type: 'test'
...
1..1
# tests 1
# pass 1
# fail 0

Whole-suite reproduction, on the path-faithful fixtures against the unfixed
reader (this is the bug's blast radius, not one test's):

test/plugins/openclaw-backfill.test.js # tests 21 # pass 6 # fail 15
test/plugins/openclaw-session-file.test.js # tests 20 # pass 16 # fail 4
test/plugins/openclaw-settlement.test.js # tests 16 # pass 7 # fail 9

After the fix, all three are green (21/21, 20/20, 16/16), as are the other six
OpenClaw suites.

Repo gates:

npm test -> # tests 3276 # pass 3275 # fail 0 # skipped 1
npm run typecheck -> clean

No pre-existing failures to disclose: the suite is fully green before and after.

Deliberately not in this PR

The issue's two adjacent decisions are design questions, not this defect, and
neither costs rows today (both parse to nothing rather than to something wrong):

  • whether *.trajectory.jsonl files should be skipped explicitly by the scanner
    rather than silently parsed-and-empty
  • whether probe-anthropic-* / probe-claude-cli-* attach-probe sessions should
    be imported or skipped by session-id prefix

Both want their own short decision LLP rather than a silent filter here.

Fixes#543

testand others added 2 commits July 31, 2026 22:00
A real OpenClaw v3 `type: "message"` record states only `id`, `parentId`,
`timestamp`, and `type` on the record line: `role`, `content`, `model`,
`provider`, `api`, `stopReason`, and `usage` are all nested under `message`.
The LLP 0158 reader read them off the record line, so every field came back
absent, every record resolved to `provider: unknown`, the backfill allowlist
excluded all of them, and `hyp status` reported `backfill @hypaware/openclaw
[done] (0 rows)` for a session it had failed to read. The settlement enricher
was broken the same way one seam later (`record.role`/`record.content`), so a
real session settled nothing either.
The reader now owns the envelope address: fields are read from the nested
`message` object, falling back to the record line for a record that nests
none, and `role`/`content` are normalized fields rather than something each
consumer picks out of the raw record. Both consumers read them off the
normalized message, so neither can drift a level again.
Fixtures across the three OpenClaw suites now write the real two-level shape
through one helper each; the old flat fixtures asserted an envelope OpenClaw
never writes, which is why the suite stayed green through the bug.
LLP 0158 records the verified record shape, the envelope read rule, and the
path-faithful-fixture consequence.
Co-Authored-By: Claude <noreply@anthropic.com>
… is the line's
Follow-up to the #543 envelope fix, from review of PR #552.
- `messageField` fell back on key *absence*, not value *usability*, so a
present-but-unusable nested value permanently masked a good record-line
one. A nested `provider: " "` beside a line-level `provider: "anthropic"`
resolved the record to `unknown` and the allowlist excluded it fail-closed;
a nested `timestamp` that did not parse dropped `message_created_at`, which
re-dates the row to session start, defeats the `--since` window (a
timestamp-less item is kept unconditionally) and puts the settlement
ordinal match outside every window so the turn never dedupes. Rule 3's
present-value test now runs at both levels before the fallback decides.
- `id` is now read line-first, envelope-fallback. LLP 0158 verified message
identity on the record line; envelope-first meant a future OpenClaw that
copied the provider's own id into the nested message would silently repoint
every `message_id` and `part_id`, so committed rows would stop deduping
against new ones and the history would double with nothing raised.
- The OPENCLAW_HOME relocation fixture still wrote the invented flat shape,
bypassing `messageLine`, so it passed with the envelope read reverted. It
now goes through the helper, and the shape pin carries `idempotencyKey` so
it matches the live key list the same file documents.
- Say which LEVEL `record` is: `parentId` is on the line, `idempotencyKey`
and `toolCallId` are at `record.message`. The old wording invited the very
read #543 was.
- LLP 0158 gains rules 6 and 7 for the two behaviors above, and the stale
"no live OpenClaw install was reachable" note on `usageAttributes` is
reconciled with the spelling this work verified.
Co-Authored-By: Claude <noreply@anthropic.com>
@philcunliffe

Copy link
Copy Markdown
ContributorAuthor

Neutral review round: PR #552 @ 44c7080

Verdict: findings (5 actionable, all fixed and pushed as df7f119).

The core change is right and I could not break it. The two-level address is correct, both consumers now read the normalized fields, and there is no remaining raw read of role/content/provider/model anywhere in the repo (readOpenclawSessionMessages has exactly two callers, backfill.js and settle.js, both converted). The allowlist was not weakened: PROJECTABLE_PROVIDERS, effectiveProviders and partitionByBackend are byte-identical to master, claude-cli is still excluded with covered_by: claude_transcript, and unknown still fails closed. The LLP 0159 match key's semantics are unchanged: the stored openclaw.match_key still comes from wireMatchKey at capture, and only the session-file side of the comparison moved, so settlement of already-stored rows is strictly repaired rather than altered.

What the fallback rule does under stress is where the findings are.


1. High: messageField fell back on key absence, not value usability, so a garbage nested value permanently masked a good record-line one

session_file.js:335-338 (pre-fix)

functionmessageField(envelope,row,key){constvalue=envelope[key]returnvalue===undefined ? row[key] : value}

Every other read in the file obeys the reader's own rule 3 ("unconfirmable is unresolvable"): nonBlankString and parseTimestampMs reject a present-but-unusable value. messageField ran before those, so an unusable envelope value consumed the read and the line was never consulted. Verified against the PR head through readOpenclawSessionMessages:

recordresult
line provider:"anthropic", nested provider:" "provider: undefined
line timestamp:"2026-07-30T00:00:00.000Z", nested timestamp:"not-a-date"timestampMs: undefined
line usage:{input:10,output:20}, nested usage:nullusage: undefined

That value is absent (rule 3) and load-bearing at the same time. Downstream, each of those is a silent drop of the same family as #543:

  • lost provider resolves the record to unknown, and the allowlist excludes it fail-closed
  • lost timestampMs omits message_created_at (backfill.js:518-520), so the materializer substitutes conversation_started_at and the row lands in the wrong date partition; filterByWindow (src/core/backfill/scan_util.js:68) keeps timestamp-less items unconditionally, so --since imports records the window should exclude; and settle.js:381 sets Infinity, which is outside every window, so the ordinal fallback can never match, part_id dedupe never collapses the pair, and the turn double-counts, which is exactly what R11 exists to prevent

Fixed (session_file.js:363-366): the field's own present-value test now runs at both levels before the fallback decides. content and usage got named present-value tests (statedValue, plainObject) so every field is guarded the same way.

2. Medium: id was read envelope-first, contradicting LLP 0158's own statement of where identity lives

session_file.js:275 (pre-fix) read id through the same envelope-first path as content fields, while the PR's own LLP text says the record line states "what identifies and positions the message, ['id','message','parentId','timestamp','type']". Verified: a record with id at both levels resolved to the nested one.

No upside (the verified shape never nests id) and an unrecoverable downside. The nested envelope is OpenClaw's normalization of a provider response and already carries idempotencyKey; a version that also copied the provider's msg_01... id there would repoint every backfilled message_id (backfill.js:517) and every settled one (settle.js:471), and therefore every part_id. Committed rows would stop deduping against new ones and the history would double, with nothing raised anywhere.

Fixed (session_file.js:287): id is line-first, envelope-fallback, so a record that states identity only in the envelope still resolves one. LLP 0158 gains rule 7 stating the exception and why.

3. Medium: the PR's own fixture-fidelity guarantee was already false

The LLP addition claims "Every test that writes a session file writes the real two-level record shape, through one helper per suite, so no fixture can quietly re-invent a flat envelope."test/plugins/openclaw-backfill.test.js:682 bypassed messageLine() and wrote JSON.stringify({ type: 'message', ...ASSISTANT_RECORD }) by hand. Measured: with the envelope read reverted, that test still passed while 15 others failed, so the sole test of the relocated-OPENCLAW_HOME path was not a regression guard at all.

Separately, the new shape pin at :236 asserted nested keys withoutidempotencyKey, contradicting the live key list stated 190 lines above it and in LLP 0158 Context. The one test whose job is "this is the shape OpenClaw appends" pinned a shape OpenClaw does not append.

Fixed: the relocation fixture routes through messageLine, and ASSISTANT_RECORD carries idempotencyKey so the pin matches the documented key list. Re-verified by mutation: with openclawMessageEnvelope regressed to return row, the relocation test now fails (it did not before).

4. Medium: record was documented at the wrong level, inviting the #543 read

session_file.js:253-254 and types.d.ts:26 told the next caller that parentId, idempotencyKey and toolCallId are "reachable through record, the untouched record line". Per LLP 0158's verified key list idempotencyKey is nested, and match_key.js:175 states toolCallId/toolName "live on the message". So record.idempotencyKey is undefined, and the doc written to prevent one-level-too-high reads was itself instructing one.

Fixed: both sites now name the level. parentId is on the line; a message-level field this reader does not normalize is at record.message.

5. Low: timestamp precedence flipped versus master, unpinned

Master always used row.timestamp; the PR prefers the nested one, and a real record states both. I could not construct a failure from the precedence itself (the two differ only by append latency, inside every tolerance), but no test pinned it: both fixture helpers write the identical timestamp to both levels, so the precedence is unobservable in the suite, and only the envelope-absent direction was covered.

Fixed by pinning rather than changing: a new reader test states different values at the two levels and asserts envelope-wins for timestamp and line-wins for id in one place. Four more reader tests cover the blank/wrong-typed nested field, the blank-nested-with-no-line-value case, and a message key that is not an object.


Not changed, worth a human's eye

  • No hermetic smoke writes an OpenClaw session file, in any shape. There is no backfill_openclaw_fixture analog to the Codex and Claude flows, so tier 2 has never touched a real-shape OpenClaw session file and the only gate that would have caught OpenClaw backfill projects 0 rows from real session files: message fields are read at the top level but OpenClaw nests them under 'message' #543 is the manual docs/ACCEPTANCE.md:437. Given CLAUDE.md's three-tier model, that gap is the reason this bug shipped green, and it outlives this PR.
  • A projectable record that yields no message is counted nowhere (backfill.js:441, if (!message) continue), so scan_complete can under-report against messages_projected + records_excluded. Pre-existing, not introduced here.
  • LLP 0158 Decision still says "The invariant is tested once in core", which contradicts the same document's decision to keep the reader plugin-local. Pre-existing; this PR added a bullet beneath it without touching it.

Gates

npm test 3281 tests / 3280 pass / 0 fail / 1 skipped, npm run typecheck clean, at df7f119. Each fix was positively verified by mutation, not by a green suite: reverting the guard-order fix fails only a blank or wrong-typed nested field reads as absent..., reverting the id direction fails only the envelope wins for timestamp, the record line wins for id, and reverting openclawMessageEnvelope now fails the relocation test it previously passed.

House style checked: no em dash (U+2014) anywhere in the change, no statement-terminating semicolons, no @typedef and no inline import('...') types, and the three added @refs resolve to LLP 0158's ## Decision with correct attachment.

…xception
Three follow-ups to the round-1 fixes, all in the same family the round-1
findings were: a new behavior that no test pins, and docs that state a rule
the code does not follow.
- `statedValue`, the `content` present-value test the guard-order fix
introduced, was entirely unpinned: replacing it with the identity function
left the whole suite green while a nulled-out nested `content` suppressed a
usable record-line value and landed `content: null` on the message. Pinned
in both directions (line supplies it; absent when neither level does).
- `types.d.ts` still stated the blanket envelope-first rule over `id`, which
the same commit made line-first (LLP 0158 rule 7), and over `content`,
whose test refuses only `null`. The published declaration is what a package
consumer reads, so it now names both exceptions.
- LLP 0158 rule 6 claimed a blank or wrong-typed nested value can never
suppress the line. True of the string fields, false of `content`: a nested
`content: " "` or `content: 42` does suppress it. The rule now says what
"reads as absent" is per field.
Also extends the non-object-`message` test with the `null` case, the one
input where the plain-object guard is the difference between reading the
record line and throwing out of the whole file read.
Co-Authored-By: Claude <noreply@anthropic.com>
@philcunliffe

Copy link
Copy Markdown
ContributorAuthor

Neutral review round 2: PR #552 @ df7f119

Verdict: findings (3 actionable, all Low, all fixed and pushed as b60ca6a).

This round targeted the round-1 delta 44c7080..df7f119. All five round-1 fixes are real and I verified each by mutation rather than by a green suite. The three findings below are the same family the round-1 findings were, one level further in: a behavior the fix introduced that no test pins, and docs that state a rule the code does not follow.


Verification of the round-1 fixes

1. The reordered guard (session_file.js:363-366) is correct in both directions. Probed at runtime through readOpenclawSessionMessages, not reasoned about:

recordresult
line provider:"anthropic", nested provider:" "provider: "anthropic"
line timestamp:"2026-07-30T00:00:00.000Z", nested timestamp:"nope"line's value
line usage:{input:1}, nested usage:null{input:1}
line content:"from-line", nested content:null"from-line"
nested-only {id:" ", timestamp:"", model:42, stopReason:null}all four absent, nothing substituted

The resurrection direction holds too: on the verified real shape the record line carries only ['id','message','parentId','timestamp','type'], so a user turn whose envelope legitimately omits model/provider/usage has nothing to resurrect them from. I confirmed the per-field fallback does pick up a same-named line field when one exists (a synthetic line-level provider:'openai' reaches a user turn), but that is rule 6 working as documented, it is unchanged from before the round-1 fix, and it is unreachable on the shape OpenClaw writes.

Mutation: reverting messageField to the absence-based version fails exactly one test, a blank or wrong-typed nested field reads as absent... (61/62 pass).

2. The id direction flip is safe downstream. Line-first restores master's behavior exactly (master read nonBlankString(row.id)), so nothing regressed. Both part_id producers consume the same reader field: backfill.js:517 (projected.message_id = message.id, the gateway derives part_id from it) and settle.js:471-476 (settled.message_id = match.id, settled.part_id = match.id + '#' + partIndex). The settlement content match key does not depend on id at all: buildOpenclawSessionIndex keys on sessionMatchKey(rawRole, message.content) and uses id only as the value to upgrade to. match_key.js is byte-identical to master. Mutation: reverting to envelope-first fails exactly one test, the envelope wins for timestamp, the record line wins for id.

3. The fixture fix is real and was load-bearing. With openclawMessageEnvelope regressed to return row, 31 of 62 OpenClaw tests fail including a relocated install is found through OPENCLAW_HOME (#17). I then reverted only the fixture line back to JSON.stringify({ type: 'message', ...ASSISTANT_RECORD }) and re-applied the same regression: test #17 goes green again. That is direct proof the round-1 fixture change converted a test that passed for the wrong reason into a real guard. Dropping idempotencyKey from ASSISTANT_RECORD fails the shape pin, so the pin is honest against LLP 0158's live key list. Every settlement fixture routes through sessionFileLine (writeSessionFile, openclaw-settlement.test.js:171); no raw type:'message' literal bypasses a nesting helper in any OpenClaw suite.

4. Whole-PR sanity re-confirmed at df7f119.PROJECTABLE_PROVIDERS, effectiveProviders and partitionByBackend are byte-identical to master (only line numbers shift); claude-cli still excludes with covered_by: claude_transcript and unknown still fails closed. readOpenclawSessionMessages still has exactly two non-test callers, backfill.js:266 and settle.js:325, and no message.record.<field> read survives anywhere. match_key.js unchanged, so LLP 0159 match-key semantics are untouched. The PR's real diff against its merge-base (57e2ec9) is 8 files; master has since moved to be07015 (#554) but touched nothing under openclaw/, so there is no semantic conflict.

House style clean: no U+2014 anywhere in the diff, no statement-terminating semicolons, no @typedef, no inline import('...') types, and test/core/llp-ref-hygiene.test.js passes on the added @refs.


Findings

1. Low: statedValue, the new content present-value test, was pinned by nothing

session_file.js:378 (at df7f119)

functionstatedValue(value){returnvalue===null ? undefined : value}

This function is new in the round-1 guard-order fix and is the only thing making a nulled-out nested content fall through to the record line. I replaced its body with return value (the pre-fix behavior) and ran the whole repo suite: 3281/3281 green. Nothing observed it. That is precisely the "a test that passes under both the fixed and the buggy reader is not pinning anything" case, and the consequence is live: with the identity version, a nested content: null both suppresses a usable line value and lands content: null on the message, which projectedMessageFromRecord drops (backfill.js:502-506) and which buildOpenclawSessionIndex hashes into a content key no live row can match (settle.js:388).

Fixed by adding a nulled-out nested content is unstated, so the record line still supplies it, which pins both halves (the line supplies it; with neither level stating it the key is absent, not null). Re-verified: the identity mutant now fails that test and only that test, 61/62.

2. Low: types.d.ts still stated the blanket envelope-first rule the same commit had carved two exceptions out of

types.d.ts:20-23 (at df7f119): "each is read off the nested message envelope, falling back to the record line, and is absent when the field is missing, non-string (for the string fields), or blank."

Both clauses are now false for a listed field. id is line-first as of the same commit (session_file.js:287, LLP 0158 rule 7), and content is neither string-tested nor blank-tested. This is the round-1 finding-4 defect in the one file it matters most in: types.d.ts is what npm run build:types publishes, so it is the description a package consumer reads, and it currently points them the way #543 was read.

Fixed: the interface doc now names both exceptions.

3. Low: LLP 0158 rule 6 overclaimed, and the code is the honest one

llp/0158-one-reader-for-openclaw-session-jsonl.decision.md:74-76 (at df7f119): "a nested value that reads as absent (blank, wrong-typed, null) cannot also be the value that suppresses the line."

True of every string field and of usage. False of content, whose test refuses only null. Verified at runtime: a nested content: " " and a nested content: 42 each suppress a line-level content: "from-line".

I deliberately did not "fix" this by tightening statedValue. The narrow test is the right call and is already argued at its own JSDoc: content has no single shape, and refusing a blank string would change message.content from '' to undefined, which changes the settlement content key for a turn whose stored match_key was computed at wire time. The overclaim is in the doc, so the doc moved: rule 6 now says "reads as absent" is each field's own answer, and states content's.

Also folded in (not a separate finding): the a message key that is not an object test (openclaw-session-file.test.js:374) asserted only outcomes that are identical with and without the isPlainObject guard, so it pinned nothing. It now covers message: null, the one input where the guard is the difference between reading the record line and throwing out of the whole file read.


Not fixed, and why

  • No hermetic smoke writes an OpenClaw session file in any shape. Reported in round 1, out of scope here by instruction, and it outlives this PR.
  • The per-field fallback can read a same-named field off the record line for a turn whose envelope deliberately omits it. Confirmed at runtime, but it is documented rule 6, it predates the round-1 fixes, and it is unreachable on the verified record shape. Not a defect; recorded so the next reader does not re-find it.
  • timestamp remains envelope-first while id is line-first, though LLP 0158 names both as line-level positioning fields. Round 1 pinned this rather than changing it; I could not construct a failure from the precedence either, both consumers read the same reader so they cannot disagree, and flipping it in a final review round would be a behavior change with no evidence behind it. Left as pinned.

Gates

At b60ca6a: npm test -> # tests 3282 # pass 3281 # fail 0 # skipped 1; npm run typecheck clean. (df7f119 baseline was 3281 / 3280 / 0 / 1.) Each fix positively verified against the committed blob: the new test name and the changed types.d.ts / LLP 0158 sentences are present in b60ca6a and absent in df7f119.

No open findings.

@philcunliffe

Copy link
Copy Markdown
ContributorAuthor

Neutral triage: PR #552 @ b60ca6a

The review fix-loop hit its 2-round cap (44c7080 → 5 findings fixed as
df7f119, df7f119 → 3 findings fixed as b60ca6a), which routed this PR to
triage rather than a third review round. Both rounds reported every finding
they raised was fixed, not left open, so triage's job was to verify that claim
against the tree rather than assume a backlog.

Verification performed (in a scratch worktree at b60ca6a, npm install

  • npm test = 3282/3281/0/1, matching round 2's reported gates exactly):
  • Grepped every finding's cited symbol/location (messageField,
    statedValue, openclawMessageEnvelope, the id line-first read, the
    messageLine/sessionFileLine fixture helpers, types.d.ts, LLP 0158
    rules 6-7) and confirmed each fix is present in the committed tree exactly
    as both rounds describe.
  • Independently mutation-tested the three highest-risk claims rather than
    trusting the review prose: reverting statedValue to identity fails
    exactly 1 test (a nulled-out nested content is unstated...); reverting
    id to envelope-first fails exactly 1 test (the envelope wins for timestamp, the record line wins for id); reverting
    openclawMessageEnvelope to return row fails 31/62 OpenClaw tests
    including the OPENCLAW_HOME relocation test. All three match the rounds'
    own mutation claims. No raw record.role/record.content reads remain in
    backfill.js or settle.js.

Residual set — two items, both explicitly named by round 2 as deferred by
design, not unresolved defects:

  1. Hermetic smoke gap: no backfill_openclaw_fixture analog to
    backfill_codex_fixture.js / backfill_claude_fixture.js exists under
    hypaware-core/smoke/flows/. This is the actual reason OpenClaw backfill projects 0 rows from real session files: message fields are read at the top level but OpenClaw nests them under 'message' #543 shipped
    green. Classified non-blocking: it is a coverage gap, not a
    behavioral defect — nothing misbehaves today because of it.
  2. timestamp/id precedence asymmetry: timestamp resolves
    envelope-first, id resolves line-first (deliberate, LLP 0158 rule 7).
    Round 2 pinned it with a dedicated test and could construct no failure;
    both consumers share the one reader so they cannot disagree. Classified
    non-blocking: pinned, argued, and unreachable as a defect under the
    current one-reader design.

Both residuals are non-blocking, so the PR is clear to merge. Filed
#555 to carry them forward with
enough detail to act on later (including what a backfill_openclaw_fixture
flow would need to stage and assert).

@philcunliffephilcunliffe added the neutral:approved neutral reviewed this and holds it for a maintainer merge (own or adopted PR; LLP 0025/0030) label Jul 31, 2026
@philcunliffe
philcunliffe merged commit 17a9730 into masterAug 2, 2026
9 checks passed
@philcunliffe
philcunliffe deleted the fix/issue-543 branch August 2, 2026 02:00
philcunliffe added a commit that referenced this pull request Aug 3, 2026
…p, json_path revival (#570)
* Design: OpenClaw two-lane capture (LLP 0172)
* Plan: OpenClaw two-lane capture executable tasks (LLP 0173)
* openclaw config: validate sweep_cron and quiesce_ms in backfill section
validateBackfillSection now accepts sweep_cron (a 5-field cron
expression, validated with core's shared isCronExpression grammar so a
malformed schedule is rejected the same way a sink's config.schedule
is) and quiesce_ms (non-negative integer) alongside the existing
on_join/window_days keys, added together so the unknown-key rejection
loop recognizes both from the same merge.
Task-Id: T6
* Restore json_path attach-probe format, add sweep field to BackfillContribution
hypaware-plugin-kernel-types.d.ts: PluginAttachProbeManifest.format regains
'json_path' (removed by LLP 0143 after #212's orphaning danger), plus the
new container_path, provider_keys, and cache_glob fields the format needs,
reusing the existing marker_header. The comment at the removal site is
revised, not deleted: it now explains that the runtime support LLP 0173
T2/T3 add closes the gap #212 warned about, so a manifest can only declare
this format once both sides exist.
BackfillContribution gains an optional sweep?: { cron: string } field for
the daemon's periodic sweep (LLP 0173 T9), absent-by-default for every
existing contribution.
Both additions are purely additive: npm run typecheck (tsc --noEmit over
the whole tree, including hypaware-core/plugins-workspace/openclaw) and
npm test pass unchanged, proving no existing consumer's typecheck shifted.
Task-Id: T1
* OpenClaw attach writes the two provider overrides, refusing to merge
LLP 0169 reverses LLP 0152's premise: there is a real, reversible
settings write for OpenClaw again, so the adapter's honest no-op
attach() has nothing left to be honest about.
New hypaware-core/plugins-workspace/openclaw/src/attach.js exports
createOpenclawAttach({homeDir, env, fs}), mirroring the Claude adapter's
attach() shape (same AiGatewayClientAttachContext, same
withSpan('client.attach', ...), same dry-run branch). It reads
openclaw.json through the one core settings-path seam (so $OPENCLAW_HOME
relocation resolves the same file the manifest's probe will), refuses
with {status:'failed', reason} when models.providers.anthropic or
.openai is already there, and otherwise writes both entries whole from
attachCtx.endpoint: bare origin for anthropic, endpoint + '/v1' for
openai, each with the x-hypaware-upstream marker header and the
mandatory empty models array.
The refusal check runs entirely before the single atomicWriteFile, so
there is no partial write to roll back, and it returns rather than
throws, which is the whole mechanism by which a refuse during
attach-on-join warns instead of failing the join. Every other key in
openclaw.json is carried through by reference. Both output modes end
with the 'openclaw gateway restart' instruction, since a --json caller
is as blocked on the restart as a human is.
index.js drops the no-op body, STEERING_PLUGIN_NAME,
ROUTING_OWNED_BY_STEERING_PLUGIN_MESSAGE and the @ref LLP 0143#decision
block, and wires activate() to the new effect. The registered attach()
keeps the kernel's Promise<void> contract, so the outcome object is
dropped there on purpose: both callers already derive success from a
throw plus the one-line JSON the effect writes.
Tests cover the refusal (including that the file is byte-identical
after it, and that it never throws), the exact two-entry shape with the
bare-origin/+v1 asymmetry, key preservation, the restart instruction on
both output modes, dry-run, $OPENCLAW_HOME, and the missing/malformed
config hard failures. The two attach tests in
openclaw-client-registration.test.js are retargeted at the new behavior
so the suite stays green; T10 owns that file's fuller rewrite.
Task-Id: T4
* daemon/status.js: restore json_path attach-probe read branch
Restores the probe.format === 'json_path' read branch removed by LLP
0143 / PR #510, parallel to the existing json/toml branches in
probeClientAttachFromDescriptor: navigate container_path + each
provider_keys entry, read headers[marker_header], and report attached
when it equals the expected provider key for at least one configured
key. Pure read, no ownership/backup concerns.
Task-Id: T3
* openclaw backfill: quiesce window skips recently-modified session files
listSessionFiles(agentsDir) gains an optional quiesceBeforeMs cutoff, and
runOpenclawBackfill() computes it once per run as Date.now() - quiesceMs.
quiesceMs resolves from the plugin's own config.backfill.quiesce_ms,
defaulting to 180000ms (QUERY_FLUSH_DEBOUNCE_MS plus a one-minute margin),
so a run never imports a session file OpenClaw is still mid-write on, or a
settlement pass is still mid-flush against. Composes with the existing
effectiveProviders/partitionByBackend CLI-backend logic (R10) rather than
replacing it.
Task-Id: T8
* Lane B sweep metadata: openclaw backfill provider + narrowed runner ctx
createOpenclawBackfillProvider now populates the contribution's opt-in
sweep field from config.backfill?.sweep_cron, defaulting to every 5
minutes when absent (LLP 0172#lane-b-sweep, R7).
src/core/commands/backfill.js's runBackfillProvider, runProvider, and
resolveOwnersForRun now declare a new BackfillRunnerContext interface
(env, config, storage, backfills, backfillMaterializers) instead of the
full CommandRunContext, a pure structural narrowing so the daemon-side
sweep driver (LLP 0173 T9) can build one without assembling registries
it never uses. Existing hyp backfill CLI-path and onboarding-finale call
sites keep typechecking and passing unchanged.
Task-Id: T7
* json_path detach returns: ownership, backup-not-discard, best-effort cache purge
LLP 0143 pulled the `json_path` branch out of the disk-driven undo because
LLP 0152 left nothing on disk for it to reverse. LLP 0169 reverses that
premise, so the branch comes back - reshaped for the two provider entries
attach now writes, not the single shadow provider of the old design.
`detachJsonPathProviders` judges each `provider_keys` entry on what it points
at, because this format has no HypAware-owned marker to replay: its undo
record IS the entry. Ours (the gateway's own `baseUrl`, in either the bare
origin or `+ /v1` spelling, with `marker_header` naming its own key, in the
shape attach produces) is deleted. Anything else present at our key is backed
up to a `_hypaware_detach_backup.<key>` sibling inside the same container
before the live key goes, following the `prev_malformed` precedent of LLP
0163: never discard a value HypAware did not write. That closes the
json/toml-vs-json_path asymmetry LLP 0163 flagged as worth its own look,
converging on the outcome without the top-level marker key LLP 0163 correctly
ruled out for this client.
The derived caches (`cache_glob`, relative to the client's config home) are
then purged of the same keys, best-effort: they do not self-heal, so a
partial purge beats none, and one unreadable cache file is logged and skipped
rather than failing a detach whose settings half already landed.
An unknown gateway base URL has no safe default here - guessing either way
silently deletes a foreign value or reports a finished detach over a client
still routed at a dead port - so it refuses (`EXPECTED_BASE_URL_UNKNOWN`).
Both callers degrade correctly: `reverse()` keeps the marker, `hyp detach`
prints the reason.
Both real callers thread the base URL: `detachClientViaCore` resolves it
through the same three rungs manual attach already walks (live
`localEndpoint()`, configured `listen`, the daemon's persisted bound port),
every one optional so detach keeps working with the gateway capability
unloaded; `reverse()` passes the `ctx.endpoint` `perform()` attached with.
Task-Id: T2
* OpenClaw manifest: restore json_path attach_probe, retire steering-plugin copy
hypaware.plugin.json gains contributes.client.attach_probe (design 1.4:
json_path format, .openclaw/openclaw.json settings file, models.providers
container, anthropic/openai provider keys, x-hypaware-upstream marker
header, agents/*/agent/models.json cache glob). description and
picker[0].summary drop every @hypaware/openclaw-steering-plugin reference
and state the two capture tiers directly: live gateway capture once
attached, plus a periodic transcript sweep.
Claude's manifest gains the LLP 0167#onboarding line naming the
claude-cli/<model> case OpenClaw's CLI-backend exclusion produces, so a
user knows which picker entry an OpenClaw-routed Claude Code session
belongs to.
projector.js's UPSTREAM_HEADER comment no longer credits the deleted
steering plugin; it now describes the config-override write attach()
itself makes.
Adds a manifest-shape test asserting attach_probe parses to the exact
design-1.4 fields and that description/summary no longer match
/openclaw-steering-plugin/. Updates the one existing assertion this
manifest change makes false (the R7 "no attach_probe" descriptor check)
to the restored json_path shape; the remaining behavioral rewrites of
that test file are T10's scope.
Task-Id: T5
* Delete openclaw-steering-plugin/ (LLP 0172 Section 5, R9)
Removes openclaw-steering-plugin/ in full (src/, test/, package.json,
openclaw.plugin.json, .d.ts files) and test/plugins/openclaw-steering-plugin.test.js:
Lane A's config-override entries make OpenClaw route to the gateway on
its own, so the credential-borrowing runtime auth shim, the live
wire-parity mirror, the steering decision logic, the live warning
ledger, and the gateway endpoint resolver that fed them no longer have
a purpose.
Also drops tsconfig.json's stray "openclaw-steering-plugin" include
entry (line 19), not named in the design's own deletion inventory but
found verifying the deletion against the real tree; leaving it would
have left a dead include path.
docs/ACCEPTANCE.md and test/plugins/openclaw-manifest.test.js still
mention the package name (an onboarding rewrite reserved for T13, and
a T5 regression test asserting the manifest no longer references it,
respectively); neither is in this task's file list.
Task-Id: T11
* openclaw-client-registration.test.js: finish T4/T5's deferred rewrite
T4 and T5 already retargeted the two attach() no-op tests and the
descriptor's attach_probe assertion to keep the suite green while they
landed; this closes the two pieces both left for T10:
- The "honest no-op" detach test's comment still credited the retired
R7 no-attach_probe guard. Since T5 restored the manifest's json_path
attach_probe, the no-op this test actually observes on a fresh temp
HOME is detachClientFromDisk's absent-settings-file guard instead.
Corrected the comment to say so.
- Added a companion case that stages a real openclaw.json via the
actual createOpenclawAttach() effect, then drives the same hyp
detach CLI entry point (buildClientDescriptorMap's real manifest
descriptor -> detachClientFromDisk's json_path branch) and asserts
the ownership-based detachJsonPathProviders (T2) actually fires:
changed:true, the removed baseUrl, and both provider entries gone
from the file while everything else in it is untouched.
Task-Id: T10
* Daemon sweep driver: run sweep-bearing backfill providers on the tick loop
New `src/core/daemon/backfill_sweep.js`. `createBackfillSweepDriver({backfills,
backfillMaterializers, env, config, storage})`'s `tick({now})` walks
`backfills.list()`, skips any contribution with no `sweep` field or a cron that
is not due (`cronMatches`, the sink driver's own due-check), and fires
`runBackfillProvider` per due contribution with a `sweep-<name>-<now>` dev run
id. Runs are fired unblocked: `tick()` resolves once each run has been started,
never once one finishes, so a provider's transcript scan cannot stall the sink
snapshots, the source-detail refresh, or `persist()` later in the same tick. Both
settlements are handled, so a failing run is a logged `backfill.sweep_failed`
record (component `openclaw`, operation `backfill.sweep`, `error_kind`) rather
than an unhandled rejection that would take the daemon process down. A malformed
`sweep.cron` is logged and treated as not due rather than thrown, so one
provider's bad metadata cannot skip the rest of the list.
`runtime.js`'s `runTick()` calls `await sweepDriver.tick({now})` directly after
the existing sink-driver tick, riding the same `DEFAULT_TICK_INTERVAL_MS`
60-second loop: a `*/5 * * * *` schedule only needs a due-check once a minute, so
this opens no second timer to start, drain, and account for at shutdown.
Also repairs the typecheck this task's branch point already failed: T7's
`sweep: { cron: opts.config?.backfill?.sweep_cron ?? ... }` does not compile,
because the plugin's config slice is a `JsonObject` and every step below its root
is a `JsonValue`. Read through a `resolveSweepCron` mirroring the
`resolveQuiesceMs` helper already sitting beside it; behavior is unchanged.
Externally blocked for real capture: until PR #552 (issue #543) merges, the
LLP 0158 reader still reads OpenClaw v3 fields flat, so a sweep projects nothing
from a real transcript. These tests passing is not evidence that it does.
Tests: the due-check fires only sweep-bearing, cron-due contributions and builds
the narrowed `BackfillRunnerContext` from the daemon's own runtime fields; a
rejected run neither throws out of `tick()` nor lands as an unhandled rejection,
and a never-settling run does not block the tick. A separate wiring test boots a
real daemon with a fixture plugin whose contribution opts into a sweep and proves
the tick actually runs it, which no unit test of the driver can show.
Task-Id: T9
* docs/ACCEPTANCE.md: rewrite openclaw_capture for two-lane capture
Drops the steering-plugin link/enable setup and the
before_model_resolve/hooks.allowConversationAccess version-gate language
(the plugin is deleted; Lane A depends on no OpenClaw hook API). Adds a
setup step running `hyp attach --client openclaw` followed by the restart
instruction it prints, a sweep step (detach to strip the live route,
confirm the row is absent, confirm it lands within one sweep interval past
the quiesce window), and a zero-duplicate assertion (a turn both lanes
observe resolves to exactly one row for its part_id, proven against the
daemon's own scheduler rather than a manual `hyp backfill`). Re-confirms
LLP 0167#verify-results items 1, 3, and 4 against the current tree's
attach/detach behavior instead of assuming them. Drops the retired
deferred-provider-family warning-ledger step (LLP 0171 retires R13; no
ledger) and the shadow-provider-id failure mode (Lane A overrides the
existing anthropic/openai entries, it does not register new ids).
States in the section's own Requires that the sweep/dedupe steps need PR
#552 merged (the LLP 0158 reader still parses OpenClaw v3 flat), and the
client_attach status-row re-confirmation needs PR #553 merged (a
now-probed openclaw otherwise falls back to pre-#553 status behavior).
This is a doc; the test is a human's successful run against a real
OpenClaw install, which this change cannot perform. It is specified
against what T2 (detach), T4 (attach), and T5 (manifest) actually
implement in this tree, read directly from
hypaware-core/plugins-workspace/openclaw/src/attach.js,
src/core/config/client_detach_disk.js, and
hypaware-core/plugins-workspace/openclaw/hypaware.plugin.json.
Task-Id: T13
* Add backfill_openclaw_fixture hermetic smoke for Lane B sweep
New backfill_openclaw_fixture.js under hypaware-core/smoke/flows,
mirroring backfill_claude_fixture.js / backfill_codex_fixture.js: writes
a minimal OpenClaw v3 session JSONL in the nested-message-envelope shape
(PR #552's reader) under a temp agents/<id>/sessions/ tree with a
controllable mtime, drives the real createBackfillSweepDriver (T9)
through a cron-due tick, and asserts (a) a file inside the default
180000ms quiesce window is skipped, (b) a file backdated past it is
captured with native message identity, and (c) rerunning the sweep on a
later cron-due tick (forcing a fresh devRunId, so the ai-gateway
materializer's dedupe genuinely re-scans committed partitions) nets zero
new rows for the already-written part_ids.
Driving a real, non-dry-run sweep write for the first time (T9's own
tests only ever exercised a mocked runBackfill seam) surfaced a latent
bug: writeRows/flushDataset read ctx.query, which BackfillRunnerContext
never carried and the daemon's createBackfillSweepDriver(...) call never
supplied, so any real sweep write actually crashed on
"Cannot read properties of undefined (reading 'getDataset')" in both the
smoke and the real daemon path. Threaded query through
BackfillRunnerContext, BackfillSweepDriverOptions,
createBackfillSweepDriver, and the daemon's sweepDriver construction, and
updated LLP 0172's field enumeration and ctx samples (Sections 4.3/4.4)
to match. Extended the existing T9 unit tests
(test/core/daemon-backfill-sweep.test.js) to cover the new required
field and its passthrough.
Task-Id: T12
* openclaw-backfill.test.js: backdate fixture mtimes to close a quiesce-window race (#570)
listSessionFiles compares stat.mtimeMs <= Date.now() even when a test sets
config.backfill.quiesce_ms: 0 to opt out of the quiesce gate for something
unrelated: 0ms only removes the margin, not the comparison. A fixture
written moments earlier could race the provider's own later Date.now() call
across two different clocks and occasionally lose, projecting 0 items
instead of 1. Confirmed non-deterministic: PR #570's commit 0c62a21 produced
both a green and a red `test (24)` CI run from the identical commit.
writeSession now backdates every fixture's mtime by a small, fixed margin
(FIXTURE_MTIME_MARGIN_MS), comfortably clearing the race while staying far
below the real 180000ms default quiesce window, so the tests that rely on
genuine freshness against that default are unaffected. The one test that
writes its session file outside writeSession (the OPENCLAW_HOME relocation
test) gets the same backdate applied directly.
* Review round 1: make an attach refusal observable, and three smaller fixes
Finding 1 (major). The registered `attach()` wrapper discarded the effect's
`OpenclawAttachOutcome`, and both callers infer success from "did it throw",
so a refusal recorded a `done` marker whose endpoint and assets_key matched
forever: the join never retried even after the user cleared the conflicting
`models.providers` entry, while the json_path attach probe kept reporting
`not attached`. `hyp attach --client openclaw` printed the refusal and exited
0. LLP 0172 1.3 is authoritative (it promises the `{status:'failed', reason}`
outcome is recorded and retried), so the wrapper now rethrows a failed
outcome: `perform()`'s catch turns it back into that shape (recorded, warned,
retried, the join's other actions untouched) and `runClientLifecycle`'s catch
makes it exit 1. Rethrowing at the wrapper rather than teaching `perform()` to
parse the adapter payload is what fixes both callers, since the CLI hands the
adapter `ctx.stdout` directly and captures nothing to inspect. LLP 0172 1.3
gains the translation step it left implicit; the test that locked in the
swallow now asserts the retryable failure, plus the reconciler and exit-code
halves.
Finding 2. `clientConfigHome` took the first segment of the settings path's
home-relative form, which is not the config home when `$OPENCLAW_HOME` is
nested inside `$HOME`: the cache glob then matched nothing and the purge
silently no-opped while the settings half reported success. Derive it by
stripping the manifest's own `settings_file` tail instead, the exact inverse
of what `resolveClientSettingsPath` joined on. Regression test uses a
two-segment `OPENCLAW_HOME`.
Finding 3. `listSessionFiles`'s JSDoc claimed the CLI path runs unfiltered.
It does not: `runOpenclawBackfill` computes the quiesce cutoff on every run,
and `run()` is the single entrypoint for the CLI, the onboarding finale, and
the sweep. Name `plan()` as the only unfiltered caller. No behavior change.
Finding 4. The sweep fired a due provider with no record of what was still
running, so a pass outliving its cron interval got a second concurrent run
against the same datasets and mid-flush spool. Add the `maintenanceInFlight`
guard shape, widened to a Set because the driver fires one run per provider,
with a `backfill.sweep_skipped` / `already_running` record and clearing on
both settlements. LLP 0172 4.4 states the re-entrancy rule it had left out.
Co-Authored-By: Claude <noreply@anthropic.com>
* Review round 2: thread the plugin config, make attach idempotent, name the sweep's component
Three review findings, each with a doc edit in the same commit.
1. `sweep_cron` and `quiesce_ms` were validated then discarded. `activate()`
built the backfill contribution without passing `ctx.config`, so both keys
this PR adds to `validateBackfillSection` resolved to the hardcoded
`*/5 * * * *` / 180000ms defaults at runtime, with no diagnostic. The
existing unit tests handed `config` straight to the factory, so the missing
wiring was invisible to them; the new test starts from an activation.
2. `attach()` refused on bare key presence, so it was not idempotent over its
own output. This PR's manifest `attach_probe` is what makes openclaw
eligible for attach-on-join, and `isCurrent()` re-performs attach on an
ephemeral-port rebind (LLP 0086) or an asset-set change (LLP 0107). Every
re-perform then refused: the marker churned to `failed`, `hyp attach
openclaw` exited 1, and `openclaw.json` stayed pinned to the dead port
while the marker-header probe still reported `attached: true`.
The refusal is now ownership-aware, on the self-identifying triple detach
already tests before deleting. `isOwnedProviderEntry`/`ownedBaseUrls` move
out of `client_detach_disk.js` into a shared
`src/core/config/provider_entry_ownership.js` so the two halves cannot
disagree about the same file. Attach passes no base-URL set (on a drift
re-attach its own entry carries the old origin); detach still passes one,
because there the wrong answer deletes a value HypAware never wrote.
Everything that fails the test still refuses, including `null`, a foreign
entry, and a hand-edited one that merely kept the header.
3. The generic sweep driver stamped `component: 'openclaw'` on all five of its
records while logging as `backfill-sweep`. It fires any contribution
carrying a `sweep` field, so a second opt-in would have been misattributed.
`component` now names the emitting module; plugin identity already rides
`hyp_plugin` and `provider`.
Docs updated to match: LLP 0167#attach-detach, LLP 0169's decision bullet and
summary, LLP 0171 R2, LLP 0172 sections 1.2 and 2.2, LLP 0173's implementer
note on the sweep's telemetry pair.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: test <test@test.com>
Co-authored-by: Claude <noreply@anthropic.com>
philcunliffe pushed a commit that referenced this pull request Aug 3, 2026
master's #552 landed the same two-level reader fix this branch carries
(session_file.js, settle.js, LLP 0158, and the session-file/settlement/
backfill tests are now byte-identical on both sides), so the merge reduces
to the two artifacts that were still only here: the shared smoke fixture
builder and the flow that uses it.
Conflicts and how they were resolved:
- hypaware-core/smoke/flows/backfill_openclaw_fixture.js (add/add). Two
different flows had claimed one filename. #570's is LLP 0173's T12 sweep
flow: it drives createBackfillSweepDriver and asserts the quiesce window
and cross-write dedupe. This branch's enters through
`hyp backfill openclaw` and asserts what a two-level record projects to,
including the envelope-first timestamp precedence nothing else covers.
Neither subsumes the other, so both survive: #570 keeps the name LLP
0172/0173 already reference, and this branch's flow moves to
backfill_openclaw_session_shape.js.
- test/plugins/openclaw-backfill.test.js. This branch had nothing at the
conflicting hunk; master added FIXTURE_MTIME_MARGIN_MS, which the
already-merged writeSession body calls. Took master's block. The file is
now identical to master's.
One semantic fix the merge forced: #570 gave the backfill provider a
180000ms default quiesce window, so the renamed flow's freshly-written
fixture was invisible to the scan and the flow projected zero rows. It now
backdates the fixture's mtime four minutes, through the mtimeMs knob the
fixture builder already carried for exactly this. Verified by removing the
backdate and watching the flow fail with sessions_seen=0.
npm test: 3284 pass, 0 fail, 1 skipped.
Both smoke flows pass: backfill_openclaw_fixture and
backfill_openclaw_session_shape.
Co-Authored-By: Claude <noreply@anthropic.com>
philcunliffe added a commit that referenced this pull request Aug 3, 2026
… 1) (#558)
* OpenClaw session records are read one level too high (#543)
A real OpenClaw v3 `type: "message"` record states only `id`, `parentId`,
`timestamp`, and `type` on the record line: `role`, `content`, `model`,
`provider`, `api`, `stopReason`, and `usage` are all nested under `message`.
The LLP 0158 reader read them off the record line, so every field came back
absent, every record resolved to `provider: unknown`, the backfill allowlist
excluded all of them, and `hyp status` reported `backfill @hypaware/openclaw
[done] (0 rows)` for a session it had failed to read. The settlement enricher
was broken the same way one seam later (`record.role`/`record.content`), so a
real session settled nothing either.
The reader now owns the envelope address: fields are read from the nested
`message` object, falling back to the record line for a record that nests
none, and `role`/`content` are normalized fields rather than something each
consumer picks out of the raw record. Both consumers read them off the
normalized message, so neither can drift a level again.
Fixtures across the three OpenClaw suites now write the real two-level shape
through one helper each; the old flat fixtures asserted an envelope OpenClaw
never writes, which is why the suite stayed green through the bug.
LLP 0158 records the verified record shape, the envelope read rule, and the
path-faithful-fixture consequence.
Co-Authored-By: Claude <noreply@anthropic.com>
* Review fixes: the two-level read must guard both levels, and identity is the line's
Follow-up to the #543 envelope fix, from review of PR #552.
- `messageField` fell back on key *absence*, not value *usability*, so a
present-but-unusable nested value permanently masked a good record-line
one. A nested `provider: " "` beside a line-level `provider: "anthropic"`
resolved the record to `unknown` and the allowlist excluded it fail-closed;
a nested `timestamp` that did not parse dropped `message_created_at`, which
re-dates the row to session start, defeats the `--since` window (a
timestamp-less item is kept unconditionally) and puts the settlement
ordinal match outside every window so the turn never dedupes. Rule 3's
present-value test now runs at both levels before the fallback decides.
- `id` is now read line-first, envelope-fallback. LLP 0158 verified message
identity on the record line; envelope-first meant a future OpenClaw that
copied the provider's own id into the nested message would silently repoint
every `message_id` and `part_id`, so committed rows would stop deduping
against new ones and the history would double with nothing raised.
- The OPENCLAW_HOME relocation fixture still wrote the invented flat shape,
bypassing `messageLine`, so it passed with the envelope read reverted. It
now goes through the helper, and the shape pin carries `idempotencyKey` so
it matches the live key list the same file documents.
- Say which LEVEL `record` is: `parentId` is on the line, `idempotencyKey`
and `toolCallId` are at `record.message`. The old wording invited the very
read #543 was.
- LLP 0158 gains rules 6 and 7 for the two behaviors above, and the stale
"no live OpenClaw install was reachable" note on `usageAttributes` is
reconciled with the spelling this work verified.
Co-Authored-By: Claude <noreply@anthropic.com>
* Round-2 review fixes: pin content's present-value test, name the id exception
Three follow-ups to the round-1 fixes, all in the same family the round-1
findings were: a new behavior that no test pins, and docs that state a rule
the code does not follow.
- `statedValue`, the `content` present-value test the guard-order fix
introduced, was entirely unpinned: replacing it with the identity function
left the whole suite green while a nulled-out nested `content` suppressed a
usable record-line value and landed `content: null` on the message. Pinned
in both directions (line supplies it; absent when neither level does).
- `types.d.ts` still stated the blanket envelope-first rule over `id`, which
the same commit made line-first (LLP 0158 rule 7), and over `content`,
whose test refuses only `null`. The published declaration is what a package
consumer reads, so it now names both exceptions.
- LLP 0158 rule 6 claimed a blank or wrong-typed nested value can never
suppress the line. True of the string fields, false of `content`: a nested
`content: " "` or `content: 42` does suppress it. The rule now says what
"reads as absent" is per field.
Also extends the non-object-`message` test with the `null` case, the one
input where the plain-object guard is the difference between reading the
record line and throwing out of the whole file read.
Co-Authored-By: Claude <noreply@anthropic.com>
* A hermetic smoke writes a real-shape OpenClaw session file (#555 item 1)
#543 shipped green because tier 2 never touched an OpenClaw session file
in any shape. Codex and Claude each have a hermetic flow that stages a
real-shape transcript and drives `hyp backfill` against it; OpenClaw had
no analog, so the only gate that could catch the two-level nesting bug
was the manual acceptance smoke.
Adds `backfill_openclaw_fixture`, mirroring `backfill_codex_fixture.js`
and `backfill_claude_fixture.js`: boots `@hypaware/ai-gateway` +
`@hypaware/openclaw` against a tmp HYP_HOME, stages a v3 session file
under the fake HOME's `.openclaw/agents/<agentId>/sessions/<id>.jsonl`,
drives `hyp backfill openclaw`, and asserts non-zero `ai_gateway_messages`
rows with native record-line ids, envelope content, `provider=anthropic`,
`conversation_source=openclaw`, the `backfill.provider_finish` /
`backfill.write` spans and the `backfill.finish` log under the run's
`dev_run_id`, and a zero-new-rows idempotent rerun.
The fixture writer lives in `smoke/lib/openclaw_session_fixture.js` so no
flow can invent a flatter, friendlier record shape, and takes a
caller-chosen `mtimeMs` so the scheduled-sweep quiesce work (LLP 0173
T12) can extend it rather than replace it.
Co-Authored-By: Claude <noreply@anthropic.com>
* Make the openclaw fixture's envelope-first claim true and give it teeth
openclawMessageLine's JSDoc claimed writing `timestamp` at both levels
"makes the envelope-first read of it observable," but the implementation
copied the same destructured binding to both levels, so they were
byte-identical and no assertion could tell envelope-first from line-first
precedence. Add an optional `messageTimestamp` field that overrides the
nested `message.timestamp` only, keeping the no-arg default (same value at
both levels) unchanged since a real session file does write it that way.
backfill_openclaw_fixture.js now stages the assistant record with a
distinct envelope timestamp and asserts the projected row's
`message_created_at` carries the envelope value, genuinely exercising the
LLP 0158 / #555 envelope-first precedence.
Also strengthen the run-2 idempotency assertion, which checked only
`status === 'ok' && rows_written === 0` under a comment claiming "all
part_ids already present" but never pinned that the session was actually
re-read. A whole-file skip (e.g. the mtime-based quiesce-window skip LLP
0173 T12 will add, using the `mtimeMs` hook this PR introduced) would
report the same zero rows_written and pass identically. Require
`items_seen >= 1 && rows_skipped >= 1` too, so the predicate pins the
dedupe mechanism the comment describes instead of being satisfied
vacuously by a skip.
Co-Authored-By: Claude <noreply@anthropic.com>
* Pin the openclaw fixture's projection-log assertion to run 1's identity
Round-2 review nits on the tier-2 gate added for #543. Both are in the
`session_projected` telemetry assertion, and this file exists purely as a
regression gate, so assertion precision is its whole job.
The filter matched on the log body alone, the only one of the four
telemetry assertions not constrained to the run. Two records exist: the
idempotency rerun dispatches with a fresh `DEV_RUN_ID`, and the plugin
logger stamps it, so run 2 emits its own `session_projected` under
`<dev_run_id>-rerun`. Reading `projected[0]` was correct only because
emission order happens to put run 1 first, which is ordering rather than
identity. Constrain it to `harness.devRunId` and this session's native
`session_id`, matching what the three sibling filters already do.
The `identity_source === 'native'` conjunct could never be false: the
plugin hardcodes the string at `session_projected`'s single emission
site, so the conjunct held whenever the log existed at all, while the
comment read as though it separated a native-identity path from a
fallback one. Drop it, and state what the assertion does pin: that the
projection was reached, for this session, with both messages. The
identity claim now sits where it can genuinely vary - `session_id` on
the log, alongside the record lines' `message_id`s already asserted on
the rows.
No behavior change. Both round-2 mutations still fail the smoke: a flat
`openclawMessageEnvelope` reproduces #543's `status: ok` /
`rows_written: 0` on the first behavioral assertion, and swapping
`messageField`'s envelope/line order for `timestamp` fails the
envelope-precedence assertion with the record-line 10:00:02 value.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: test <test@test.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: test <test@example.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

neutral:approvedneutral reviewed this and holds it for a maintainer merge (own or adopted PR; LLP 0025/0030)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenClaw backfill projects 0 rows from real session files: message fields are read at the top level but OpenClaw nests them under 'message'

1 participant

@philcunliffe