feat: support existing changelog incremental reads

[Pandas886]

Purpose

Linked issue: close#504

Depends on:#508 (Delta/Auto). Please review that one first — this branch is stacked on top of it, so the diff against main currently includes both commits.

This adds reading of already-written changelog files (changelog_manifest_list) plus an AuditLogTable view that exposes rowkind as the first column.

Important: this does not generate lookup/compact changelogs. If a snapshot has no changelog list, we just skip it / return empty for that snapshot rather than inventing files.

Brief change log

• Implement Changelog planning over (start, end]
• TableScan::plan_snapshot_changelog reusing the existing split pipeline
• AuditLogTable wrapper: rowkind first, optional _SEQUENCE_NUMBER
• TableRead::to_audit_log_arrow for Delta (all +I) and Changelog (from _VALUE_KIND)
• Auto → Changelog when producer is not none
• Tests with changelog-producer=input fixtures and multi-snapshot ranges

Diff mode is still unsupported here on purpose.

Tests

cargo test -p paimon --test incremental_batch_scan_test
cargo test -p paimon --test audit_log_table_test
cargo fmt --all -- --check
cargo clippy -p paimon --all-targets --features fulltext,vortex -- -D warnings

API and Format

• New AuditLogTable + Changelog path on the incremental API
• No storage format change; only reads existing changelog manifests/files

Documentation

No standalone docs in this PR (SQL docs come with the DataFusion follow-up).

Review tip

If you're looking at the full branch vs main, the interesting commit for this PR is the top one (feat: support existing changelog incremental reads). The parent is #508.

[JingsongLi]

Please resolve conflicts.

[JingsongLi]

The Delta audit at table_read.rs:243 calls audit_raw_stream(plan, false) and then hard-codes all rowkinds to +I. Even when changelog-producer=none is set for the PK table, Delta files still contain _VALUE_KIND; Java incremental audit tests explicitly require types such as -D to be preserved, so delete/update records are incorrectly reported as inserts. It is recommended that PK Delta also read _VALUE_KIND and add regression tests for DELETE/UPDATE operations. Additionally, the rowkind field uses the ID -1 and varchar(8), which is inconsistent with Java’s Integer.MAX_VALUE-4 and the maximum varchar length; synchronization is recommended. The existing incremental/audit/exception rowkind tests have passed 16 out of 16, but do not cover Delta delete records.

[Pandas886]

Fixed in 2f72b51 — PK Delta audit now preserves _VALUE_KIND, and rowkind matches Java’s field ID and varchar length. Added regression coverage too. Thanks for catching this.

[JingsongLi]

+1