Uh oh!
There was an error while loading. Please reload this page.
[core] Keep raw-value pushdown off masked columns in query auth - #8582
Closed
plusplusjiajia wants to merge 2 commits into
Closed
[core] Keep raw-value pushdown off masked columns in query auth#8582plusplusjiajia wants to merge 2 commits into
plusplusjiajia wants to merge 2 commits into
Conversation
plusplusjiajiaforce-pushed
the
core-mask-pushdown-safety
branch
3 times, most recently
from
July 13, 2026 02:21
67f242b to
51f3b9dCompareA cross-column mask (e.g. display := concat_ws('-', first, last)) threw
at read time when the query projected the masked target but not the
mask's input columns: "Column masking refers to field 'first' which is
not present in output row type".
Row-filter operands are already added to the read projection and
projected back out (apache#8447); this does the same for column-mask inputs.
TableQueryAuthResult.requiredAuthFields collects filter operands plus,
transitively, the inputs of every mask whose target is readable; the
widening appends the missing table columns as-is (preserving nested
pruning), and the scan pushes the widened read type before planning so
file-level column pruning (e.g. data evolution) keeps the files those
columns live in.
Stale rules (columns absent from the latest schema, e.g. after a rename
or drop) fail closed once at plan time in AbstractDataTableScan.authQuery,
the common path of every scan; re-validated only when the fetched rules
change. Projected system fields (e.g. _ROW_ID) stay valid, unprojected
ones are rejected; rules touching a nested-pruned or unprojected
blob-view column are rejected too.
The read schema is fixed once the first split reader exists (split reads
cache their format readers and ignore later read-type changes); every
split is projected back to the query's read type, and masks apply only
to columns readable from the query, so a column merely retained in a
previously widened schema does not activate a mask after a rules change.
Previously the first createReader call mutated this.readType, so later
splits of the same TableRead leaked the auth-added columns -- also on
the pre-existing row-filter path.
Related re-configuration fixes: MergeFileSplitRead.withReadType and
DataEvolutionFileStoreScan.withReadType reset stale projections, and
IncrementalDiffSplitRead projects from the merge read's actual output
type instead of the full table row type.
Raw-value pushdown safety on masked columns (filter/TopN/limit stats
pruning) is deliberately left to a follow-up PR.
Behavior changes are scoped to query-auth.enabled tables. Tables without
query auth and the write path are unaffected. System tables and
vector/full-text search reads do not consult auth rules (pre-existing;
restrict at the permission layer).A column mask changes the value domain of its target column. A predicate on a masked column therefore evaluates on the masked value (view semantics): matching the raw value would reveal it. This makes every optimization that consumes the column's raw values or raw statistics unsafe on masked columns -- file/split pruning by min-max or partition value, TopN selection, limit pushdown and reader-level row filtering would drop or leak rows with respect to what the query is entitled to see. Stacks on the projection-augmentation change, and follows the pattern ReadBuilderImpl already uses for read-level TopN on query-auth-enabled tables: never hand unsafe state to the reader, rather than pushing it and retracting it later. Scan side: the query filter is deferred to plan()/listPartitionEntries and pushed once through the two-argument SnapshotReader.withFilter. The full filter marks read-time filtering, and only the conjuncts free of masked columns feed statistics and partition pruning. Since masked conjuncts drop rows only at read time, their presence -- even partition-only -- also keeps limit/TopN split pruning off; TopN split pruning additionally skips masked ordering columns. A mask appearing on an already-pushed filter column fails closed, because raw statistics were already consumed. Read side: on query-auth-enabled tables the filter is stored (for executeFilter) but not forwarded to the reader internals; engines re-evaluate data filters on the masked output. The conjuncts on masked columns are evaluated inside the auth read, post-mask, since engines do not re-evaluate the partition filters they consumed. When the query does not project a masked filter column, the read schema is widened by it (same machinery as mask inputs) and projected back out afterwards. Spark statistics-based aggregate pushdown already degrades for authed reads (auth splits are not DataSplits), anchored by a regression test. Scoped to query-auth.enabled tables; reader-level filter pushdown is conservatively off for them even without mask rules, matching the existing read-level TopN behavior.
plusplusjiajiaforce-pushed
the
core-mask-pushdown-safety
branch
from
July 13, 2026 03:06
51f3b9d to
986fa71Compareplusplusjiajia
commented
Jul 13, 2026
MemberAuthor
Folded into #8570 per review. Closing. |
JingsongLi pushed a commit
that referenced
this pull request
Aug 30, 2026
### Purpose Makes column masking correct on the read path of `query-auth.enabled` tables. Folds in #8582 as asked in the first review; #8582 is closed. Two bugs: a cross-column mask threw `Column masking refers to field 'first' which is not present in output row type` when the query projected the masked target but not the mask's inputs; and a predicate on a masked column was pushed into raw statistics, where it matched the raw value rather than the masked one, returning an empty result. Mask inputs are now widened into the read projection and projected back out, as row-filter operands already are (#8447), transitively and preserving nested pruning. The widened type is pushed before planning so column-file pruning keeps the files those columns live in. The read schema is then fixed once the first split reader exists — split reads cache their format readers, so without this the auth-added columns leaked into later splits of the same `TableRead`, which also affected the pre-existing row-filter path. `MergeFileSplitRead`, `DataEvolutionFileStoreScan` and `IncrementalDiffSplitRead` reset stale projections on re-configuration accordingly. The query filter is deferred to `plan()`: only the conjuncts free of masked columns feed statistics and partition pruning, and the masked ones are evaluated inside the auth read, post-mask. Rules referencing a column absent from the latest schema fail closed at plan time (projected system fields such as `_ROW_ID` stay valid), and rules bind by field id, so a dropped and re-added column of the same name cannot make a time-travel read apply a rule to unrelated data. **Behaviour changes.** Each is a case where the mask cannot be enforced, so it fails closed — accepting the query would leave the rules silently inert, which is worse than an error. | Change | Why | |---|---| | `t$files`, `t$file_key_ranges`, `t$binlog` rejected | They publish per-column min/max raw values. `t$audit_log` and `t$ro` read through the masking reader and keep working. | | Vector / full-text / hybrid search rejected | The indexes rank raw values. | | `LocalTableQuery` (lookup join) rejected | Serves rows straight from the lookup cache. | | Partition predicate on a masked partition key rejected | Pruning consumes it and Spark drops it from post-scan evaluation. The same predicate through `withFilter` still works. | | `query-auth.enabled` rejected on a non-file-store table | Their reads never reach the auth reader. Checked before and after catalog table defaults. | | A mask reading another masked column rejected | Transforms evaluate on the raw row. Masking a column with itself stays valid. | | Global index not consulted | Ranks raw values; falls back to a full scan. Performance, not correctness. | | Read-level limit applied only after the filter | Engines re-apply the filter afterwards, so capping first dropped matching rows. | **Known limitations.** The list of bypass paths is not proven complete: the guard lives on `AbstractDataTableScan`, but `DataEvolutionBatchScan` is a sibling and `PrimaryKeyBatchScan` keeps a private copy of the filter — both found by review, not by design. Reviewers who know the global-index and data-evolution code should take a second look. Not handled here: - **Branch reads.** `t$branch_x` asks the catalog for its own identifier, and paimon does not enforce that it inherits the main table's rules; whether they apply is up to the catalog. - **The physical row count** carried on a split is still visible under a row filter. - **Partition metadata.** `$partitions`, `$buckets` and `$manifests`, and the global `all_partitions` / `all_tables`, report partition values straight from the manifest, so a masked partition key shows its raw value there. None of them reads through the auth reader. Rejecting them would also block tables that enable query auth without masking a partition key, which is the common case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #8570 — review the top commit only; will be rebased once #8570 merges.
Purpose
A column mask changes the value domain of its target column. A predicate on a masked column therefore evaluates on the masked value (view semantics, as in Snowflake/Ranger): matching the raw value would reveal
it. This makes every optimization that consumes the column's raw values or raw statistics unsafe on masked columns — file/split pruning by min-max or partition value, TopN selection, limit pushdown, and
reader-level row filtering would drop or leak rows relative to what the query is entitled to see. This is a pre-existing issue of the merged masking feature (#8447/#8458), independent of #8570.
The design follows the pattern ReadBuilderImpl already uses for read-level TopN on query-auth.enabled tables: never hand unsafe state to the reader, rather than pushing it and retracting it later.
free of masked columns feed statistics and partition pruning. Masked conjuncts (even partition-only ones) keep limit/TopN split pruning off; TopN split pruning also skips masked ordering columns. A mask
appearing on an already-pushed filter column fails closed, since raw statistics were already consumed.
are evaluated inside the auth read, post-mask — engines do not re-evaluate the partition filters they consumed. A masked filter column the query does not project is widened into the read schema (same machinery
as mask inputs in [core] Fix column masking correctness in query-auth reads #8570) and projected back out.
Scoped to query-auth.enabled tables: they conservatively lose reader-level filter pushdown even without mask rules, matching the existing read-level TopN behavior; scan-level pruning on safe conjuncts is
retained. Tables without query auth are unaffected.
Tests
partition-only filter, masked pk filter not applied on raw key ranges, mask growth on a pushed filter column fails the scan closed while a fresh scan + existing reader return masked-correct results,
TopN/limit/stats-pruning disablement.