Spark: [3/N] Support sending referenced-by to all endpoints by singhpk234 · Pull Request #13979 · apache/iceberg

singhpk234 · 2025-09-03T00:54:15Z

About the change

This provides a reference implementation for passing the view name that table is referenced in as part of which the loadTable call is being made.

Details on the spec change proposal and motivation here

SPEC: Add referenced-by in loadTable API #13810

gaborkaszab

Thanks for the PR @singhpk234 !
I took a look for my own benefit, and left some observations and questions meanwhile. Let me know what you think!

gaborkaszab · 2025-09-05T08:13:36Z


+  @Override
+  public Table loadTableViaView(TableIdentifier identifier, Map<String, Object> viewContext) {
+    if (delegate instanceof ContextAwareTableCatalog) {


delegate is a BaseSessionCatalog.AsCatalog that now implements ContextAwareTableCatalog. Is there a use-case or configuration where delegate doesn't implement the new interface?

I see the concept in this class is rather to have a new member of type ContextAwareTableCatalog that is converted from delegate and this new function could directly call that. Similarly as nsCatalog.

Would this also work here? Do I miss something?

github-actions · 2025-12-04T00:18:43Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

gaborkaszab

I went through the api/ and core/ part for now (haven't checked Spark). Left some questions and nits. I also found some of my previous comments so I'm not sure this was ready for review.
Anyway, some test coverage in TestRESTCatalog would be great!

chris922 · 2026-02-20T16:02:45Z

Hi all - the Iceberg REST spec was merged 😃 This PR is now the next step in getting this referenced-by feature into the different Iceberg clients, right? Looking forward to get it merged & released

bryanck · 2026-04-07T17:39:08Z

+          isStreaming,
+          timeTravelVersion,
+          timeTravelTimestamp) =>
+      val referencedBy = ViewUtil.buildReferencedByChain(viewChain, catalog.name())


I'm wondering if it's feasible to use the view UUID as the referenced by instead of the view name identifier? That might address issues around cross catalog access and having the catalog name in the identifier.

I gave it more thought, the thing is catalog don't index things from uuid, all our specification is based on identifiers / name, hence this would be a tricky, do you have any thoughts on server side implementation pov ?

…tualLoad Addresses review feedback from danielcweeks: - PR apache#15895: "generalize this to something more like SupportsContextualLoad and not make it so narrowly specific to this particular field" - PR apache#15895: "we should probably be adding some sort of LoadContext as opposed to specific parameters that are targeted to this one behavior" - PR apache#13979: "much prefer specific context objects" over Map<String, Object> Introduces a typed LoadContext with a builder pattern that can evolve with new fields (e.g., planId, credentials scope) without requiring new interfaces or method overloads for each addition.

RjLi13 · 2026-09-07T19:49:01Z

+ * the loading contract to accept a {@link LoadContext} that carries the view reference chain and
+ * other contextual information.
+ */
+public interface SparkSupportsContextualLoad {


will this be renamed to SparkSupportsLoadContext instead?

Append the pre-encoded referenced-by value verbatim in HTTPRequest.requestUri rather than handing it to URIBuilder.addParameter. The spec's wire form mixes encoding levels, a literal comma delimiter over percent-encoded entries, and nothing on the classpath produces it: addParameter double-encodes % under both encoding policies (%1F becomes %251F) and encodes the delimiter under ALL_RESERVED, setCustomQuery re-encodes % the same way, and URIBuilder's own formatQuery, which takes a BitSet of characters to leave alone, is package-private. Passing raw separators instead does not help, because then a %2C standing for a comma inside a view name is double-encoded while a raw one is indistinguishable from the delimiter. However many views the chain holds, they travel as one parameter. Encode namespace levels and the view name with PercentCodec, which produces the same bytes as the parent query parameter as the spec requires, and join them with the configured namespace separator as-is. URLEncoder renders a space as +, which RFC 3986 percent-decoding reads as a literal plus. Leave the ETag table cache untouched for view-mediated loads. Servers are already required to produce distinct ETags for responses that differ by query parameter, with the snapshots parameter given as the example, so a server whose response varies on referenced-by owes a distinct ETag and answers 200 rather than 304. Carry the chain to credential providers as the internal rest-referenced-by property, mirroring rest-scan-plan-id, rather than putting a wire parameter name into the config map, where it shared a namespace with server-returned keys and would override them. Add it inside tableFileIO instead of merging it into the server's response config, so that config alone still decides whether the catalog FileIO can be reused; merging it in made every view-mediated load build a FileIO by reflection. Views build no FileIO and tableSession filters config through an allow list, so loadView does not inject it at all. Pass the whole LoadContext down to loadInternal and tableFileIO rather than one field of it, so later context fields need no signature change and a caller with no context says so with LoadContext.empty(). Forward the contextual loads through the same delegate the rest of RESTCatalog uses, and override loadTable/loadView on the AsCatalog and AsViewCatalog bridges so the context is not dropped there. Test the encoder in TestRESTUtil next to the namespace encoders it follows, across the same separator forms, and add its separator validation to the test that already covers that message. Cover request building in TestHTTPRequest, and the chain through the catalog: that it reaches the table FileIO properties, that it leaves the catalog FileIO reusable when the server returns no table config, that the metadata-table retry against the base table still carries it, and that the credentials endpoint receives it single-encoded alongside planId. Fold the duplicated view-load tests into TestReferencedByQueryParam and reuse RESTUtil.merge rather than reimplementing it.

Resolve the chain of views a relation was reached through and pass it to the catalog as a LoadContext, so a REST catalog can forward it to the server. SparkSupportsLoadContext extends the loading contract with context-carrying overloads of loadTable and loadView, including the time-travel forms, and SparkCatalog and SparkSessionCatalog implement it. ResolveViews records the chain while resolving nested views, with UnResolvedRelationFromView carrying it through analysis. Cross-catalog view references fail rather than silently dropping the chain. Gated behind spark.sql.iceberg.referenced-by-enabled, off by default.

github-actions Bot added API spark core labels Sep 3, 2025

singhpk234 mentioned this pull request Sep 3, 2025

SPEC: Add referenced-by in loadTable API #13810

Merged

singhpk234 marked this pull request as ready for review September 5, 2025 01:04

gaborkaszab reviewed Sep 5, 2025

View reviewed changes

nandorKollar reviewed Sep 5, 2025

View reviewed changes

Comment thread api/src/main/java/org/apache/iceberg/catalog/ContextAwareTableCatalog.java Outdated

gaborkaszab reviewed Sep 5, 2025

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java Outdated

Comment thread core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java Outdated

singhpk234 force-pushed the feature/ref-imp-referenced-by branch 2 times, most recently from 97296ab to bcec3f2 Compare September 8, 2025 00:22

nastra reviewed Sep 22, 2025

View reviewed changes

Comment thread api/src/main/java/org/apache/iceberg/catalog/ContextAwareTableCatalog.java Outdated

nastra reviewed Sep 22, 2025

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java Outdated

nastra reviewed Sep 22, 2025

View reviewed changes

Comment thread ...ain/scala/org/apache/spark/sql/catalyst/plans/logical/views/UnResolvedRelationFromView.scala Outdated

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from a34c0f2 to fb6e824 Compare November 3, 2025 12:12

singhpk234 changed the title ~~Spark, Rest: Reference implementation of referenced-by in the loadTable call~~ [REST | SPARK]: Reference implementation of referenced-by in the loadTable call Nov 3, 2025

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from 20d30b3 to 737fc50 Compare November 3, 2025 12:31

github-actions Bot added the stale label Dec 4, 2025

singhpk234 added not-stale and removed stale labels Dec 4, 2025

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch 3 times, most recently from f368355 to 57e3e30 Compare January 11, 2026 05:39

gaborkaszab reviewed Jan 13, 2026

View reviewed changes

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from 57e3e30 to a300631 Compare February 21, 2026 19:05

github-actions Bot added AWS GCP AZURE labels Feb 21, 2026

singhpk234 changed the title ~~[REST | SPARK]: Reference implementation of referenced-by in the loadTable call~~ [REST | SPARK]: Reference implementation of referenced-by to all endpoints for DEFINER views Feb 21, 2026

singhpk234 requested a review from rdblue April 3, 2026 18:51

singhpk234 mentioned this pull request Apr 6, 2026

API: Add contextual load apis #15895

Merged

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch 2 times, most recently from 64d8981 to 59258c3 Compare April 7, 2026 17:17

bryanck reviewed Apr 7, 2026

View reviewed changes

RjLi13 reviewed May 11, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/rest/RESTUtil.java Outdated

singhpk234 marked this pull request as draft May 29, 2026 19:46

singhpk234 marked this pull request as ready for review May 29, 2026 19:47

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from 59258c3 to 32e0455 Compare May 30, 2026 17:17

RjLi13 reviewed Jun 4, 2026

View reviewed changes

Comment thread ....1/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from a4de8ca to dfcd362 Compare August 24, 2026 17:13

singhpk234 changed the title ~~[REST | SPARK]: Send referenced-by to all endpoints~~ Core,Spark: Send referenced-by to all endpoints Aug 24, 2026

singhpk234 requested review from RjLi13, bryanck, danielcweeks, ldsantos0911 and nastra August 24, 2026 18:20

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch 2 times, most recently from 3472ae0 to 9053739 Compare September 4, 2026 19:07

github-actions Bot removed the API label Sep 4, 2026

Core: Pass referenced-by view chain to catalogs via LoadContext

1035bdc

RjLi13 reviewed Sep 7, 2026

View reviewed changes

sfc-gh-prsingh added 2 commits September 8, 2026 09:42

sfc-gh-prsingh force-pushed the feature/ref-imp-referenced-by branch from 9053739 to 373d92e Compare September 8, 2026 21:42

singhpk234 requested review from RussellSpitzer and removed request for ldsantos0911 September 8, 2026 22:21

singhpk234 changed the title ~~Core,Spark: Send referenced-by to all endpoints~~ Spark: [3/N] Support sending referenced-by to all endpoints Sep 8, 2026

Uh oh!

Conversation

singhpk234 commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

About the change

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gaborkaszab Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Dec 4, 2025

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chris922 commented Feb 20, 2026

Uh oh!

bryanck Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

singhpk234 May 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

RjLi13 Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

singhpk234 commented Sep 3, 2025 •

edited

Loading