Skip to content

Add new delta_sharing node-type - #997

Open
riteshk001 wants to merge 32 commits into
masterfrom
node-delta-sharing
Open

Add new delta_sharing node-type#997
riteshk001 wants to merge 32 commits into
masterfrom
node-delta-sharing

Conversation

@riteshk001

@riteshk001riteshk001 commented Jan 26, 2026

Copy link
Copy Markdown
Collaborator

A new node-type delta sharing is implemented to allow the user to fetch, read and write Apache tables using the Delta Sharing protocol.

Features:

  • Fetch metadata from the delta sharing server (query metadata, list schemas, list shares, list tables).
  • Download Apache Parquet tables to a local cache.
  • Requires a profile file to identify the share, schema and table and the bearer token.

I have used magpierre/delta-sharing, which provides C++ based API for accessing Delta Shares natively in C++.

@al3xa23

riteshk001and others added 30 commits November 28, 2025 12:11
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
…ASHARING' is not set
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
…onfig
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
…ables
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Alexandra <alexandra.bach@eonerc.rwth-aachen.de>
…delines
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
…d ubuntu dockerfiles by providing .deb before installation
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
…debian and ubuntu
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
…debian and ubuntu
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
…debian and ubuntu
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
…rrow in Ubuntu dockerfile.
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
…rrow in Ubuntu dockerfile.
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
…rrow in Ubuntu dockerfile.
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
…erfiles
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
@riteshk001
riteshk001 marked this pull request as ready for review April 27, 2026 10:45
…bian and fedora
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
Signed-off-by: Ritesh.K <riteshkarki6@gmail.com>
@riteshk001

Copy link
Copy Markdown
CollaboratorAuthor

There is a version mismatch of the Apache Arrow packages in Debian and Fedora based package repositories. Debian has the newer version and Fedora 42 has the older version. There are some breaking changes in Apache Parquet because of this, due to which I am installing from source (in deps.sh) for non-Debian based Dockerfiles.

@al3xa23
al3xa23 requested a review from fuentesgrauJuly 29, 2026 08:22
@steffen-vogel-opalrtsteffen-vogel-opalrt changed the title feat (node-delta-sharing): Create new node-type delta_sharing. Add new node-type delta_sharingAug 3, 2026
@steffen-vogel-opalrtsteffen-vogel-opalrt changed the title Add new node-type delta_sharingAdd new delta_sharing node-typeAug 3, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@riteshk001@al3xa23