ARROW-7744: [Java] Implement Flight JDBC Driver [WIP] - #6343

Closed
andygrove wants to merge 11 commits into
apache:masterfrom
andygrove:ARROW-7744
Closed

ARROW-7744: [Java] Implement Flight JDBC Driver [WIP]#6343
andygrove wants to merge 11 commits into
apache:masterfrom
andygrove:ARROW-7744

Conversation

@andygrove

@andygroveandygrove commented Feb 3, 2020•
edited
Loading

Copy link
Copy Markdown
Member

This is a starting point for the Flight JDBC driver. The following test passes, executing a query against the Rust flight-server example and collecting the results.

@TestpublicvoidexecuteQuery() throwsSQLException {
try (Connectionconn = driver.connect("jdbc:arrow://localhost:50051", newProperties())) {
try (Statementstmt = conn.createStatement()) {
try (ResultSetrs = stmt.executeQuery("SELECT id FROM alltypes_plain")) {
List<Integer> ids = newArrayList<>();
while (rs.next()) {
ids.add(rs.getInt(1));
}
assertEquals(ImmutableList.of(4, 5, 6, 7, 2, 3, 0, 1), ids);
}
}
}
}

@andygroveandygrove self-assigned this Feb 3, 2020
@andygroveandygrove changed the title ARROW-7744: [Java] Implement Flight JDBC DriverARROW-7744: [Java] Implement Flight JDBC Driver [WIP]Feb 3, 2020
@github-actions

github-actionsBot commented Feb 3, 2020

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-7744

@andygrove
andygrove marked this pull request as ready for review February 3, 2020 22:43
lidavidm
lidavidm reviewed Feb 9, 2020
View reviewed changes

@lidavidmlidavidm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some minor notes on the Flight stuff. Overall this is pretty interesting...

Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
private static final org.slf4j.Logger logger = LoggerFactory.getLogger(Driver.class);

/** JDBC connection string prefix. */
private static final String PREFIX = "jdbc:arrow://";

@lidavidmlidavidmFeb 9, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flight has a set of URI schemes (grpc+tcp, grpc+tls, etc), maybe those should be reflected?

@lidavidmlidavidmFeb 9, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then the Flight URI (minus the jdbc prefix) could get passed to the FlightConnection.

Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
@Override
public Connection connect(String url, Properties properties) throws SQLException {
logger.info("connect() url={}", url);
//TODO this needs much more work to parse full URLs but this is enough to get end to end tests running

@lidavidmlidavidmFeb 9, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

java.net.URI should be able to do all the parsing.

Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
@Override
public ResultSet executeQuery(String query) throws SQLException {

FlightClient client = FlightClient.builder()

@lidavidmlidavidmFeb 9, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally this client should be able to live in the driver instance and get shared between statements.

Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java

CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);

Ticket ticket = new Ticket(query.getBytes());

@lidavidmlidavidmFeb 9, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's been a "right way" to use Flight but I think so far things call GetFlightInfo to get a ticket, then DoGet (isn't that how Dremio uses Flight for their database-like product?)

@andygroveandygroveFeb 11, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I've been reading the Flight documentation again and I can see that this could make sense. However, this is all very specific to how the Flight server might implement support for arbitrary queries so I'm already running into questions about what makes sense to contribute to the Arrow repo versus just creating a JDBC driver directly for a particular Flight implementation.

Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
.location(Location.forGrpcInsecure(flightConnection.host, flightConnection.port))
.build();

CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);

@lidavidmlidavidmFeb 9, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this timeout is for the entire RPC, i.e. a query that takes more than 5 seconds (to stream all the data or something) would get cancelled. Unfortunately gRPC doesn't make it easy to have per-message timeouts.

@julianhyde

julianhyde commented Feb 13, 2020

Copy link
Copy Markdown

It's worth considering Calcite's Avatica as the basis for the JDBC driver.

Avatica's remote driver has a pluggable RPC layer - existing implementations are JSON over HTTP and Protobuf - and I think Flight would slot in as a third.

Avatica also has a local driver that shares a lot of code with the remote driver. That would allow people to switch between a Flight-based remote driver and a local driver, and see the same behavior.

Not directly relevant to this effort, but I would love to use Arrow (even without Flight) as the bulk data format for other Avatica-based drivers. There are certainly Calcite clients who would like to receive a batch of rows as an Arrow blob rather than making thousands of calls to ResultSet methods just to pull the data through the membrane. Any JDBC client could call resultSet.unwrap(ArrowBatch.class) just on the off-chance that their driver supports direct access to the Arrow data.

@andygrove

andygrove commented Feb 16, 2020

Copy link
Copy Markdown
MemberAuthor

@julianhyde Thanks for the info. This does sound interesting and it would be great to see Avatica benefit from Arrow.

liyafan82
liyafan82 reviewed Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java
public boolean next() throws SQLException {
if (batchIndex + 1 == root.getRowCount()) {
if (getNextBatch()) {
batchIndex++;

@liyafan82liyafan82Feb 19, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we should reset batchIndex to 0?

liyafan82
liyafan82 reviewed Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

@Override
public int getInt(String columnName) throws SQLException {
return ResultSetHelper.getInt(getObject(columnName));

@liyafan82liyafan82Feb 19, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are boxing/unboxing overheads here, which should be eliminated in the future.

liyafan82
liyafan82 reviewed Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

@Override
public BigDecimal getBigDecimal(String s) throws SQLException {
throw new SQLFeatureNotSupportedException();

@liyafan82liyafan82Feb 19, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this can also be implemented in a way similar to getBigDecimal(int i)?

@andygroveandygroveFeb 21, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes there are a lot of methods to be implemented still. I just wanted to share an early draft to start a discussion.

@andygrove

andygrove commented Feb 21, 2020

Copy link
Copy Markdown
MemberAuthor

@julianhyde Your proposal raises some interesting questions about dependencies between Avatica and Arrow. Should Arrow depend on Avatica or the other way around?

@andygrove

andygrove commented Mar 29, 2020

Copy link
Copy Markdown
MemberAuthor

I'm closing this PR for now since it is stale.

@asfimportasfimport mentioned this pull request Sep 30, 2022
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@liyafan82liyafan82liyafan82 left review comments

Assignees

@andygroveandygrove

Labels

Component: Java

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@andygrove@julianhyde@lidavidm@liyafan82
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-7744: [Java] Implement Flight JDBC Driver [WIP] - #6343
Closed
andygrove wants to merge 11 commits into
apache:masterfrom 
andygrove:ARROW-7744
Closed
ARROW-7744: [Java] Implement Flight JDBC Driver [WIP]#6343
andygrove wants to merge 11 commits into
apache:masterfrom 
andygrove:ARROW-7744
Conversation
@andygrove
@andygroveandygrove
commented
Feb 3, 2020•

edited

Loading

Copy link


Copy Markdown

Member
This is a starting point for the Flight JDBC driver. The following test passes, executing a query against the Rust flight-server example and collecting the results.
@TestpublicvoidexecuteQuery() throwsSQLException {
try (Connectionconn = driver.connect("jdbc:arrow://localhost:50051", newProperties())) {
try (Statementstmt = conn.createStatement()) {
try (ResultSetrs = stmt.executeQuery("SELECT id FROM alltypes_plain")) {
List<Integer> ids = newArrayList<>();
while (rs.next()) {
ids.add(rs.getInt(1));
}
assertEquals(ImmutableList.of(4, 5, 6, 7, 2, 3, 0, 1), ids);
}
}
}
}
@andygroveandygrove
self-assigned this
Feb 3, 2020
@andygroveandygrove
changed the title
ARROW-7744: [Java] Implement Flight JDBC DriverARROW-7744: [Java] Implement Flight JDBC Driver [WIP]Feb 3, 2020
@github-actions
github-actionsBot
commented
Feb 3, 2020

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-7744
@andygrove
andygrove
marked this pull request as ready for review
February 3, 2020 22:43
lidavidm
lidavidm
reviewed
Feb 9, 2020
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some minor notes on the Flight stuff. Overall this is pretty interesting...
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
 private static final org.slf4j.Logger logger = LoggerFactory.getLogger(Driver.class);

 /** JDBC connection string prefix. */
 private static final String PREFIX = "jdbc:arrow://";
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flight has a set of URI schemes (grpc+tcp, grpc+tls, etc), maybe those should be reflected?
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then the Flight URI (minus the jdbc prefix) could get passed to the FlightConnection.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
 @Override
 public Connection connect(String url, Properties properties) throws SQLException {
 logger.info("connect() url={}", url);
 //TODO this needs much more work to parse full URLs but this is enough to get end to end tests running
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

java.net.URI should be able to do all the parsing.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
 @Override
 public ResultSet executeQuery(String query) throws SQLException {

 FlightClient client = FlightClient.builder()
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally this client should be able to live in the driver instance and get shared between statements.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java

 CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);

 Ticket ticket = new Ticket(query.getBytes());
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's been a "right way" to use Flight but I think so far things call GetFlightInfo to get a ticket, then DoGet (isn't that how Dremio uses Flight for their database-like product?)
@andygroveandygroveFeb 11, 2020

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I've been reading the Flight documentation again and I can see that this could make sense. However, this is all very specific to how the Flight server might implement support for arbitrary queries so I'm already running into questions about what makes sense to contribute to the Arrow repo versus just creating a JDBC driver directly for a particular Flight implementation.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
 .location(Location.forGrpcInsecure(flightConnection.host, flightConnection.port))
 .build();

 CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this timeout is for the entire RPC, i.e. a query that takes more than 5 seconds (to stream all the data or something) would get cancelled. Unfortunately gRPC doesn't make it easy to have per-message timeouts.
@julianhyde
julianhyde
commented
Feb 13, 2020

Copy link


Copy Markdown

It's worth considering Calcite's Avatica as the basis for the JDBC driver.
Avatica's remote driver has a pluggable RPC layer - existing implementations are JSON over HTTP and Protobuf - and I think Flight would slot in as a third.
Avatica also has a local driver that shares a lot of code with the remote driver. That would allow people to switch between a Flight-based remote driver and a local driver, and see the same behavior.
Not directly relevant to this effort, but I would love to use Arrow (even without Flight) as the bulk data format for other Avatica-based drivers. There are certainly Calcite clients who would like to receive a batch of rows as an Arrow blob rather than making thousands of calls to ResultSet methods just to pull the data through the membrane. Any JDBC client could call resultSet.unwrap(ArrowBatch.class) just on the off-chance that their driver supports direct access to the Arrow data.
@andygrove
andygrove
commented
Feb 16, 2020

Copy link


Copy Markdown

MemberAuthor
@julianhyde Thanks for the info. This does sound interesting and it would be great to see Avatica benefit from Arrow.
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java
 public boolean next() throws SQLException {
 if (batchIndex + 1 == root.getRowCount()) {
 if (getNextBatch()) {
 batchIndex++;
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we should reset batchIndex to 0?
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

 @Override
 public int getInt(String columnName) throws SQLException {
 return ResultSetHelper.getInt(getObject(columnName));
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are boxing/unboxing overheads here, which should be eliminated in the future.
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

 @Override
 public BigDecimal getBigDecimal(String s) throws SQLException {
 throw new SQLFeatureNotSupportedException();
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this can also be implemented in a way similar to getBigDecimal(int i)?
@andygroveandygroveFeb 21, 2020

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes there are a lot of methods to be implemented still. I just wanted to share an early draft to start a discussion.
@andygrove
andygrove
commented
Feb 21, 2020

Copy link


Copy Markdown

MemberAuthor
@julianhyde Your proposal raises some interesting questions about dependencies between Avatica and Arrow. Should Arrow depend on Avatica or the other way around?
@andygrove
andygrove
commented
Mar 29, 2020

Copy link


Copy Markdown

MemberAuthor
I'm closing this PR for now since it is stale.
@asfimportasfimport
mentioned this pull request
Sep 30, 2022

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@liyafan82liyafan82liyafan82 left review comments

Assignees

@andygroveandygrove

Labels

Component: Java

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@andygrove@julianhyde@lidavidm@liyafan82

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-7744: [Java] Implement Flight JDBC Driver [WIP] - #6343
Closed
andygrove wants to merge 11 commits into
apache:masterfrom 
andygrove:ARROW-7744
Closed
ARROW-7744: [Java] Implement Flight JDBC Driver [WIP]#6343
andygrove wants to merge 11 commits into
apache:masterfrom 
andygrove:ARROW-7744
Conversation
@andygrove
@andygroveandygrove
commented
Feb 3, 2020•

edited

Loading

Copy link


Copy Markdown

Member
This is a starting point for the Flight JDBC driver. The following test passes, executing a query against the Rust flight-server example and collecting the results.
@TestpublicvoidexecuteQuery() throwsSQLException {
try (Connectionconn = driver.connect("jdbc:arrow://localhost:50051", newProperties())) {
try (Statementstmt = conn.createStatement()) {
try (ResultSetrs = stmt.executeQuery("SELECT id FROM alltypes_plain")) {
List<Integer> ids = newArrayList<>();
while (rs.next()) {
ids.add(rs.getInt(1));
}
assertEquals(ImmutableList.of(4, 5, 6, 7, 2, 3, 0, 1), ids);
}
}
}
}
@andygroveandygrove
self-assigned this
Feb 3, 2020
@andygroveandygrove
changed the title
ARROW-7744: [Java] Implement Flight JDBC DriverARROW-7744: [Java] Implement Flight JDBC Driver [WIP]Feb 3, 2020
@github-actions
github-actionsBot
commented
Feb 3, 2020

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-7744
@andygrove
andygrove
marked this pull request as ready for review
February 3, 2020 22:43
lidavidm
lidavidm
reviewed
Feb 9, 2020
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some minor notes on the Flight stuff. Overall this is pretty interesting...
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
 private static final org.slf4j.Logger logger = LoggerFactory.getLogger(Driver.class);

 /** JDBC connection string prefix. */
 private static final String PREFIX = "jdbc:arrow://";
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flight has a set of URI schemes (grpc+tcp, grpc+tls, etc), maybe those should be reflected?
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then the Flight URI (minus the jdbc prefix) could get passed to the FlightConnection.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
 @Override
 public Connection connect(String url, Properties properties) throws SQLException {
 logger.info("connect() url={}", url);
 //TODO this needs much more work to parse full URLs but this is enough to get end to end tests running
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

java.net.URI should be able to do all the parsing.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
 @Override
 public ResultSet executeQuery(String query) throws SQLException {

 FlightClient client = FlightClient.builder()
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally this client should be able to live in the driver instance and get shared between statements.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java

 CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);

 Ticket ticket = new Ticket(query.getBytes());
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's been a "right way" to use Flight but I think so far things call GetFlightInfo to get a ticket, then DoGet (isn't that how Dremio uses Flight for their database-like product?)
@andygroveandygroveFeb 11, 2020

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I've been reading the Flight documentation again and I can see that this could make sense. However, this is all very specific to how the Flight server might implement support for arbitrary queries so I'm already running into questions about what makes sense to contribute to the Arrow repo versus just creating a JDBC driver directly for a particular Flight implementation.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
 .location(Location.forGrpcInsecure(flightConnection.host, flightConnection.port))
 .build();

 CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this timeout is for the entire RPC, i.e. a query that takes more than 5 seconds (to stream all the data or something) would get cancelled. Unfortunately gRPC doesn't make it easy to have per-message timeouts.
@julianhyde
julianhyde
commented
Feb 13, 2020

Copy link


Copy Markdown

It's worth considering Calcite's Avatica as the basis for the JDBC driver.
Avatica's remote driver has a pluggable RPC layer - existing implementations are JSON over HTTP and Protobuf - and I think Flight would slot in as a third.
Avatica also has a local driver that shares a lot of code with the remote driver. That would allow people to switch between a Flight-based remote driver and a local driver, and see the same behavior.
Not directly relevant to this effort, but I would love to use Arrow (even without Flight) as the bulk data format for other Avatica-based drivers. There are certainly Calcite clients who would like to receive a batch of rows as an Arrow blob rather than making thousands of calls to ResultSet methods just to pull the data through the membrane. Any JDBC client could call resultSet.unwrap(ArrowBatch.class) just on the off-chance that their driver supports direct access to the Arrow data.
@andygrove
andygrove
commented
Feb 16, 2020

Copy link


Copy Markdown

MemberAuthor
@julianhyde Thanks for the info. This does sound interesting and it would be great to see Avatica benefit from Arrow.
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java
 public boolean next() throws SQLException {
 if (batchIndex + 1 == root.getRowCount()) {
 if (getNextBatch()) {
 batchIndex++;
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we should reset batchIndex to 0?
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

 @Override
 public int getInt(String columnName) throws SQLException {
 return ResultSetHelper.getInt(getObject(columnName));
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are boxing/unboxing overheads here, which should be eliminated in the future.
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

 @Override
 public BigDecimal getBigDecimal(String s) throws SQLException {
 throw new SQLFeatureNotSupportedException();
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this can also be implemented in a way similar to getBigDecimal(int i)?
@andygroveandygroveFeb 21, 2020

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes there are a lot of methods to be implemented still. I just wanted to share an early draft to start a discussion.
@andygrove
andygrove
commented
Feb 21, 2020

Copy link


Copy Markdown

MemberAuthor
@julianhyde Your proposal raises some interesting questions about dependencies between Avatica and Arrow. Should Arrow depend on Avatica or the other way around?
@andygrove
andygrove
commented
Mar 29, 2020

Copy link


Copy Markdown

MemberAuthor
I'm closing this PR for now since it is stale.
@asfimportasfimport
mentioned this pull request
Sep 30, 2022

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@liyafan82liyafan82liyafan82 left review comments

Assignees

@andygroveandygrove

Labels

Component: Java

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@andygrove@julianhyde@lidavidm@liyafan82

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-7744: [Java] Implement Flight JDBC Driver [WIP] - #6343
Closed
andygrove wants to merge 11 commits into
apache:masterfrom 
andygrove:ARROW-7744
Closed
ARROW-7744: [Java] Implement Flight JDBC Driver [WIP]#6343
andygrove wants to merge 11 commits into
apache:masterfrom 
andygrove:ARROW-7744
Conversation
@andygrove
@andygroveandygrove
commented
Feb 3, 2020•

edited

Loading

Copy link


Copy Markdown

Member
This is a starting point for the Flight JDBC driver. The following test passes, executing a query against the Rust flight-server example and collecting the results.
@TestpublicvoidexecuteQuery() throwsSQLException {
try (Connectionconn = driver.connect("jdbc:arrow://localhost:50051", newProperties())) {
try (Statementstmt = conn.createStatement()) {
try (ResultSetrs = stmt.executeQuery("SELECT id FROM alltypes_plain")) {
List<Integer> ids = newArrayList<>();
while (rs.next()) {
ids.add(rs.getInt(1));
}
assertEquals(ImmutableList.of(4, 5, 6, 7, 2, 3, 0, 1), ids);
}
}
}
}
@andygroveandygrove
self-assigned this
Feb 3, 2020
@andygroveandygrove
changed the title
ARROW-7744: [Java] Implement Flight JDBC DriverARROW-7744: [Java] Implement Flight JDBC Driver [WIP]Feb 3, 2020
@github-actions
github-actionsBot
commented
Feb 3, 2020

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-7744
@andygrove
andygrove
marked this pull request as ready for review
February 3, 2020 22:43
lidavidm
lidavidm
reviewed
Feb 9, 2020
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some minor notes on the Flight stuff. Overall this is pretty interesting...
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
 private static final org.slf4j.Logger logger = LoggerFactory.getLogger(Driver.class);

 /** JDBC connection string prefix. */
 private static final String PREFIX = "jdbc:arrow://";
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flight has a set of URI schemes (grpc+tcp, grpc+tls, etc), maybe those should be reflected?
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then the Flight URI (minus the jdbc prefix) could get passed to the FlightConnection.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
 @Override
 public Connection connect(String url, Properties properties) throws SQLException {
 logger.info("connect() url={}", url);
 //TODO this needs much more work to parse full URLs but this is enough to get end to end tests running
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

java.net.URI should be able to do all the parsing.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
 @Override
 public ResultSet executeQuery(String query) throws SQLException {

 FlightClient client = FlightClient.builder()
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally this client should be able to live in the driver instance and get shared between statements.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java

 CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);

 Ticket ticket = new Ticket(query.getBytes());
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's been a "right way" to use Flight but I think so far things call GetFlightInfo to get a ticket, then DoGet (isn't that how Dremio uses Flight for their database-like product?)
@andygroveandygroveFeb 11, 2020

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I've been reading the Flight documentation again and I can see that this could make sense. However, this is all very specific to how the Flight server might implement support for arbitrary queries so I'm already running into questions about what makes sense to contribute to the Arrow repo versus just creating a JDBC driver directly for a particular Flight implementation.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
 .location(Location.forGrpcInsecure(flightConnection.host, flightConnection.port))
 .build();

 CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this timeout is for the entire RPC, i.e. a query that takes more than 5 seconds (to stream all the data or something) would get cancelled. Unfortunately gRPC doesn't make it easy to have per-message timeouts.
@julianhyde
julianhyde
commented
Feb 13, 2020

Copy link


Copy Markdown

It's worth considering Calcite's Avatica as the basis for the JDBC driver.
Avatica's remote driver has a pluggable RPC layer - existing implementations are JSON over HTTP and Protobuf - and I think Flight would slot in as a third.
Avatica also has a local driver that shares a lot of code with the remote driver. That would allow people to switch between a Flight-based remote driver and a local driver, and see the same behavior.
Not directly relevant to this effort, but I would love to use Arrow (even without Flight) as the bulk data format for other Avatica-based drivers. There are certainly Calcite clients who would like to receive a batch of rows as an Arrow blob rather than making thousands of calls to ResultSet methods just to pull the data through the membrane. Any JDBC client could call resultSet.unwrap(ArrowBatch.class) just on the off-chance that their driver supports direct access to the Arrow data.
@andygrove
andygrove
commented
Feb 16, 2020

Copy link


Copy Markdown

MemberAuthor
@julianhyde Thanks for the info. This does sound interesting and it would be great to see Avatica benefit from Arrow.
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java
 public boolean next() throws SQLException {
 if (batchIndex + 1 == root.getRowCount()) {
 if (getNextBatch()) {
 batchIndex++;
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we should reset batchIndex to 0?
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

 @Override
 public int getInt(String columnName) throws SQLException {
 return ResultSetHelper.getInt(getObject(columnName));
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are boxing/unboxing overheads here, which should be eliminated in the future.
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

 @Override
 public BigDecimal getBigDecimal(String s) throws SQLException {
 throw new SQLFeatureNotSupportedException();
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this can also be implemented in a way similar to getBigDecimal(int i)?
@andygroveandygroveFeb 21, 2020

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes there are a lot of methods to be implemented still. I just wanted to share an early draft to start a discussion.
@andygrove
andygrove
commented
Feb 21, 2020

Copy link


Copy Markdown

MemberAuthor
@julianhyde Your proposal raises some interesting questions about dependencies between Avatica and Arrow. Should Arrow depend on Avatica or the other way around?
@andygrove
andygrove
commented
Mar 29, 2020

Copy link


Copy Markdown

MemberAuthor
I'm closing this PR for now since it is stale.
@asfimportasfimport
mentioned this pull request
Sep 30, 2022

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@liyafan82liyafan82liyafan82 left review comments

Assignees

@andygroveandygrove

Labels

Component: Java

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@andygrove@julianhyde@lidavidm@liyafan82

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-7744: [Java] Implement Flight JDBC Driver [WIP] - #6343
Closed
andygrove wants to merge 11 commits into
apache:masterfrom 
andygrove:ARROW-7744
Closed
ARROW-7744: [Java] Implement Flight JDBC Driver [WIP]#6343
andygrove wants to merge 11 commits into
apache:masterfrom 
andygrove:ARROW-7744
Conversation
@andygrove
@andygroveandygrove
commented
Feb 3, 2020•

edited

Loading

Copy link


Copy Markdown

Member
This is a starting point for the Flight JDBC driver. The following test passes, executing a query against the Rust flight-server example and collecting the results.
@TestpublicvoidexecuteQuery() throwsSQLException {
try (Connectionconn = driver.connect("jdbc:arrow://localhost:50051", newProperties())) {
try (Statementstmt = conn.createStatement()) {
try (ResultSetrs = stmt.executeQuery("SELECT id FROM alltypes_plain")) {
List<Integer> ids = newArrayList<>();
while (rs.next()) {
ids.add(rs.getInt(1));
}
assertEquals(ImmutableList.of(4, 5, 6, 7, 2, 3, 0, 1), ids);
}
}
}
}
@andygroveandygrove
self-assigned this
Feb 3, 2020
@andygroveandygrove
changed the title
ARROW-7744: [Java] Implement Flight JDBC DriverARROW-7744: [Java] Implement Flight JDBC Driver [WIP]Feb 3, 2020
@github-actions
github-actionsBot
commented
Feb 3, 2020

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-7744
@andygrove
andygrove
marked this pull request as ready for review
February 3, 2020 22:43
lidavidm
lidavidm
reviewed
Feb 9, 2020
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some minor notes on the Flight stuff. Overall this is pretty interesting...
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
 private static final org.slf4j.Logger logger = LoggerFactory.getLogger(Driver.class);

 /** JDBC connection string prefix. */
 private static final String PREFIX = "jdbc:arrow://";
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flight has a set of URI schemes (grpc+tcp, grpc+tls, etc), maybe those should be reflected?
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then the Flight URI (minus the jdbc prefix) could get passed to the FlightConnection.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
 @Override
 public Connection connect(String url, Properties properties) throws SQLException {
 logger.info("connect() url={}", url);
 //TODO this needs much more work to parse full URLs but this is enough to get end to end tests running
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

java.net.URI should be able to do all the parsing.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
 @Override
 public ResultSet executeQuery(String query) throws SQLException {

 FlightClient client = FlightClient.builder()
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally this client should be able to live in the driver instance and get shared between statements.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java

 CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);

 Ticket ticket = new Ticket(query.getBytes());
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's been a "right way" to use Flight but I think so far things call GetFlightInfo to get a ticket, then DoGet (isn't that how Dremio uses Flight for their database-like product?)
@andygroveandygroveFeb 11, 2020

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I've been reading the Flight documentation again and I can see that this could make sense. However, this is all very specific to how the Flight server might implement support for arbitrary queries so I'm already running into questions about what makes sense to contribute to the Arrow repo versus just creating a JDBC driver directly for a particular Flight implementation.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
 .location(Location.forGrpcInsecure(flightConnection.host, flightConnection.port))
 .build();

 CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this timeout is for the entire RPC, i.e. a query that takes more than 5 seconds (to stream all the data or something) would get cancelled. Unfortunately gRPC doesn't make it easy to have per-message timeouts.
@julianhyde
julianhyde
commented
Feb 13, 2020

Copy link


Copy Markdown

It's worth considering Calcite's Avatica as the basis for the JDBC driver.
Avatica's remote driver has a pluggable RPC layer - existing implementations are JSON over HTTP and Protobuf - and I think Flight would slot in as a third.
Avatica also has a local driver that shares a lot of code with the remote driver. That would allow people to switch between a Flight-based remote driver and a local driver, and see the same behavior.
Not directly relevant to this effort, but I would love to use Arrow (even without Flight) as the bulk data format for other Avatica-based drivers. There are certainly Calcite clients who would like to receive a batch of rows as an Arrow blob rather than making thousands of calls to ResultSet methods just to pull the data through the membrane. Any JDBC client could call resultSet.unwrap(ArrowBatch.class) just on the off-chance that their driver supports direct access to the Arrow data.
@andygrove
andygrove
commented
Feb 16, 2020

Copy link


Copy Markdown

MemberAuthor
@julianhyde Thanks for the info. This does sound interesting and it would be great to see Avatica benefit from Arrow.
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java
 public boolean next() throws SQLException {
 if (batchIndex + 1 == root.getRowCount()) {
 if (getNextBatch()) {
 batchIndex++;
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we should reset batchIndex to 0?
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

 @Override
 public int getInt(String columnName) throws SQLException {
 return ResultSetHelper.getInt(getObject(columnName));
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are boxing/unboxing overheads here, which should be eliminated in the future.
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

 @Override
 public BigDecimal getBigDecimal(String s) throws SQLException {
 throw new SQLFeatureNotSupportedException();
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this can also be implemented in a way similar to getBigDecimal(int i)?
@andygroveandygroveFeb 21, 2020

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes there are a lot of methods to be implemented still. I just wanted to share an early draft to start a discussion.
@andygrove
andygrove
commented
Feb 21, 2020

Copy link


Copy Markdown

MemberAuthor
@julianhyde Your proposal raises some interesting questions about dependencies between Avatica and Arrow. Should Arrow depend on Avatica or the other way around?
@andygrove
andygrove
commented
Mar 29, 2020

Copy link


Copy Markdown

MemberAuthor
I'm closing this PR for now since it is stale.
@asfimportasfimport
mentioned this pull request
Sep 30, 2022

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@liyafan82liyafan82liyafan82 left review comments

Assignees

@andygroveandygrove

Labels

Component: Java

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@andygrove@julianhyde@lidavidm@liyafan82

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-7744: [Java] Implement Flight JDBC Driver [WIP] - #6343
Closed
andygrove wants to merge 11 commits into
apache:masterfrom 
andygrove:ARROW-7744
Closed
ARROW-7744: [Java] Implement Flight JDBC Driver [WIP]#6343
andygrove wants to merge 11 commits into
apache:masterfrom 
andygrove:ARROW-7744
Conversation
@andygrove
@andygroveandygrove
commented
Feb 3, 2020•

edited

Loading

Copy link


Copy Markdown

Member
This is a starting point for the Flight JDBC driver. The following test passes, executing a query against the Rust flight-server example and collecting the results.
@TestpublicvoidexecuteQuery() throwsSQLException {
try (Connectionconn = driver.connect("jdbc:arrow://localhost:50051", newProperties())) {
try (Statementstmt = conn.createStatement()) {
try (ResultSetrs = stmt.executeQuery("SELECT id FROM alltypes_plain")) {
List<Integer> ids = newArrayList<>();
while (rs.next()) {
ids.add(rs.getInt(1));
}
assertEquals(ImmutableList.of(4, 5, 6, 7, 2, 3, 0, 1), ids);
}
}
}
}
@andygroveandygrove
self-assigned this
Feb 3, 2020
@andygroveandygrove
changed the title
ARROW-7744: [Java] Implement Flight JDBC DriverARROW-7744: [Java] Implement Flight JDBC Driver [WIP]Feb 3, 2020
@github-actions
github-actionsBot
commented
Feb 3, 2020

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-7744
@andygrove
andygrove
marked this pull request as ready for review
February 3, 2020 22:43
lidavidm
lidavidm
reviewed
Feb 9, 2020
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some minor notes on the Flight stuff. Overall this is pretty interesting...
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
 private static final org.slf4j.Logger logger = LoggerFactory.getLogger(Driver.class);

 /** JDBC connection string prefix. */
 private static final String PREFIX = "jdbc:arrow://";
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flight has a set of URI schemes (grpc+tcp, grpc+tls, etc), maybe those should be reflected?
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then the Flight URI (minus the jdbc prefix) could get passed to the FlightConnection.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
 @Override
 public Connection connect(String url, Properties properties) throws SQLException {
 logger.info("connect() url={}", url);
 //TODO this needs much more work to parse full URLs but this is enough to get end to end tests running
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

java.net.URI should be able to do all the parsing.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
 @Override
 public ResultSet executeQuery(String query) throws SQLException {

 FlightClient client = FlightClient.builder()
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally this client should be able to live in the driver instance and get shared between statements.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java

 CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);

 Ticket ticket = new Ticket(query.getBytes());
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's been a "right way" to use Flight but I think so far things call GetFlightInfo to get a ticket, then DoGet (isn't that how Dremio uses Flight for their database-like product?)
@andygroveandygroveFeb 11, 2020

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I've been reading the Flight documentation again and I can see that this could make sense. However, this is all very specific to how the Flight server might implement support for arbitrary queries so I'm already running into questions about what makes sense to contribute to the Arrow repo versus just creating a JDBC driver directly for a particular Flight implementation.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
 .location(Location.forGrpcInsecure(flightConnection.host, flightConnection.port))
 .build();

 CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this timeout is for the entire RPC, i.e. a query that takes more than 5 seconds (to stream all the data or something) would get cancelled. Unfortunately gRPC doesn't make it easy to have per-message timeouts.
@julianhyde
julianhyde
commented
Feb 13, 2020

Copy link


Copy Markdown

It's worth considering Calcite's Avatica as the basis for the JDBC driver.
Avatica's remote driver has a pluggable RPC layer - existing implementations are JSON over HTTP and Protobuf - and I think Flight would slot in as a third.
Avatica also has a local driver that shares a lot of code with the remote driver. That would allow people to switch between a Flight-based remote driver and a local driver, and see the same behavior.
Not directly relevant to this effort, but I would love to use Arrow (even without Flight) as the bulk data format for other Avatica-based drivers. There are certainly Calcite clients who would like to receive a batch of rows as an Arrow blob rather than making thousands of calls to ResultSet methods just to pull the data through the membrane. Any JDBC client could call resultSet.unwrap(ArrowBatch.class) just on the off-chance that their driver supports direct access to the Arrow data.
@andygrove
andygrove
commented
Feb 16, 2020

Copy link


Copy Markdown

MemberAuthor
@julianhyde Thanks for the info. This does sound interesting and it would be great to see Avatica benefit from Arrow.
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java
 public boolean next() throws SQLException {
 if (batchIndex + 1 == root.getRowCount()) {
 if (getNextBatch()) {
 batchIndex++;
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we should reset batchIndex to 0?
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

 @Override
 public int getInt(String columnName) throws SQLException {
 return ResultSetHelper.getInt(getObject(columnName));
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are boxing/unboxing overheads here, which should be eliminated in the future.
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

 @Override
 public BigDecimal getBigDecimal(String s) throws SQLException {
 throw new SQLFeatureNotSupportedException();
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this can also be implemented in a way similar to getBigDecimal(int i)?
@andygroveandygroveFeb 21, 2020

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes there are a lot of methods to be implemented still. I just wanted to share an early draft to start a discussion.
@andygrove
andygrove
commented
Feb 21, 2020

Copy link


Copy Markdown

MemberAuthor
@julianhyde Your proposal raises some interesting questions about dependencies between Avatica and Arrow. Should Arrow depend on Avatica or the other way around?
@andygrove
andygrove
commented
Mar 29, 2020

Copy link


Copy Markdown

MemberAuthor
I'm closing this PR for now since it is stale.
@asfimportasfimport
mentioned this pull request
Sep 30, 2022

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@liyafan82liyafan82liyafan82 left review comments

Assignees

@andygroveandygrove

Labels

Component: Java

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@andygrove@julianhyde@lidavidm@liyafan82

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-7744: [Java] Implement Flight JDBC Driver [WIP] - #6343
Closed
andygrove wants to merge 11 commits into
apache:masterfrom 
andygrove:ARROW-7744
Closed
ARROW-7744: [Java] Implement Flight JDBC Driver [WIP]#6343
andygrove wants to merge 11 commits into
apache:masterfrom 
andygrove:ARROW-7744
Conversation
@andygrove
@andygroveandygrove
commented
Feb 3, 2020•

edited

Loading

Copy link


Copy Markdown

Member
This is a starting point for the Flight JDBC driver. The following test passes, executing a query against the Rust flight-server example and collecting the results.
@TestpublicvoidexecuteQuery() throwsSQLException {
try (Connectionconn = driver.connect("jdbc:arrow://localhost:50051", newProperties())) {
try (Statementstmt = conn.createStatement()) {
try (ResultSetrs = stmt.executeQuery("SELECT id FROM alltypes_plain")) {
List<Integer> ids = newArrayList<>();
while (rs.next()) {
ids.add(rs.getInt(1));
}
assertEquals(ImmutableList.of(4, 5, 6, 7, 2, 3, 0, 1), ids);
}
}
}
}
@andygroveandygrove
self-assigned this
Feb 3, 2020
@andygroveandygrove
changed the title
ARROW-7744: [Java] Implement Flight JDBC DriverARROW-7744: [Java] Implement Flight JDBC Driver [WIP]Feb 3, 2020
@github-actions
github-actionsBot
commented
Feb 3, 2020

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-7744
@andygrove
andygrove
marked this pull request as ready for review
February 3, 2020 22:43
lidavidm
lidavidm
reviewed
Feb 9, 2020
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some minor notes on the Flight stuff. Overall this is pretty interesting...
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
 private static final org.slf4j.Logger logger = LoggerFactory.getLogger(Driver.class);

 /** JDBC connection string prefix. */
 private static final String PREFIX = "jdbc:arrow://";
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flight has a set of URI schemes (grpc+tcp, grpc+tls, etc), maybe those should be reflected?
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then the Flight URI (minus the jdbc prefix) could get passed to the FlightConnection.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
 @Override
 public Connection connect(String url, Properties properties) throws SQLException {
 logger.info("connect() url={}", url);
 //TODO this needs much more work to parse full URLs but this is enough to get end to end tests running
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

java.net.URI should be able to do all the parsing.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
 @Override
 public ResultSet executeQuery(String query) throws SQLException {

 FlightClient client = FlightClient.builder()
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally this client should be able to live in the driver instance and get shared between statements.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java

 CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);

 Ticket ticket = new Ticket(query.getBytes());
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's been a "right way" to use Flight but I think so far things call GetFlightInfo to get a ticket, then DoGet (isn't that how Dremio uses Flight for their database-like product?)
@andygroveandygroveFeb 11, 2020

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I've been reading the Flight documentation again and I can see that this could make sense. However, this is all very specific to how the Flight server might implement support for arbitrary queries so I'm already running into questions about what makes sense to contribute to the Arrow repo versus just creating a JDBC driver directly for a particular Flight implementation.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
 .location(Location.forGrpcInsecure(flightConnection.host, flightConnection.port))
 .build();

 CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this timeout is for the entire RPC, i.e. a query that takes more than 5 seconds (to stream all the data or something) would get cancelled. Unfortunately gRPC doesn't make it easy to have per-message timeouts.
@julianhyde
julianhyde
commented
Feb 13, 2020

Copy link


Copy Markdown

It's worth considering Calcite's Avatica as the basis for the JDBC driver.
Avatica's remote driver has a pluggable RPC layer - existing implementations are JSON over HTTP and Protobuf - and I think Flight would slot in as a third.
Avatica also has a local driver that shares a lot of code with the remote driver. That would allow people to switch between a Flight-based remote driver and a local driver, and see the same behavior.
Not directly relevant to this effort, but I would love to use Arrow (even without Flight) as the bulk data format for other Avatica-based drivers. There are certainly Calcite clients who would like to receive a batch of rows as an Arrow blob rather than making thousands of calls to ResultSet methods just to pull the data through the membrane. Any JDBC client could call resultSet.unwrap(ArrowBatch.class) just on the off-chance that their driver supports direct access to the Arrow data.
@andygrove
andygrove
commented
Feb 16, 2020

Copy link


Copy Markdown

MemberAuthor
@julianhyde Thanks for the info. This does sound interesting and it would be great to see Avatica benefit from Arrow.
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java
 public boolean next() throws SQLException {
 if (batchIndex + 1 == root.getRowCount()) {
 if (getNextBatch()) {
 batchIndex++;
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we should reset batchIndex to 0?
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

 @Override
 public int getInt(String columnName) throws SQLException {
 return ResultSetHelper.getInt(getObject(columnName));
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are boxing/unboxing overheads here, which should be eliminated in the future.
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

 @Override
 public BigDecimal getBigDecimal(String s) throws SQLException {
 throw new SQLFeatureNotSupportedException();
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this can also be implemented in a way similar to getBigDecimal(int i)?
@andygroveandygroveFeb 21, 2020

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes there are a lot of methods to be implemented still. I just wanted to share an early draft to start a discussion.
@andygrove
andygrove
commented
Feb 21, 2020

Copy link


Copy Markdown

MemberAuthor
@julianhyde Your proposal raises some interesting questions about dependencies between Avatica and Arrow. Should Arrow depend on Avatica or the other way around?
@andygrove
andygrove
commented
Mar 29, 2020

Copy link


Copy Markdown

MemberAuthor
I'm closing this PR for now since it is stale.
@asfimportasfimport
mentioned this pull request
Sep 30, 2022

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@liyafan82liyafan82liyafan82 left review comments

Assignees

@andygroveandygrove

Labels

Component: Java

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@andygrove@julianhyde@lidavidm@liyafan82

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-7744: [Java] Implement Flight JDBC Driver [WIP] - #6343
Closed
andygrove wants to merge 11 commits into
apache:masterfrom 
andygrove:ARROW-7744
Closed
ARROW-7744: [Java] Implement Flight JDBC Driver [WIP]#6343
andygrove wants to merge 11 commits into
apache:masterfrom 
andygrove:ARROW-7744
Conversation
@andygrove
@andygroveandygrove
commented
Feb 3, 2020•

edited

Loading

Copy link


Copy Markdown

Member
This is a starting point for the Flight JDBC driver. The following test passes, executing a query against the Rust flight-server example and collecting the results.
@TestpublicvoidexecuteQuery() throwsSQLException {
try (Connectionconn = driver.connect("jdbc:arrow://localhost:50051", newProperties())) {
try (Statementstmt = conn.createStatement()) {
try (ResultSetrs = stmt.executeQuery("SELECT id FROM alltypes_plain")) {
List<Integer> ids = newArrayList<>();
while (rs.next()) {
ids.add(rs.getInt(1));
}
assertEquals(ImmutableList.of(4, 5, 6, 7, 2, 3, 0, 1), ids);
}
}
}
}
@andygroveandygrove
self-assigned this
Feb 3, 2020
@andygroveandygrove
changed the title
ARROW-7744: [Java] Implement Flight JDBC DriverARROW-7744: [Java] Implement Flight JDBC Driver [WIP]Feb 3, 2020
@github-actions
github-actionsBot
commented
Feb 3, 2020

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-7744
@andygrove
andygrove
marked this pull request as ready for review
February 3, 2020 22:43
lidavidm
lidavidm
reviewed
Feb 9, 2020
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some minor notes on the Flight stuff. Overall this is pretty interesting...
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
 private static final org.slf4j.Logger logger = LoggerFactory.getLogger(Driver.class);

 /** JDBC connection string prefix. */
 private static final String PREFIX = "jdbc:arrow://";
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flight has a set of URI schemes (grpc+tcp, grpc+tls, etc), maybe those should be reflected?
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then the Flight URI (minus the jdbc prefix) could get passed to the FlightConnection.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/Driver.java
 @Override
 public Connection connect(String url, Properties properties) throws SQLException {
 logger.info("connect() url={}", url);
 //TODO this needs much more work to parse full URLs but this is enough to get end to end tests running
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

java.net.URI should be able to do all the parsing.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
 @Override
 public ResultSet executeQuery(String query) throws SQLException {

 FlightClient client = FlightClient.builder()
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally this client should be able to live in the driver instance and get shared between statements.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java

 CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);

 Ticket ticket = new Ticket(query.getBytes());
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there's been a "right way" to use Flight but I think so far things call GetFlightInfo to get a ticket, then DoGet (isn't that how Dremio uses Flight for their database-like product?)
@andygroveandygroveFeb 11, 2020

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I've been reading the Flight documentation again and I can see that this could make sense. However, this is all very specific to how the Flight server might implement support for arbitrary queries so I'm already running into questions about what makes sense to contribute to the Arrow repo versus just creating a JDBC driver directly for a particular Flight implementation.
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightStatement.java
 .location(Location.forGrpcInsecure(flightConnection.host, flightConnection.port))
 .build();

 CallOption callOptions = CallOptions.timeout(5, TimeUnit.SECONDS);
@lidavidmlidavidmFeb 9, 2020

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this timeout is for the entire RPC, i.e. a query that takes more than 5 seconds (to stream all the data or something) would get cancelled. Unfortunately gRPC doesn't make it easy to have per-message timeouts.
@julianhyde
julianhyde
commented
Feb 13, 2020

Copy link


Copy Markdown

It's worth considering Calcite's Avatica as the basis for the JDBC driver.
Avatica's remote driver has a pluggable RPC layer - existing implementations are JSON over HTTP and Protobuf - and I think Flight would slot in as a third.
Avatica also has a local driver that shares a lot of code with the remote driver. That would allow people to switch between a Flight-based remote driver and a local driver, and see the same behavior.
Not directly relevant to this effort, but I would love to use Arrow (even without Flight) as the bulk data format for other Avatica-based drivers. There are certainly Calcite clients who would like to receive a batch of rows as an Arrow blob rather than making thousands of calls to ResultSet methods just to pull the data through the membrane. Any JDBC client could call resultSet.unwrap(ArrowBatch.class) just on the off-chance that their driver supports direct access to the Arrow data.
@andygrove
andygrove
commented
Feb 16, 2020

Copy link


Copy Markdown

MemberAuthor
@julianhyde Thanks for the info. This does sound interesting and it would be great to see Avatica benefit from Arrow.
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java
 public boolean next() throws SQLException {
 if (batchIndex + 1 == root.getRowCount()) {
 if (getNextBatch()) {
 batchIndex++;
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we should reset batchIndex to 0?
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

 @Override
 public int getInt(String columnName) throws SQLException {
 return ResultSetHelper.getInt(getObject(columnName));
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are boxing/unboxing overheads here, which should be eliminated in the future.
liyafan82
liyafan82
reviewed
Feb 19, 2020
View reviewed changes
Comment threadjava/flight/flight-jdbc/src/main/java/org/apache/arrow/jdbc/FlightResultSet.java

 @Override
 public BigDecimal getBigDecimal(String s) throws SQLException {
 throw new SQLFeatureNotSupportedException();
@liyafan82liyafan82Feb 19, 2020

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems this can also be implemented in a way similar to getBigDecimal(int i)?
@andygroveandygroveFeb 21, 2020

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes there are a lot of methods to be implemented still. I just wanted to share an early draft to start a discussion.
@andygrove
andygrove
commented
Feb 21, 2020

Copy link


Copy Markdown

MemberAuthor
@julianhyde Your proposal raises some interesting questions about dependencies between Avatica and Arrow. Should Arrow depend on Avatica or the other way around?
@andygrove
andygrove
commented
Mar 29, 2020

Copy link


Copy Markdown

MemberAuthor
I'm closing this PR for now since it is stale.
@asfimportasfimport
mentioned this pull request
Sep 30, 2022

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@liyafan82liyafan82liyafan82 left review comments

Assignees

@andygroveandygrove

Labels

Component: Java

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@andygrove@julianhyde@lidavidm@liyafan82