ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests - #11989

Closed
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom
rafael-telles:flight-sql-integration-tests-2
Closed

ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests#11989
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom
rafael-telles:flight-sql-integration-tests-2

Conversation

@rafael-telles

@rafael-tellesrafael-telles commented Dec 17, 2021•
edited
Loading

Copy link
Copy Markdown
Contributor

This adds Flight SQL scenarios for integration tests for C++ and Java implementations.
The integrations tests assert that:

  • RPC objects built on clients are parsed correctly on servers
  • Arrow vectors built on servers have the expected Arrow schemas when received on clients

Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)

While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.

@github-actions

github-actionsBot commented Dec 17, 2021

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-15112

@rafael-telles

rafael-telles commented Dec 17, 2021

Copy link
Copy Markdown
ContributorAuthor

Hi @lidavidm ! We wrote the integration tests for Flight SQL as requested :)

lidavidm
lidavidm reviewed Dec 17, 2021
View reviewed changes

@lidavidmlidavidm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround, this looks good overall. I left some comments.

Comment threadcpp/src/arrow/flight/integration_tests/CMakeLists.txt Outdated

@lidavidmlidavidmDec 17, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the target necessary?

@rafael-tellesrafael-tellesDec 20, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was already there on arrow/flight/CMakeLists.txt before moving to the new directory

Comment threadcpp/src/arrow/CMakeLists.txt Outdated
Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc Outdated
Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc Outdated
Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc Outdated
Comment threadcpp/src/arrow/flight/integration_tests/test_integration.h Outdated
Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java Outdated
Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java Outdated
Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java Outdated
@lidavidm

lidavidm commented Dec 20, 2021

Copy link
Copy Markdown
Member

We need to exclude Go and Rust from the new test scenario.

Also, the PR needs to be formatted (see https://arrow.apache.org/docs/developers/cpp/development.html#code-style-linting-and-ci, note that we use ClangTools 12 now not version 8)

@rafael-telles

rafael-telles commented Dec 20, 2021

Copy link
Copy Markdown
ContributorAuthor

Done @lidavidm

@rafael-telles

rafael-telles commented Dec 20, 2021

Copy link
Copy Markdown
ContributorAuthor

Hey @lidavidm , I don't know if these CI failures has something to do with these changes... Please let me know if there is anything else to do here

@lidavidm

lidavidm commented Dec 20, 2021

Copy link
Copy Markdown
Member

There's some lint failures:


[1/1] cd /tmp/arrow-lint-6fb2hzaq/cpp-build && /usr/local/bin/python /arrow/cpp/build-support/lint_cpp_cli.py /arrow/cpp/src
INFO:archery:Running Python formatter (autopep8)
INFO:archery:Running Python linter (flake8)
/arrow/dev/archery/archery/integration/tester_java.py:52:80: E501 line too long (95 > 79 characters)
/arrow/dev/archery/archery/integration/runner.py:387:80: E501 line too long (82 > 79 characters)
INFO:archery:Running cmake-format linters
ERROR __main__.py:618: Check failed: /arrow/cpp/src/arrow/CMakeLists.txt

I think the other one is unrelated, but could you rebase on master to see if it goes away?

@rafael-telles
rafael-tellesforce-pushed the flight-sql-integration-tests-2 branch 2 times, most recently from 94f98fe to 5866b91CompareDecember 20, 2021 22:33
@rafael-telles
rafael-tellesforce-pushed the flight-sql-integration-tests-2 branch from 5866b91 to 78352e9CompareDecember 20, 2021 22:37
@rafael-telles

rafael-telles commented Dec 20, 2021

Copy link
Copy Markdown
ContributorAuthor

Ops @lidavidm ... Sorry, I don't know if I messed up when git rebase master or if also need to run it in the base branch flight-sql...

@lidavidm

lidavidm commented Dec 20, 2021

Copy link
Copy Markdown
Member

Ah, right, we have a branch…How about I rebase the branch, and then we can rebase the PRs?

@lidavidm

lidavidm commented Dec 20, 2021

Copy link
Copy Markdown
Member

Hmm. I just rebased the branch…so do you mind trying to rebase again here? Sorry for the trouble.

@rafael-telles
rafael-tellesforce-pushed the flight-sql-integration-tests-2 branch from 78352e9 to 97fe3ceCompareDecember 20, 2021 23:12
@rafael-telles

rafael-telles commented Dec 20, 2021

Copy link
Copy Markdown
ContributorAuthor

No problem! Just rebased :)

@lidavidm

lidavidm commented Dec 20, 2021

Copy link
Copy Markdown
Member

Well, looks like the Docker registry isn't doing so well right now. I'll kick all the CI pipelines in a bit to try again.

@lidavidm

lidavidm commented Dec 21, 2021

Copy link
Copy Markdown
Member

Hmm, I wonder if the test failure in Dev is just because it's getting confused since we're on a branch. I think we can resolve that when we make the final PR from flight-sql into master, then.

lidavidm
lidavidm approved these changes Dec 21, 2021
View reviewed changes

@lidavidmlidavidm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just one more nit that caught my eye.

Comment thread...gration-tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenario.java Outdated
sqlClient.execute("SELECT STATEMENT", options), sqlClient);

IntegrationAssertions.assertEquals(sqlClient.executeUpdate("UPDATE STATEMENT", options),
10000L);

@lidavidmlidavidmDec 21, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit, but can we move these numbers to constants so it's clear what they represent? (UPDATE_STATEMENT_EXPECTED_ROWS or something)

@lidavidmlidavidmDec 21, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The same goes for the producer, C++, etc.)

@rafael-tellesrafael-tellesDec 21, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Done

@rafael-telles

rafael-telles commented Dec 21, 2021

Copy link
Copy Markdown
ContributorAuthor

Hey @lidavidm ! Are we good to merge this on flight-sql branch and then on master?

Thank you

@lidavidm

lidavidm commented Dec 21, 2021

Copy link
Copy Markdown
Member

We should be. I just kicked off CI.

@lidavidm

lidavidm commented Dec 21, 2021

Copy link
Copy Markdown
Member

I think the failures are spurious. (I filed ARROW-15181 for the crash in the Flight tests.)

lidavidm
lidavidm approved these changes Dec 21, 2021
View reviewed changes

@lidavidmlidavidm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Sorry, I just noticed a couple more small things - we can merge into the branch next.

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc Outdated
Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc Outdated
@rafael-telles

rafael-telles commented Dec 21, 2021

Copy link
Copy Markdown
ContributorAuthor

Glad to see everything passing! Except to that "Dev/ Source Release and Merge Script", but it indeed seem to fail because we are in our own branch

lidavidm pushed a commit that referenced this pull request Dec 21, 2021
@rafael-telles@lidavidm
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integratio…
16a9df1
…n tests
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.
The integrations tests assert that:
- RPC objects built on clients are parsed correctly on servers
- Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
Closes#11989 from rafael-telles/flight-sql-integration-tests-2
Authored-by: Rafael Telles <rafael@telles.dev>
Signed-off-by: David Li <li.davidm96@gmail.com>
@lidavidmlidavidm mentioned this pull request Dec 21, 2021
Closed
@lidavidm

lidavidm commented Dec 21, 2021

Copy link
Copy Markdown
Member

Mega-PR is up at #12013

@asfimportasfimport mentioned this pull request Dec 23, 2021
Closed
pribor pushed a commit to GlobalWebIndex/arrow that referenced this pull request Oct 24, 2025
@rafael-tellesAbner Eduardo Ferreira
@jduo@jcralmeida@JrJuscelino@kylep-dremioRyan Nicholson@vfraga@tifflhl@kou
ARROW-14421: [C++] Implement Flight SQL
95460a1
Implement Flight SQL in C++ and Java, and add the protocol definitions.
This is a combination of multiple pull requests, merged into one branch before merging into master.
Closesapache#10906 (ARROW-12922).
Closesapache#11507 (ARROW-14421).
Closesapache#11989 (ARROW-15112).
Closesapache#12021 (ARROW-15187).
Closesapache#12035 (ARROW-15198).
Closesapache#12013 from apache/flight-sql
Lead-authored-by: Rafael Telles <rafael@telles.dev>
Co-authored-by: Abner Eduardo Ferreira <abenaru@protonmail.ch>
Co-authored-by: James Duong <duong.james@gmail.com>
Co-authored-by: Jose Almeida <almeidajcr90@gmail.com>
Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br>
Co-authored-by: Kyle Porter <kporter@dremio.com>
Co-authored-by: Ryan Nicholson <rnicholson@dremio.com>
Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com>
Co-authored-by: tifflhl <tiffanylamhl@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes

Assignees

No one assigned

Labels

Component: C++Component: FlightRPCComponent: GLibComponent: JavaComponent: ParquetComponent: PythonComponent: R

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rafael-telles@lidavidm
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests - #11989
Closed
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom 
rafael-telles:flight-sql-integration-tests-2
Closed
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests#11989
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom 
rafael-telles:flight-sql-integration-tests-2
Conversation
@rafael-telles
@rafael-tellesrafael-telles
commented
Dec 17, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.

The integrations tests assert that:
  • RPC objects built on clients are parsed correctly on servers
  • Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
@github-actions
github-actionsBot
commented
Dec 17, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-15112
@rafael-telles
rafael-telles
commented
Dec 17, 2021

Copy link


Copy Markdown

ContributorAuthor
Hi @lidavidm ! We wrote the integration tests for Flight SQL as requested :)
lidavidm
lidavidm
reviewed
Dec 17, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround, this looks good overall. I left some comments.
Comment threadcpp/src/arrow/flight/integration_tests/CMakeLists.txt
Outdated

@lidavidmlidavidmDec 17, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the target necessary?
@rafael-tellesrafael-tellesDec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was already there on arrow/flight/CMakeLists.txt before moving to the new directory
Comment threadcpp/src/arrow/CMakeLists.txt
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.h
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
We need to exclude Go and Rust from the new test scenario.
Also, the PR needs to be formatted (see https://arrow.apache.org/docs/developers/cpp/development.html#code-style-linting-and-ci, note that we use ClangTools 12 now not version 8)
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Done @lidavidm
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Hey @lidavidm , I don't know if these CI failures has something to do with these changes... Please let me know if there is anything else to do here
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
There's some lint failures:

[1/1] cd /tmp/arrow-lint-6fb2hzaq/cpp-build && /usr/local/bin/python /arrow/cpp/build-support/lint_cpp_cli.py /arrow/cpp/src
INFO:archery:Running Python formatter (autopep8)
INFO:archery:Running Python linter (flake8)
/arrow/dev/archery/archery/integration/tester_java.py:52:80: E501 line too long (95 > 79 characters)
/arrow/dev/archery/archery/integration/runner.py:387:80: E501 line too long (82 > 79 characters)
INFO:archery:Running cmake-format linters
ERROR __main__.py:618: Check failed: /arrow/cpp/src/arrow/CMakeLists.txt

I think the other one is unrelated, but could you rebase on master to see if it goes away?
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
2 times, most recently
from
94f98fe to
5866b91CompareDecember 20, 2021 22:33
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
from
5866b91 to
78352e9CompareDecember 20, 2021 22:37
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Ops @lidavidm ... Sorry, I don't know if I messed up when git rebase master or if also need to run it in the base branch flight-sql...
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Ah, right, we have a branch…How about I rebase the branch, and then we can rebase the PRs?
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Hmm. I just rebased the branch…so do you mind trying to rebase again here? Sorry for the trouble.
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
from
78352e9 to
97fe3ceCompareDecember 20, 2021 23:12
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
No problem! Just rebased :)
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Well, looks like the Docker registry isn't doing so well right now. I'll kick all the CI pipelines in a bit to try again.
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
Hmm, I wonder if the test failure in Dev is just because it's getting confused since we're on a branch. I think we can resolve that when we make the final PR from flight-sql into master, then.
lidavidm
lidavidm
approved these changes
Dec 21, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just one more nit that caught my eye.
Comment thread...gration-tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenario.java
Outdated

 sqlClient.execute("SELECT STATEMENT", options), sqlClient);

 IntegrationAssertions.assertEquals(sqlClient.executeUpdate("UPDATE STATEMENT", options),
 10000L);
@lidavidmlidavidmDec 21, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit, but can we move these numbers to constants so it's clear what they represent? (UPDATE_STATEMENT_EXPECTED_ROWS or something)
@lidavidmlidavidmDec 21, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The same goes for the producer, C++, etc.)
@rafael-tellesrafael-tellesDec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Done
@rafael-telles
rafael-telles
commented
Dec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Hey @lidavidm ! Are we good to merge this on flight-sql branch and then on master?
Thank you
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
We should be. I just kicked off CI.
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
I think the failures are spurious. (I filed ARROW-15181 for the crash in the Flight tests.)
lidavidm
lidavidm
approved these changes
Dec 21, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Sorry, I just noticed a couple more small things - we can merge into the branch next.
Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

@rafael-telles
rafael-telles
commented
Dec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Glad to see everything passing! Except to that "Dev/ Source Release and Merge Script", but it indeed seem to fail because we are in our own branch
lidavidm pushed a commit
that referenced
this pull request
Dec 21, 2021
@rafael-telles@lidavidm
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integratio…
16a9df1
…n tests
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.
The integrations tests assert that:
- RPC objects built on clients are parsed correctly on servers
- Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
Closes#11989 from rafael-telles/flight-sql-integration-tests-2
Authored-by: Rafael Telles <rafael@telles.dev>
Signed-off-by: David Li <li.davidm96@gmail.com>
@lidavidmlidavidm
mentioned this pull request
Dec 21, 2021
 Closed

@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
Mega-PR is up at #12013
@asfimportasfimport
mentioned this pull request
Dec 23, 2021

Closed

pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@rafael-tellesAbner Eduardo Ferreira
@jduo@jcralmeida@JrJuscelino@kylep-dremioRyan Nicholson@vfraga@tifflhl@kou
ARROW-14421: [C++] Implement Flight SQL
95460a1
Implement Flight SQL in C++ and Java, and add the protocol definitions.
This is a combination of multiple pull requests, merged into one branch before merging into master.
Closesapache#10906 (ARROW-12922).
Closesapache#11507 (ARROW-14421).
Closesapache#11989 (ARROW-15112).
Closesapache#12021 (ARROW-15187).
Closesapache#12035 (ARROW-15198).
Closesapache#12013 from apache/flight-sql
Lead-authored-by: Rafael Telles <rafael@telles.dev>
Co-authored-by: Abner Eduardo Ferreira <abenaru@protonmail.ch>
Co-authored-by: James Duong <duong.james@gmail.com>
Co-authored-by: Jose Almeida <almeidajcr90@gmail.com>
Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br>
Co-authored-by: Kyle Porter <kporter@dremio.com>
Co-authored-by: Ryan Nicholson <rnicholson@dremio.com>
Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com>
Co-authored-by: tifflhl <tiffanylamhl@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes

Assignees


No one assigned


Labels

Component: C++Component: FlightRPCComponent: GLibComponent: JavaComponent: ParquetComponent: PythonComponent: R

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@rafael-telles@lidavidm

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests - #11989
Closed
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom 
rafael-telles:flight-sql-integration-tests-2
Closed
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests#11989
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom 
rafael-telles:flight-sql-integration-tests-2
Conversation
@rafael-telles
@rafael-tellesrafael-telles
commented
Dec 17, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.

The integrations tests assert that:
  • RPC objects built on clients are parsed correctly on servers
  • Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
@github-actions
github-actionsBot
commented
Dec 17, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-15112
@rafael-telles
rafael-telles
commented
Dec 17, 2021

Copy link


Copy Markdown

ContributorAuthor
Hi @lidavidm ! We wrote the integration tests for Flight SQL as requested :)
lidavidm
lidavidm
reviewed
Dec 17, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround, this looks good overall. I left some comments.
Comment threadcpp/src/arrow/flight/integration_tests/CMakeLists.txt
Outdated

@lidavidmlidavidmDec 17, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the target necessary?
@rafael-tellesrafael-tellesDec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was already there on arrow/flight/CMakeLists.txt before moving to the new directory
Comment threadcpp/src/arrow/CMakeLists.txt
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.h
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
We need to exclude Go and Rust from the new test scenario.
Also, the PR needs to be formatted (see https://arrow.apache.org/docs/developers/cpp/development.html#code-style-linting-and-ci, note that we use ClangTools 12 now not version 8)
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Done @lidavidm
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Hey @lidavidm , I don't know if these CI failures has something to do with these changes... Please let me know if there is anything else to do here
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
There's some lint failures:

[1/1] cd /tmp/arrow-lint-6fb2hzaq/cpp-build && /usr/local/bin/python /arrow/cpp/build-support/lint_cpp_cli.py /arrow/cpp/src
INFO:archery:Running Python formatter (autopep8)
INFO:archery:Running Python linter (flake8)
/arrow/dev/archery/archery/integration/tester_java.py:52:80: E501 line too long (95 > 79 characters)
/arrow/dev/archery/archery/integration/runner.py:387:80: E501 line too long (82 > 79 characters)
INFO:archery:Running cmake-format linters
ERROR __main__.py:618: Check failed: /arrow/cpp/src/arrow/CMakeLists.txt

I think the other one is unrelated, but could you rebase on master to see if it goes away?
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
2 times, most recently
from
94f98fe to
5866b91CompareDecember 20, 2021 22:33
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
from
5866b91 to
78352e9CompareDecember 20, 2021 22:37
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Ops @lidavidm ... Sorry, I don't know if I messed up when git rebase master or if also need to run it in the base branch flight-sql...
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Ah, right, we have a branch…How about I rebase the branch, and then we can rebase the PRs?
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Hmm. I just rebased the branch…so do you mind trying to rebase again here? Sorry for the trouble.
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
from
78352e9 to
97fe3ceCompareDecember 20, 2021 23:12
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
No problem! Just rebased :)
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Well, looks like the Docker registry isn't doing so well right now. I'll kick all the CI pipelines in a bit to try again.
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
Hmm, I wonder if the test failure in Dev is just because it's getting confused since we're on a branch. I think we can resolve that when we make the final PR from flight-sql into master, then.
lidavidm
lidavidm
approved these changes
Dec 21, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just one more nit that caught my eye.
Comment thread...gration-tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenario.java
Outdated

 sqlClient.execute("SELECT STATEMENT", options), sqlClient);

 IntegrationAssertions.assertEquals(sqlClient.executeUpdate("UPDATE STATEMENT", options),
 10000L);
@lidavidmlidavidmDec 21, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit, but can we move these numbers to constants so it's clear what they represent? (UPDATE_STATEMENT_EXPECTED_ROWS or something)
@lidavidmlidavidmDec 21, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The same goes for the producer, C++, etc.)
@rafael-tellesrafael-tellesDec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Done
@rafael-telles
rafael-telles
commented
Dec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Hey @lidavidm ! Are we good to merge this on flight-sql branch and then on master?
Thank you
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
We should be. I just kicked off CI.
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
I think the failures are spurious. (I filed ARROW-15181 for the crash in the Flight tests.)
lidavidm
lidavidm
approved these changes
Dec 21, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Sorry, I just noticed a couple more small things - we can merge into the branch next.
Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

@rafael-telles
rafael-telles
commented
Dec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Glad to see everything passing! Except to that "Dev/ Source Release and Merge Script", but it indeed seem to fail because we are in our own branch
lidavidm pushed a commit
that referenced
this pull request
Dec 21, 2021
@rafael-telles@lidavidm
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integratio…
16a9df1
…n tests
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.
The integrations tests assert that:
- RPC objects built on clients are parsed correctly on servers
- Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
Closes#11989 from rafael-telles/flight-sql-integration-tests-2
Authored-by: Rafael Telles <rafael@telles.dev>
Signed-off-by: David Li <li.davidm96@gmail.com>
@lidavidmlidavidm
mentioned this pull request
Dec 21, 2021
 Closed

@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
Mega-PR is up at #12013
@asfimportasfimport
mentioned this pull request
Dec 23, 2021

Closed

pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@rafael-tellesAbner Eduardo Ferreira
@jduo@jcralmeida@JrJuscelino@kylep-dremioRyan Nicholson@vfraga@tifflhl@kou
ARROW-14421: [C++] Implement Flight SQL
95460a1
Implement Flight SQL in C++ and Java, and add the protocol definitions.
This is a combination of multiple pull requests, merged into one branch before merging into master.
Closesapache#10906 (ARROW-12922).
Closesapache#11507 (ARROW-14421).
Closesapache#11989 (ARROW-15112).
Closesapache#12021 (ARROW-15187).
Closesapache#12035 (ARROW-15198).
Closesapache#12013 from apache/flight-sql
Lead-authored-by: Rafael Telles <rafael@telles.dev>
Co-authored-by: Abner Eduardo Ferreira <abenaru@protonmail.ch>
Co-authored-by: James Duong <duong.james@gmail.com>
Co-authored-by: Jose Almeida <almeidajcr90@gmail.com>
Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br>
Co-authored-by: Kyle Porter <kporter@dremio.com>
Co-authored-by: Ryan Nicholson <rnicholson@dremio.com>
Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com>
Co-authored-by: tifflhl <tiffanylamhl@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes

Assignees


No one assigned


Labels

Component: C++Component: FlightRPCComponent: GLibComponent: JavaComponent: ParquetComponent: PythonComponent: R

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@rafael-telles@lidavidm

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests - #11989
Closed
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom 
rafael-telles:flight-sql-integration-tests-2
Closed
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests#11989
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom 
rafael-telles:flight-sql-integration-tests-2
Conversation
@rafael-telles
@rafael-tellesrafael-telles
commented
Dec 17, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.

The integrations tests assert that:
  • RPC objects built on clients are parsed correctly on servers
  • Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
@github-actions
github-actionsBot
commented
Dec 17, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-15112
@rafael-telles
rafael-telles
commented
Dec 17, 2021

Copy link


Copy Markdown

ContributorAuthor
Hi @lidavidm ! We wrote the integration tests for Flight SQL as requested :)
lidavidm
lidavidm
reviewed
Dec 17, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround, this looks good overall. I left some comments.
Comment threadcpp/src/arrow/flight/integration_tests/CMakeLists.txt
Outdated

@lidavidmlidavidmDec 17, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the target necessary?
@rafael-tellesrafael-tellesDec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was already there on arrow/flight/CMakeLists.txt before moving to the new directory
Comment threadcpp/src/arrow/CMakeLists.txt
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.h
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
We need to exclude Go and Rust from the new test scenario.
Also, the PR needs to be formatted (see https://arrow.apache.org/docs/developers/cpp/development.html#code-style-linting-and-ci, note that we use ClangTools 12 now not version 8)
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Done @lidavidm
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Hey @lidavidm , I don't know if these CI failures has something to do with these changes... Please let me know if there is anything else to do here
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
There's some lint failures:

[1/1] cd /tmp/arrow-lint-6fb2hzaq/cpp-build && /usr/local/bin/python /arrow/cpp/build-support/lint_cpp_cli.py /arrow/cpp/src
INFO:archery:Running Python formatter (autopep8)
INFO:archery:Running Python linter (flake8)
/arrow/dev/archery/archery/integration/tester_java.py:52:80: E501 line too long (95 > 79 characters)
/arrow/dev/archery/archery/integration/runner.py:387:80: E501 line too long (82 > 79 characters)
INFO:archery:Running cmake-format linters
ERROR __main__.py:618: Check failed: /arrow/cpp/src/arrow/CMakeLists.txt

I think the other one is unrelated, but could you rebase on master to see if it goes away?
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
2 times, most recently
from
94f98fe to
5866b91CompareDecember 20, 2021 22:33
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
from
5866b91 to
78352e9CompareDecember 20, 2021 22:37
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Ops @lidavidm ... Sorry, I don't know if I messed up when git rebase master or if also need to run it in the base branch flight-sql...
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Ah, right, we have a branch…How about I rebase the branch, and then we can rebase the PRs?
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Hmm. I just rebased the branch…so do you mind trying to rebase again here? Sorry for the trouble.
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
from
78352e9 to
97fe3ceCompareDecember 20, 2021 23:12
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
No problem! Just rebased :)
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Well, looks like the Docker registry isn't doing so well right now. I'll kick all the CI pipelines in a bit to try again.
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
Hmm, I wonder if the test failure in Dev is just because it's getting confused since we're on a branch. I think we can resolve that when we make the final PR from flight-sql into master, then.
lidavidm
lidavidm
approved these changes
Dec 21, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just one more nit that caught my eye.
Comment thread...gration-tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenario.java
Outdated

 sqlClient.execute("SELECT STATEMENT", options), sqlClient);

 IntegrationAssertions.assertEquals(sqlClient.executeUpdate("UPDATE STATEMENT", options),
 10000L);
@lidavidmlidavidmDec 21, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit, but can we move these numbers to constants so it's clear what they represent? (UPDATE_STATEMENT_EXPECTED_ROWS or something)
@lidavidmlidavidmDec 21, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The same goes for the producer, C++, etc.)
@rafael-tellesrafael-tellesDec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Done
@rafael-telles
rafael-telles
commented
Dec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Hey @lidavidm ! Are we good to merge this on flight-sql branch and then on master?
Thank you
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
We should be. I just kicked off CI.
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
I think the failures are spurious. (I filed ARROW-15181 for the crash in the Flight tests.)
lidavidm
lidavidm
approved these changes
Dec 21, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Sorry, I just noticed a couple more small things - we can merge into the branch next.
Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

@rafael-telles
rafael-telles
commented
Dec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Glad to see everything passing! Except to that "Dev/ Source Release and Merge Script", but it indeed seem to fail because we are in our own branch
lidavidm pushed a commit
that referenced
this pull request
Dec 21, 2021
@rafael-telles@lidavidm
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integratio…
16a9df1
…n tests
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.
The integrations tests assert that:
- RPC objects built on clients are parsed correctly on servers
- Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
Closes#11989 from rafael-telles/flight-sql-integration-tests-2
Authored-by: Rafael Telles <rafael@telles.dev>
Signed-off-by: David Li <li.davidm96@gmail.com>
@lidavidmlidavidm
mentioned this pull request
Dec 21, 2021
 Closed

@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
Mega-PR is up at #12013
@asfimportasfimport
mentioned this pull request
Dec 23, 2021

Closed

pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@rafael-tellesAbner Eduardo Ferreira
@jduo@jcralmeida@JrJuscelino@kylep-dremioRyan Nicholson@vfraga@tifflhl@kou
ARROW-14421: [C++] Implement Flight SQL
95460a1
Implement Flight SQL in C++ and Java, and add the protocol definitions.
This is a combination of multiple pull requests, merged into one branch before merging into master.
Closesapache#10906 (ARROW-12922).
Closesapache#11507 (ARROW-14421).
Closesapache#11989 (ARROW-15112).
Closesapache#12021 (ARROW-15187).
Closesapache#12035 (ARROW-15198).
Closesapache#12013 from apache/flight-sql
Lead-authored-by: Rafael Telles <rafael@telles.dev>
Co-authored-by: Abner Eduardo Ferreira <abenaru@protonmail.ch>
Co-authored-by: James Duong <duong.james@gmail.com>
Co-authored-by: Jose Almeida <almeidajcr90@gmail.com>
Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br>
Co-authored-by: Kyle Porter <kporter@dremio.com>
Co-authored-by: Ryan Nicholson <rnicholson@dremio.com>
Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com>
Co-authored-by: tifflhl <tiffanylamhl@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes

Assignees


No one assigned


Labels

Component: C++Component: FlightRPCComponent: GLibComponent: JavaComponent: ParquetComponent: PythonComponent: R

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@rafael-telles@lidavidm

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests - #11989
Closed
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom 
rafael-telles:flight-sql-integration-tests-2
Closed
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests#11989
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom 
rafael-telles:flight-sql-integration-tests-2
Conversation
@rafael-telles
@rafael-tellesrafael-telles
commented
Dec 17, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.

The integrations tests assert that:
  • RPC objects built on clients are parsed correctly on servers
  • Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
@github-actions
github-actionsBot
commented
Dec 17, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-15112
@rafael-telles
rafael-telles
commented
Dec 17, 2021

Copy link


Copy Markdown

ContributorAuthor
Hi @lidavidm ! We wrote the integration tests for Flight SQL as requested :)
lidavidm
lidavidm
reviewed
Dec 17, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround, this looks good overall. I left some comments.
Comment threadcpp/src/arrow/flight/integration_tests/CMakeLists.txt
Outdated

@lidavidmlidavidmDec 17, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the target necessary?
@rafael-tellesrafael-tellesDec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was already there on arrow/flight/CMakeLists.txt before moving to the new directory
Comment threadcpp/src/arrow/CMakeLists.txt
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.h
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
We need to exclude Go and Rust from the new test scenario.
Also, the PR needs to be formatted (see https://arrow.apache.org/docs/developers/cpp/development.html#code-style-linting-and-ci, note that we use ClangTools 12 now not version 8)
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Done @lidavidm
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Hey @lidavidm , I don't know if these CI failures has something to do with these changes... Please let me know if there is anything else to do here
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
There's some lint failures:

[1/1] cd /tmp/arrow-lint-6fb2hzaq/cpp-build && /usr/local/bin/python /arrow/cpp/build-support/lint_cpp_cli.py /arrow/cpp/src
INFO:archery:Running Python formatter (autopep8)
INFO:archery:Running Python linter (flake8)
/arrow/dev/archery/archery/integration/tester_java.py:52:80: E501 line too long (95 > 79 characters)
/arrow/dev/archery/archery/integration/runner.py:387:80: E501 line too long (82 > 79 characters)
INFO:archery:Running cmake-format linters
ERROR __main__.py:618: Check failed: /arrow/cpp/src/arrow/CMakeLists.txt

I think the other one is unrelated, but could you rebase on master to see if it goes away?
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
2 times, most recently
from
94f98fe to
5866b91CompareDecember 20, 2021 22:33
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
from
5866b91 to
78352e9CompareDecember 20, 2021 22:37
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Ops @lidavidm ... Sorry, I don't know if I messed up when git rebase master or if also need to run it in the base branch flight-sql...
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Ah, right, we have a branch…How about I rebase the branch, and then we can rebase the PRs?
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Hmm. I just rebased the branch…so do you mind trying to rebase again here? Sorry for the trouble.
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
from
78352e9 to
97fe3ceCompareDecember 20, 2021 23:12
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
No problem! Just rebased :)
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Well, looks like the Docker registry isn't doing so well right now. I'll kick all the CI pipelines in a bit to try again.
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
Hmm, I wonder if the test failure in Dev is just because it's getting confused since we're on a branch. I think we can resolve that when we make the final PR from flight-sql into master, then.
lidavidm
lidavidm
approved these changes
Dec 21, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just one more nit that caught my eye.
Comment thread...gration-tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenario.java
Outdated

 sqlClient.execute("SELECT STATEMENT", options), sqlClient);

 IntegrationAssertions.assertEquals(sqlClient.executeUpdate("UPDATE STATEMENT", options),
 10000L);
@lidavidmlidavidmDec 21, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit, but can we move these numbers to constants so it's clear what they represent? (UPDATE_STATEMENT_EXPECTED_ROWS or something)
@lidavidmlidavidmDec 21, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The same goes for the producer, C++, etc.)
@rafael-tellesrafael-tellesDec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Done
@rafael-telles
rafael-telles
commented
Dec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Hey @lidavidm ! Are we good to merge this on flight-sql branch and then on master?
Thank you
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
We should be. I just kicked off CI.
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
I think the failures are spurious. (I filed ARROW-15181 for the crash in the Flight tests.)
lidavidm
lidavidm
approved these changes
Dec 21, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Sorry, I just noticed a couple more small things - we can merge into the branch next.
Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

@rafael-telles
rafael-telles
commented
Dec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Glad to see everything passing! Except to that "Dev/ Source Release and Merge Script", but it indeed seem to fail because we are in our own branch
lidavidm pushed a commit
that referenced
this pull request
Dec 21, 2021
@rafael-telles@lidavidm
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integratio…
16a9df1
…n tests
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.
The integrations tests assert that:
- RPC objects built on clients are parsed correctly on servers
- Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
Closes#11989 from rafael-telles/flight-sql-integration-tests-2
Authored-by: Rafael Telles <rafael@telles.dev>
Signed-off-by: David Li <li.davidm96@gmail.com>
@lidavidmlidavidm
mentioned this pull request
Dec 21, 2021
 Closed

@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
Mega-PR is up at #12013
@asfimportasfimport
mentioned this pull request
Dec 23, 2021

Closed

pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@rafael-tellesAbner Eduardo Ferreira
@jduo@jcralmeida@JrJuscelino@kylep-dremioRyan Nicholson@vfraga@tifflhl@kou
ARROW-14421: [C++] Implement Flight SQL
95460a1
Implement Flight SQL in C++ and Java, and add the protocol definitions.
This is a combination of multiple pull requests, merged into one branch before merging into master.
Closesapache#10906 (ARROW-12922).
Closesapache#11507 (ARROW-14421).
Closesapache#11989 (ARROW-15112).
Closesapache#12021 (ARROW-15187).
Closesapache#12035 (ARROW-15198).
Closesapache#12013 from apache/flight-sql
Lead-authored-by: Rafael Telles <rafael@telles.dev>
Co-authored-by: Abner Eduardo Ferreira <abenaru@protonmail.ch>
Co-authored-by: James Duong <duong.james@gmail.com>
Co-authored-by: Jose Almeida <almeidajcr90@gmail.com>
Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br>
Co-authored-by: Kyle Porter <kporter@dremio.com>
Co-authored-by: Ryan Nicholson <rnicholson@dremio.com>
Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com>
Co-authored-by: tifflhl <tiffanylamhl@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes

Assignees


No one assigned


Labels

Component: C++Component: FlightRPCComponent: GLibComponent: JavaComponent: ParquetComponent: PythonComponent: R

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@rafael-telles@lidavidm

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests - #11989
Closed
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom 
rafael-telles:flight-sql-integration-tests-2
Closed
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests#11989
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom 
rafael-telles:flight-sql-integration-tests-2
Conversation
@rafael-telles
@rafael-tellesrafael-telles
commented
Dec 17, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.

The integrations tests assert that:
  • RPC objects built on clients are parsed correctly on servers
  • Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
@github-actions
github-actionsBot
commented
Dec 17, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-15112
@rafael-telles
rafael-telles
commented
Dec 17, 2021

Copy link


Copy Markdown

ContributorAuthor
Hi @lidavidm ! We wrote the integration tests for Flight SQL as requested :)
lidavidm
lidavidm
reviewed
Dec 17, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround, this looks good overall. I left some comments.
Comment threadcpp/src/arrow/flight/integration_tests/CMakeLists.txt
Outdated

@lidavidmlidavidmDec 17, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the target necessary?
@rafael-tellesrafael-tellesDec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was already there on arrow/flight/CMakeLists.txt before moving to the new directory
Comment threadcpp/src/arrow/CMakeLists.txt
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.h
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
We need to exclude Go and Rust from the new test scenario.
Also, the PR needs to be formatted (see https://arrow.apache.org/docs/developers/cpp/development.html#code-style-linting-and-ci, note that we use ClangTools 12 now not version 8)
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Done @lidavidm
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Hey @lidavidm , I don't know if these CI failures has something to do with these changes... Please let me know if there is anything else to do here
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
There's some lint failures:

[1/1] cd /tmp/arrow-lint-6fb2hzaq/cpp-build && /usr/local/bin/python /arrow/cpp/build-support/lint_cpp_cli.py /arrow/cpp/src
INFO:archery:Running Python formatter (autopep8)
INFO:archery:Running Python linter (flake8)
/arrow/dev/archery/archery/integration/tester_java.py:52:80: E501 line too long (95 > 79 characters)
/arrow/dev/archery/archery/integration/runner.py:387:80: E501 line too long (82 > 79 characters)
INFO:archery:Running cmake-format linters
ERROR __main__.py:618: Check failed: /arrow/cpp/src/arrow/CMakeLists.txt

I think the other one is unrelated, but could you rebase on master to see if it goes away?
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
2 times, most recently
from
94f98fe to
5866b91CompareDecember 20, 2021 22:33
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
from
5866b91 to
78352e9CompareDecember 20, 2021 22:37
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Ops @lidavidm ... Sorry, I don't know if I messed up when git rebase master or if also need to run it in the base branch flight-sql...
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Ah, right, we have a branch…How about I rebase the branch, and then we can rebase the PRs?
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Hmm. I just rebased the branch…so do you mind trying to rebase again here? Sorry for the trouble.
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
from
78352e9 to
97fe3ceCompareDecember 20, 2021 23:12
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
No problem! Just rebased :)
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Well, looks like the Docker registry isn't doing so well right now. I'll kick all the CI pipelines in a bit to try again.
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
Hmm, I wonder if the test failure in Dev is just because it's getting confused since we're on a branch. I think we can resolve that when we make the final PR from flight-sql into master, then.
lidavidm
lidavidm
approved these changes
Dec 21, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just one more nit that caught my eye.
Comment thread...gration-tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenario.java
Outdated

 sqlClient.execute("SELECT STATEMENT", options), sqlClient);

 IntegrationAssertions.assertEquals(sqlClient.executeUpdate("UPDATE STATEMENT", options),
 10000L);
@lidavidmlidavidmDec 21, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit, but can we move these numbers to constants so it's clear what they represent? (UPDATE_STATEMENT_EXPECTED_ROWS or something)
@lidavidmlidavidmDec 21, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The same goes for the producer, C++, etc.)
@rafael-tellesrafael-tellesDec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Done
@rafael-telles
rafael-telles
commented
Dec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Hey @lidavidm ! Are we good to merge this on flight-sql branch and then on master?
Thank you
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
We should be. I just kicked off CI.
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
I think the failures are spurious. (I filed ARROW-15181 for the crash in the Flight tests.)
lidavidm
lidavidm
approved these changes
Dec 21, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Sorry, I just noticed a couple more small things - we can merge into the branch next.
Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

@rafael-telles
rafael-telles
commented
Dec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Glad to see everything passing! Except to that "Dev/ Source Release and Merge Script", but it indeed seem to fail because we are in our own branch
lidavidm pushed a commit
that referenced
this pull request
Dec 21, 2021
@rafael-telles@lidavidm
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integratio…
16a9df1
…n tests
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.
The integrations tests assert that:
- RPC objects built on clients are parsed correctly on servers
- Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
Closes#11989 from rafael-telles/flight-sql-integration-tests-2
Authored-by: Rafael Telles <rafael@telles.dev>
Signed-off-by: David Li <li.davidm96@gmail.com>
@lidavidmlidavidm
mentioned this pull request
Dec 21, 2021
 Closed

@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
Mega-PR is up at #12013
@asfimportasfimport
mentioned this pull request
Dec 23, 2021

Closed

pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@rafael-tellesAbner Eduardo Ferreira
@jduo@jcralmeida@JrJuscelino@kylep-dremioRyan Nicholson@vfraga@tifflhl@kou
ARROW-14421: [C++] Implement Flight SQL
95460a1
Implement Flight SQL in C++ and Java, and add the protocol definitions.
This is a combination of multiple pull requests, merged into one branch before merging into master.
Closesapache#10906 (ARROW-12922).
Closesapache#11507 (ARROW-14421).
Closesapache#11989 (ARROW-15112).
Closesapache#12021 (ARROW-15187).
Closesapache#12035 (ARROW-15198).
Closesapache#12013 from apache/flight-sql
Lead-authored-by: Rafael Telles <rafael@telles.dev>
Co-authored-by: Abner Eduardo Ferreira <abenaru@protonmail.ch>
Co-authored-by: James Duong <duong.james@gmail.com>
Co-authored-by: Jose Almeida <almeidajcr90@gmail.com>
Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br>
Co-authored-by: Kyle Porter <kporter@dremio.com>
Co-authored-by: Ryan Nicholson <rnicholson@dremio.com>
Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com>
Co-authored-by: tifflhl <tiffanylamhl@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes

Assignees


No one assigned


Labels

Component: C++Component: FlightRPCComponent: GLibComponent: JavaComponent: ParquetComponent: PythonComponent: R

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@rafael-telles@lidavidm

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests - #11989
Closed
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom 
rafael-telles:flight-sql-integration-tests-2
Closed
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests#11989
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom 
rafael-telles:flight-sql-integration-tests-2
Conversation
@rafael-telles
@rafael-tellesrafael-telles
commented
Dec 17, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.

The integrations tests assert that:
  • RPC objects built on clients are parsed correctly on servers
  • Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
@github-actions
github-actionsBot
commented
Dec 17, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-15112
@rafael-telles
rafael-telles
commented
Dec 17, 2021

Copy link


Copy Markdown

ContributorAuthor
Hi @lidavidm ! We wrote the integration tests for Flight SQL as requested :)
lidavidm
lidavidm
reviewed
Dec 17, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround, this looks good overall. I left some comments.
Comment threadcpp/src/arrow/flight/integration_tests/CMakeLists.txt
Outdated

@lidavidmlidavidmDec 17, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the target necessary?
@rafael-tellesrafael-tellesDec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was already there on arrow/flight/CMakeLists.txt before moving to the new directory
Comment threadcpp/src/arrow/CMakeLists.txt
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.h
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
We need to exclude Go and Rust from the new test scenario.
Also, the PR needs to be formatted (see https://arrow.apache.org/docs/developers/cpp/development.html#code-style-linting-and-ci, note that we use ClangTools 12 now not version 8)
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Done @lidavidm
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Hey @lidavidm , I don't know if these CI failures has something to do with these changes... Please let me know if there is anything else to do here
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
There's some lint failures:

[1/1] cd /tmp/arrow-lint-6fb2hzaq/cpp-build && /usr/local/bin/python /arrow/cpp/build-support/lint_cpp_cli.py /arrow/cpp/src
INFO:archery:Running Python formatter (autopep8)
INFO:archery:Running Python linter (flake8)
/arrow/dev/archery/archery/integration/tester_java.py:52:80: E501 line too long (95 > 79 characters)
/arrow/dev/archery/archery/integration/runner.py:387:80: E501 line too long (82 > 79 characters)
INFO:archery:Running cmake-format linters
ERROR __main__.py:618: Check failed: /arrow/cpp/src/arrow/CMakeLists.txt

I think the other one is unrelated, but could you rebase on master to see if it goes away?
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
2 times, most recently
from
94f98fe to
5866b91CompareDecember 20, 2021 22:33
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
from
5866b91 to
78352e9CompareDecember 20, 2021 22:37
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Ops @lidavidm ... Sorry, I don't know if I messed up when git rebase master or if also need to run it in the base branch flight-sql...
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Ah, right, we have a branch…How about I rebase the branch, and then we can rebase the PRs?
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Hmm. I just rebased the branch…so do you mind trying to rebase again here? Sorry for the trouble.
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
from
78352e9 to
97fe3ceCompareDecember 20, 2021 23:12
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
No problem! Just rebased :)
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Well, looks like the Docker registry isn't doing so well right now. I'll kick all the CI pipelines in a bit to try again.
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
Hmm, I wonder if the test failure in Dev is just because it's getting confused since we're on a branch. I think we can resolve that when we make the final PR from flight-sql into master, then.
lidavidm
lidavidm
approved these changes
Dec 21, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just one more nit that caught my eye.
Comment thread...gration-tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenario.java
Outdated

 sqlClient.execute("SELECT STATEMENT", options), sqlClient);

 IntegrationAssertions.assertEquals(sqlClient.executeUpdate("UPDATE STATEMENT", options),
 10000L);
@lidavidmlidavidmDec 21, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit, but can we move these numbers to constants so it's clear what they represent? (UPDATE_STATEMENT_EXPECTED_ROWS or something)
@lidavidmlidavidmDec 21, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The same goes for the producer, C++, etc.)
@rafael-tellesrafael-tellesDec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Done
@rafael-telles
rafael-telles
commented
Dec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Hey @lidavidm ! Are we good to merge this on flight-sql branch and then on master?
Thank you
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
We should be. I just kicked off CI.
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
I think the failures are spurious. (I filed ARROW-15181 for the crash in the Flight tests.)
lidavidm
lidavidm
approved these changes
Dec 21, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Sorry, I just noticed a couple more small things - we can merge into the branch next.
Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

@rafael-telles
rafael-telles
commented
Dec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Glad to see everything passing! Except to that "Dev/ Source Release and Merge Script", but it indeed seem to fail because we are in our own branch
lidavidm pushed a commit
that referenced
this pull request
Dec 21, 2021
@rafael-telles@lidavidm
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integratio…
16a9df1
…n tests
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.
The integrations tests assert that:
- RPC objects built on clients are parsed correctly on servers
- Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
Closes#11989 from rafael-telles/flight-sql-integration-tests-2
Authored-by: Rafael Telles <rafael@telles.dev>
Signed-off-by: David Li <li.davidm96@gmail.com>
@lidavidmlidavidm
mentioned this pull request
Dec 21, 2021
 Closed

@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
Mega-PR is up at #12013
@asfimportasfimport
mentioned this pull request
Dec 23, 2021

Closed

pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@rafael-tellesAbner Eduardo Ferreira
@jduo@jcralmeida@JrJuscelino@kylep-dremioRyan Nicholson@vfraga@tifflhl@kou
ARROW-14421: [C++] Implement Flight SQL
95460a1
Implement Flight SQL in C++ and Java, and add the protocol definitions.
This is a combination of multiple pull requests, merged into one branch before merging into master.
Closesapache#10906 (ARROW-12922).
Closesapache#11507 (ARROW-14421).
Closesapache#11989 (ARROW-15112).
Closesapache#12021 (ARROW-15187).
Closesapache#12035 (ARROW-15198).
Closesapache#12013 from apache/flight-sql
Lead-authored-by: Rafael Telles <rafael@telles.dev>
Co-authored-by: Abner Eduardo Ferreira <abenaru@protonmail.ch>
Co-authored-by: James Duong <duong.james@gmail.com>
Co-authored-by: Jose Almeida <almeidajcr90@gmail.com>
Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br>
Co-authored-by: Kyle Porter <kporter@dremio.com>
Co-authored-by: Ryan Nicholson <rnicholson@dremio.com>
Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com>
Co-authored-by: tifflhl <tiffanylamhl@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes

Assignees


No one assigned


Labels

Component: C++Component: FlightRPCComponent: GLibComponent: JavaComponent: ParquetComponent: PythonComponent: R

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@rafael-telles@lidavidm

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests - #11989
Closed
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom 
rafael-telles:flight-sql-integration-tests-2
Closed
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integration tests#11989
rafael-telles wants to merge 8 commits into
apache:flight-sqlfrom 
rafael-telles:flight-sql-integration-tests-2
Conversation
@rafael-telles
@rafael-tellesrafael-telles
commented
Dec 17, 2021•

edited

Loading

Copy link


Copy Markdown

Contributor
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.

The integrations tests assert that:
  • RPC objects built on clients are parsed correctly on servers
  • Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
@github-actions
github-actionsBot
commented
Dec 17, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-15112
@rafael-telles
rafael-telles
commented
Dec 17, 2021

Copy link


Copy Markdown

ContributorAuthor
Hi @lidavidm ! We wrote the integration tests for Flight SQL as requested :)
lidavidm
lidavidm
reviewed
Dec 17, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround, this looks good overall. I left some comments.
Comment threadcpp/src/arrow/flight/integration_tests/CMakeLists.txt
Outdated

@lidavidmlidavidmDec 17, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the target necessary?
@rafael-tellesrafael-tellesDec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was already there on arrow/flight/CMakeLists.txt before moving to the new directory
Comment threadcpp/src/arrow/CMakeLists.txt
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.h
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

Comment thread...tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenarioProducer.java
Outdated

@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
We need to exclude Go and Rust from the new test scenario.
Also, the PR needs to be formatted (see https://arrow.apache.org/docs/developers/cpp/development.html#code-style-linting-and-ci, note that we use ClangTools 12 now not version 8)
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Done @lidavidm
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Hey @lidavidm , I don't know if these CI failures has something to do with these changes... Please let me know if there is anything else to do here
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
There's some lint failures:

[1/1] cd /tmp/arrow-lint-6fb2hzaq/cpp-build && /usr/local/bin/python /arrow/cpp/build-support/lint_cpp_cli.py /arrow/cpp/src
INFO:archery:Running Python formatter (autopep8)
INFO:archery:Running Python linter (flake8)
/arrow/dev/archery/archery/integration/tester_java.py:52:80: E501 line too long (95 > 79 characters)
/arrow/dev/archery/archery/integration/runner.py:387:80: E501 line too long (82 > 79 characters)
INFO:archery:Running cmake-format linters
ERROR __main__.py:618: Check failed: /arrow/cpp/src/arrow/CMakeLists.txt

I think the other one is unrelated, but could you rebase on master to see if it goes away?
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
2 times, most recently
from
94f98fe to
5866b91CompareDecember 20, 2021 22:33
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
from
5866b91 to
78352e9CompareDecember 20, 2021 22:37
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
Ops @lidavidm ... Sorry, I don't know if I messed up when git rebase master or if also need to run it in the base branch flight-sql...
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Ah, right, we have a branch…How about I rebase the branch, and then we can rebase the PRs?
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Hmm. I just rebased the branch…so do you mind trying to rebase again here? Sorry for the trouble.
@rafael-telles
rafael-tellesforce-pushed
the
flight-sql-integration-tests-2
branch
from
78352e9 to
97fe3ceCompareDecember 20, 2021 23:12
@rafael-telles
rafael-telles
commented
Dec 20, 2021

Copy link


Copy Markdown

ContributorAuthor
No problem! Just rebased :)
@lidavidm
lidavidm
commented
Dec 20, 2021

Copy link


Copy Markdown

Member
Well, looks like the Docker registry isn't doing so well right now. I'll kick all the CI pipelines in a bit to try again.
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
Hmm, I wonder if the test failure in Dev is just because it's getting confused since we're on a branch. I think we can resolve that when we make the final PR from flight-sql into master, then.
lidavidm
lidavidm
approved these changes
Dec 21, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just one more nit that caught my eye.
Comment thread...gration-tests/src/main/java/org/apache/arrow/flight/integration/tests/FlightSqlScenario.java
Outdated

 sqlClient.execute("SELECT STATEMENT", options), sqlClient);

 IntegrationAssertions.assertEquals(sqlClient.executeUpdate("UPDATE STATEMENT", options),
 10000L);
@lidavidmlidavidmDec 21, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit, but can we move these numbers to constants so it's clear what they represent? (UPDATE_STATEMENT_EXPECTED_ROWS or something)
@lidavidmlidavidmDec 21, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The same goes for the producer, C++, etc.)
@rafael-tellesrafael-tellesDec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Done
@rafael-telles
rafael-telles
commented
Dec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Hey @lidavidm ! Are we good to merge this on flight-sql branch and then on master?
Thank you
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
We should be. I just kicked off CI.
@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
I think the failures are spurious. (I filed ARROW-15181 for the crash in the Flight tests.)
lidavidm
lidavidm
approved these changes
Dec 21, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Sorry, I just noticed a couple more small things - we can merge into the branch next.
Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

Comment threadcpp/src/arrow/flight/integration_tests/test_integration.cc
Outdated

@rafael-telles
rafael-telles
commented
Dec 21, 2021

Copy link


Copy Markdown

ContributorAuthor
Glad to see everything passing! Except to that "Dev/ Source Release and Merge Script", but it indeed seem to fail because we are in our own branch
lidavidm pushed a commit
that referenced
this pull request
Dec 21, 2021
@rafael-telles@lidavidm
ARROW-15112: [Integration][C++][Java] Implement Flight SQL integratio…
16a9df1
…n tests
This adds Flight SQL scenarios for integration tests for C++ and Java implementations.
The integrations tests assert that:
- RPC objects built on clients are parsed correctly on servers
- Arrow vectors built on servers have the expected Arrow schemas when received on clients
Note: Had to separate integration tests in a new module within Java and C++ to avoid circular dependencies (Flight <-> Flight SQL)
While working on this I found some inconsistencies between both implementation, this PR also fixes these problems.
Closes#11989 from rafael-telles/flight-sql-integration-tests-2
Authored-by: Rafael Telles <rafael@telles.dev>
Signed-off-by: David Li <li.davidm96@gmail.com>
@lidavidmlidavidm
mentioned this pull request
Dec 21, 2021
 Closed

@lidavidm
lidavidm
commented
Dec 21, 2021

Copy link


Copy Markdown

Member
Mega-PR is up at #12013
@asfimportasfimport
mentioned this pull request
Dec 23, 2021

Closed

pribor pushed a commit
to GlobalWebIndex/arrow
that referenced
this pull request
Oct 24, 2025
@rafael-tellesAbner Eduardo Ferreira
@jduo@jcralmeida@JrJuscelino@kylep-dremioRyan Nicholson@vfraga@tifflhl@kou
ARROW-14421: [C++] Implement Flight SQL
95460a1
Implement Flight SQL in C++ and Java, and add the protocol definitions.
This is a combination of multiple pull requests, merged into one branch before merging into master.
Closesapache#10906 (ARROW-12922).
Closesapache#11507 (ARROW-14421).
Closesapache#11989 (ARROW-15112).
Closesapache#12021 (ARROW-15187).
Closesapache#12035 (ARROW-15198).
Closesapache#12013 from apache/flight-sql
Lead-authored-by: Rafael Telles <rafael@telles.dev>
Co-authored-by: Abner Eduardo Ferreira <abenaru@protonmail.ch>
Co-authored-by: James Duong <duong.james@gmail.com>
Co-authored-by: Jose Almeida <almeidajcr90@gmail.com>
Co-authored-by: Juscelino Junior <juscelinojunior@id.uff.br>
Co-authored-by: Kyle Porter <kporter@dremio.com>
Co-authored-by: Ryan Nicholson <rnicholson@dremio.com>
Co-authored-by: Vinicius Fraga <sxvinifp@gmail.com>
Co-authored-by: tifflhl <tiffanylamhl@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes

Assignees


No one assigned


Labels

Component: C++Component: FlightRPCComponent: GLibComponent: JavaComponent: ParquetComponent: PythonComponent: R

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@rafael-telles@lidavidm