ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojects by lidavidm · Pull Request #12573 · apache/arrow

lidavidm · 2022-03-05T15:53:45Z

Adds a Flight SQL documentation page that we can link to from the website.

github-actions · 2022-03-05T15:54:00Z

https://issues.apache.org/jira/browse/ARROW-15721

lidavidm · 2022-03-05T15:54:15Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

lidavidm · 2022-03-05T15:56:35Z

CC @jduo if you'd like to take a look

jduo · 2022-03-06T13:36:43Z

+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints.


Let's mention that Flight SQL clients decorate existing Flight client classes to add methods corresponding to the new RPC calls.

Co-authored-by: James Duong <duong.james@gmail.com>

lidavidm · 2022-03-07T13:07:08Z

Thanks for the suggestions!

pitrou · 2022-03-07T16:55:36Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

Looks interesting. @jorisvandenbossche What do you think?

pitrou

Nice addition, thank you.

pitrou · 2022-03-07T17:06:19Z

+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.
+
+ When used with DoPut: binds parameter values to the prepared statement.


So one must call first DoPut then GetFlightInfo to execute a query with parameters? Wouldn't it be nicer to be able to use DoExchange for that?

I believe this was considered during the initial discussion but was rejected; @jduo do you remember why?

Ah, it was due to output parameters potentially changing the schema mid-stream. #9368 (comment)

pitrou · 2022-03-07T17:08:22Z

+ Create a new prepared statement for a SQL query.
+
+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.


Does this also close the prepared statement or can it be reused multiple times?

It is reusable (else, you don't save much from having a prepared statement).

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

lidavidm · 2022-03-09T14:30:19Z

@jduo @pitrou any other comments here? I think we can use ARROW-14698 to expand the documentation once this lands

pitrou · 2022-03-09T17:54:15Z

+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.


For the record, would there be an easy way to cross-reference the table definitions in the Flight protobuf file (such as FlightDescriptor here)? I guess not, but just asking.

Hmm, we would probably need to have Sphinx index the Protobuf file somehow and then use Sphinx's cross-reference support. Unfortunately there doesn't seem to be such integration from a quick glance (and if we wanted to do it, we would need to create a custom Protobuf plugin or modify the existing one (which supports neither Doxygen nor Sphinx): https://github.com/pseudomuto/protoc-gen-doc/

That may be worthwhile, though, since then we could avoid repeating docs between both files.

pitrou

+1

ursabot · 2022-03-09T18:52:06Z

Benchmark runs are scheduled for baseline = 875af62 and contender = 76b1403. 76b1403 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.08% ⬆️0.04%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.09% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

lidavidm · 2022-03-05T15:53:45Z

Adds a Flight SQL documentation page that we can link to from the website.

github-actions · 2022-03-05T15:54:00Z

https://issues.apache.org/jira/browse/ARROW-15721

lidavidm · 2022-03-05T15:54:15Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

lidavidm · 2022-03-05T15:56:35Z

CC @jduo if you'd like to take a look

jduo · 2022-03-06T13:36:43Z

+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints.


Let's mention that Flight SQL clients decorate existing Flight client classes to add methods corresponding to the new RPC calls.

Co-authored-by: James Duong <duong.james@gmail.com>

lidavidm · 2022-03-07T13:07:08Z

Thanks for the suggestions!

pitrou · 2022-03-07T16:55:36Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

Looks interesting. @jorisvandenbossche What do you think?

pitrou

Nice addition, thank you.

pitrou · 2022-03-07T17:06:19Z

+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.
+
+ When used with DoPut: binds parameter values to the prepared statement.


So one must call first DoPut then GetFlightInfo to execute a query with parameters? Wouldn't it be nicer to be able to use DoExchange for that?

I believe this was considered during the initial discussion but was rejected; @jduo do you remember why?

Ah, it was due to output parameters potentially changing the schema mid-stream. #9368 (comment)

pitrou · 2022-03-07T17:08:22Z

+ Create a new prepared statement for a SQL query.
+
+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.


Does this also close the prepared statement or can it be reused multiple times?

It is reusable (else, you don't save much from having a prepared statement).

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

lidavidm · 2022-03-09T14:30:19Z

@jduo @pitrou any other comments here? I think we can use ARROW-14698 to expand the documentation once this lands

pitrou · 2022-03-09T17:54:15Z

+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.


For the record, would there be an easy way to cross-reference the table definitions in the Flight protobuf file (such as FlightDescriptor here)? I guess not, but just asking.

Hmm, we would probably need to have Sphinx index the Protobuf file somehow and then use Sphinx's cross-reference support. Unfortunately there doesn't seem to be such integration from a quick glance (and if we wanted to do it, we would need to create a custom Protobuf plugin or modify the existing one (which supports neither Doxygen nor Sphinx): https://github.com/pseudomuto/protoc-gen-doc/

That may be worthwhile, though, since then we could avoid repeating docs between both files.

pitrou

+1

ursabot · 2022-03-09T18:52:06Z

Benchmark runs are scheduled for baseline = 875af62 and contender = 76b1403. 76b1403 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.08% ⬆️0.04%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.09% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

lidavidm · 2022-03-05T15:53:45Z

Adds a Flight SQL documentation page that we can link to from the website.

github-actions · 2022-03-05T15:54:00Z

https://issues.apache.org/jira/browse/ARROW-15721

lidavidm · 2022-03-05T15:54:15Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

lidavidm · 2022-03-05T15:56:35Z

CC @jduo if you'd like to take a look

jduo · 2022-03-06T13:36:43Z

+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints.


Let's mention that Flight SQL clients decorate existing Flight client classes to add methods corresponding to the new RPC calls.

Co-authored-by: James Duong <duong.james@gmail.com>

lidavidm · 2022-03-07T13:07:08Z

Thanks for the suggestions!

pitrou · 2022-03-07T16:55:36Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

Looks interesting. @jorisvandenbossche What do you think?

pitrou

Nice addition, thank you.

pitrou · 2022-03-07T17:06:19Z

+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.
+
+ When used with DoPut: binds parameter values to the prepared statement.


So one must call first DoPut then GetFlightInfo to execute a query with parameters? Wouldn't it be nicer to be able to use DoExchange for that?

I believe this was considered during the initial discussion but was rejected; @jduo do you remember why?

Ah, it was due to output parameters potentially changing the schema mid-stream. #9368 (comment)

pitrou · 2022-03-07T17:08:22Z

+ Create a new prepared statement for a SQL query.
+
+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.


Does this also close the prepared statement or can it be reused multiple times?

It is reusable (else, you don't save much from having a prepared statement).

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

lidavidm · 2022-03-09T14:30:19Z

@jduo @pitrou any other comments here? I think we can use ARROW-14698 to expand the documentation once this lands

pitrou · 2022-03-09T17:54:15Z

+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.


For the record, would there be an easy way to cross-reference the table definitions in the Flight protobuf file (such as FlightDescriptor here)? I guess not, but just asking.

Hmm, we would probably need to have Sphinx index the Protobuf file somehow and then use Sphinx's cross-reference support. Unfortunately there doesn't seem to be such integration from a quick glance (and if we wanted to do it, we would need to create a custom Protobuf plugin or modify the existing one (which supports neither Doxygen nor Sphinx): https://github.com/pseudomuto/protoc-gen-doc/

That may be worthwhile, though, since then we could avoid repeating docs between both files.

pitrou

+1

ursabot · 2022-03-09T18:52:06Z

Benchmark runs are scheduled for baseline = 875af62 and contender = 76b1403. 76b1403 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.08% ⬆️0.04%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.09% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

lidavidm · 2022-03-05T15:53:45Z

Adds a Flight SQL documentation page that we can link to from the website.

github-actions · 2022-03-05T15:54:00Z

https://issues.apache.org/jira/browse/ARROW-15721

lidavidm · 2022-03-05T15:54:15Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

lidavidm · 2022-03-05T15:56:35Z

CC @jduo if you'd like to take a look

jduo · 2022-03-06T13:36:43Z

+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints.


Let's mention that Flight SQL clients decorate existing Flight client classes to add methods corresponding to the new RPC calls.

Co-authored-by: James Duong <duong.james@gmail.com>

lidavidm · 2022-03-07T13:07:08Z

Thanks for the suggestions!

pitrou · 2022-03-07T16:55:36Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

Looks interesting. @jorisvandenbossche What do you think?

pitrou

Nice addition, thank you.

pitrou · 2022-03-07T17:06:19Z

+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.
+
+ When used with DoPut: binds parameter values to the prepared statement.


So one must call first DoPut then GetFlightInfo to execute a query with parameters? Wouldn't it be nicer to be able to use DoExchange for that?

I believe this was considered during the initial discussion but was rejected; @jduo do you remember why?

Ah, it was due to output parameters potentially changing the schema mid-stream. #9368 (comment)

pitrou · 2022-03-07T17:08:22Z

+ Create a new prepared statement for a SQL query.
+
+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.


Does this also close the prepared statement or can it be reused multiple times?

It is reusable (else, you don't save much from having a prepared statement).

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

lidavidm · 2022-03-09T14:30:19Z

@jduo @pitrou any other comments here? I think we can use ARROW-14698 to expand the documentation once this lands

pitrou · 2022-03-09T17:54:15Z

+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.


For the record, would there be an easy way to cross-reference the table definitions in the Flight protobuf file (such as FlightDescriptor here)? I guess not, but just asking.

Hmm, we would probably need to have Sphinx index the Protobuf file somehow and then use Sphinx's cross-reference support. Unfortunately there doesn't seem to be such integration from a quick glance (and if we wanted to do it, we would need to create a custom Protobuf plugin or modify the existing one (which supports neither Doxygen nor Sphinx): https://github.com/pseudomuto/protoc-gen-doc/

That may be worthwhile, though, since then we could avoid repeating docs between both files.

pitrou

+1

ursabot · 2022-03-09T18:52:06Z

Benchmark runs are scheduled for baseline = 875af62 and contender = 76b1403. 76b1403 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.08% ⬆️0.04%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.09% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

lidavidm · 2022-03-05T15:53:45Z

Adds a Flight SQL documentation page that we can link to from the website.

github-actions · 2022-03-05T15:54:00Z

https://issues.apache.org/jira/browse/ARROW-15721

lidavidm · 2022-03-05T15:54:15Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

lidavidm · 2022-03-05T15:56:35Z

CC @jduo if you'd like to take a look

jduo · 2022-03-06T13:36:43Z

+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints.


Let's mention that Flight SQL clients decorate existing Flight client classes to add methods corresponding to the new RPC calls.

Co-authored-by: James Duong <duong.james@gmail.com>

lidavidm · 2022-03-07T13:07:08Z

Thanks for the suggestions!

pitrou · 2022-03-07T16:55:36Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

Looks interesting. @jorisvandenbossche What do you think?

pitrou

Nice addition, thank you.

pitrou · 2022-03-07T17:06:19Z

+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.
+
+ When used with DoPut: binds parameter values to the prepared statement.


So one must call first DoPut then GetFlightInfo to execute a query with parameters? Wouldn't it be nicer to be able to use DoExchange for that?

I believe this was considered during the initial discussion but was rejected; @jduo do you remember why?

Ah, it was due to output parameters potentially changing the schema mid-stream. #9368 (comment)

pitrou · 2022-03-07T17:08:22Z

+ Create a new prepared statement for a SQL query.
+
+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.


Does this also close the prepared statement or can it be reused multiple times?

It is reusable (else, you don't save much from having a prepared statement).

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

lidavidm · 2022-03-09T14:30:19Z

@jduo @pitrou any other comments here? I think we can use ARROW-14698 to expand the documentation once this lands

pitrou · 2022-03-09T17:54:15Z

+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.


For the record, would there be an easy way to cross-reference the table definitions in the Flight protobuf file (such as FlightDescriptor here)? I guess not, but just asking.

Hmm, we would probably need to have Sphinx index the Protobuf file somehow and then use Sphinx's cross-reference support. Unfortunately there doesn't seem to be such integration from a quick glance (and if we wanted to do it, we would need to create a custom Protobuf plugin or modify the existing one (which supports neither Doxygen nor Sphinx): https://github.com/pseudomuto/protoc-gen-doc/

That may be worthwhile, though, since then we could avoid repeating docs between both files.

pitrou

+1

ursabot · 2022-03-09T18:52:06Z

Benchmark runs are scheduled for baseline = 875af62 and contender = 76b1403. 76b1403 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.08% ⬆️0.04%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.09% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

lidavidm · 2022-03-05T15:53:45Z

Adds a Flight SQL documentation page that we can link to from the website.

github-actions · 2022-03-05T15:54:00Z

https://issues.apache.org/jira/browse/ARROW-15721

lidavidm · 2022-03-05T15:54:15Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

lidavidm · 2022-03-05T15:56:35Z

CC @jduo if you'd like to take a look

jduo · 2022-03-06T13:36:43Z

+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints.


Let's mention that Flight SQL clients decorate existing Flight client classes to add methods corresponding to the new RPC calls.

Co-authored-by: James Duong <duong.james@gmail.com>

lidavidm · 2022-03-07T13:07:08Z

Thanks for the suggestions!

pitrou · 2022-03-07T16:55:36Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

Looks interesting. @jorisvandenbossche What do you think?

pitrou

Nice addition, thank you.

pitrou · 2022-03-07T17:06:19Z

+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.
+
+ When used with DoPut: binds parameter values to the prepared statement.


So one must call first DoPut then GetFlightInfo to execute a query with parameters? Wouldn't it be nicer to be able to use DoExchange for that?

I believe this was considered during the initial discussion but was rejected; @jduo do you remember why?

Ah, it was due to output parameters potentially changing the schema mid-stream. #9368 (comment)

pitrou · 2022-03-07T17:08:22Z

+ Create a new prepared statement for a SQL query.
+
+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.


Does this also close the prepared statement or can it be reused multiple times?

It is reusable (else, you don't save much from having a prepared statement).

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

lidavidm · 2022-03-09T14:30:19Z

@jduo @pitrou any other comments here? I think we can use ARROW-14698 to expand the documentation once this lands

pitrou · 2022-03-09T17:54:15Z

+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.


For the record, would there be an easy way to cross-reference the table definitions in the Flight protobuf file (such as FlightDescriptor here)? I guess not, but just asking.

Hmm, we would probably need to have Sphinx index the Protobuf file somehow and then use Sphinx's cross-reference support. Unfortunately there doesn't seem to be such integration from a quick glance (and if we wanted to do it, we would need to create a custom Protobuf plugin or modify the existing one (which supports neither Doxygen nor Sphinx): https://github.com/pseudomuto/protoc-gen-doc/

That may be worthwhile, though, since then we could avoid repeating docs between both files.

pitrou

+1

ursabot · 2022-03-09T18:52:06Z

Benchmark runs are scheduled for baseline = 875af62 and contender = 76b1403. 76b1403 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.08% ⬆️0.04%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.09% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

lidavidm · 2022-03-05T15:53:45Z

Adds a Flight SQL documentation page that we can link to from the website.

github-actions · 2022-03-05T15:54:00Z

https://issues.apache.org/jira/browse/ARROW-15721

lidavidm · 2022-03-05T15:54:15Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

lidavidm · 2022-03-05T15:56:35Z

CC @jduo if you'd like to take a look

jduo · 2022-03-06T13:36:43Z

+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints.


Let's mention that Flight SQL clients decorate existing Flight client classes to add methods corresponding to the new RPC calls.

Co-authored-by: James Duong <duong.james@gmail.com>

lidavidm · 2022-03-07T13:07:08Z

Thanks for the suggestions!

pitrou · 2022-03-07T16:55:36Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

Looks interesting. @jorisvandenbossche What do you think?

pitrou

Nice addition, thank you.

pitrou · 2022-03-07T17:06:19Z

+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.
+
+ When used with DoPut: binds parameter values to the prepared statement.


So one must call first DoPut then GetFlightInfo to execute a query with parameters? Wouldn't it be nicer to be able to use DoExchange for that?

I believe this was considered during the initial discussion but was rejected; @jduo do you remember why?

Ah, it was due to output parameters potentially changing the schema mid-stream. #9368 (comment)

pitrou · 2022-03-07T17:08:22Z

+ Create a new prepared statement for a SQL query.
+
+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.


Does this also close the prepared statement or can it be reused multiple times?

It is reusable (else, you don't save much from having a prepared statement).

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

lidavidm · 2022-03-09T14:30:19Z

@jduo @pitrou any other comments here? I think we can use ARROW-14698 to expand the documentation once this lands

pitrou · 2022-03-09T17:54:15Z

+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.


For the record, would there be an easy way to cross-reference the table definitions in the Flight protobuf file (such as FlightDescriptor here)? I guess not, but just asking.

Hmm, we would probably need to have Sphinx index the Protobuf file somehow and then use Sphinx's cross-reference support. Unfortunately there doesn't seem to be such integration from a quick glance (and if we wanted to do it, we would need to create a custom Protobuf plugin or modify the existing one (which supports neither Doxygen nor Sphinx): https://github.com/pseudomuto/protoc-gen-doc/

That may be worthwhile, though, since then we could avoid repeating docs between both files.

pitrou

+1

ursabot · 2022-03-09T18:52:06Z

Benchmark runs are scheduled for baseline = 875af62 and contender = 76b1403. 76b1403 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.08% ⬆️0.04%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.09% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

lidavidm · 2022-03-05T15:53:45Z

Adds a Flight SQL documentation page that we can link to from the website.

github-actions · 2022-03-05T15:54:00Z

https://issues.apache.org/jira/browse/ARROW-15721

lidavidm · 2022-03-05T15:54:15Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

lidavidm · 2022-03-05T15:56:35Z

CC @jduo if you'd like to take a look

jduo · 2022-03-06T13:36:43Z

+Generally, a database will implement the RPC methods according to the
+specification, but does not need to implement a client-side driver. A
+database client can use the provided Flight SQL client to interact
+with any database that supports the necessary endpoints.


Let's mention that Flight SQL clients decorate existing Flight client classes to add methods corresponding to the new RPC calls.

Co-authored-by: James Duong <duong.james@gmail.com>

lidavidm · 2022-03-07T13:07:08Z

Thanks for the suggestions!

pitrou · 2022-03-07T16:55:36Z

I would like to add sequence diagrams, how do we feel about something like sphinxcontrib-mermaid?

Looks interesting. @jorisvandenbossche What do you think?

pitrou

Nice addition, thank you.

pitrou · 2022-03-07T17:06:19Z

+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.
+
+ When used with DoPut: binds parameter values to the prepared statement.


So one must call first DoPut then GetFlightInfo to execute a query with parameters? Wouldn't it be nicer to be able to use DoExchange for that?

I believe this was considered during the initial discussion but was rejected; @jduo do you remember why?

Ah, it was due to output parameters potentially changing the schema mid-stream. #9368 (comment)

pitrou · 2022-03-07T17:08:22Z

+ Create a new prepared statement for a SQL query.
+
+``CommandPreparedStatementQuery``
+ Execute a previously created prepared statement and get the results.


Does this also close the prepared statement or can it be reused multiple times?

It is reusable (else, you don't save much from having a prepared statement).

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

lidavidm · 2022-03-09T14:30:19Z

@jduo @pitrou any other comments here? I think we can use ARROW-14698 to expand the documentation once this lands

pitrou · 2022-03-09T17:54:15Z

+All of these commands can be used with the GetFlightInfo and GetSchema
+RPC methods. The Protobuf request message should be packed into a
+google.protobuf.Any message, then serialized and packed as the ``cmd``
+field in a CMD-type FlightDescriptor.


For the record, would there be an easy way to cross-reference the table definitions in the Flight protobuf file (such as FlightDescriptor here)? I guess not, but just asking.

Hmm, we would probably need to have Sphinx index the Protobuf file somehow and then use Sphinx's cross-reference support. Unfortunately there doesn't seem to be such integration from a quick glance (and if we wanted to do it, we would need to create a custom Protobuf plugin or modify the existing one (which supports neither Doxygen nor Sphinx): https://github.com/pseudomuto/protoc-gen-doc/

That may be worthwhile, though, since then we could avoid repeating docs between both files.

pitrou

+1

ursabot · 2022-03-09T18:52:06Z

Benchmark runs are scheduled for baseline = 875af62 and contender = 76b1403. 76b1403 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.08% ⬆️0.04%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.09% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

ARROW-15721: [C++][FlightRPC] Add Flight/Flight SQL to subprojects
723a7f4

lidavidm mentioned this pull request Mar 5, 2022
ARROW-15721: [Website] Add Flight/Flight SQL to subprojects apache/arrow-site#197
Closed

apache deleted a comment from github-actionsBotMar 5, 2022

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

lidavidmand others added 2 commits March 7, 2022 08:04

Apply suggestions from code review
e32f971
Co-authored-by: James Duong <duong.james@gmail.com>

ARROW-15721: [C++][FlightRPC] Tweak wording
314d3e0

pitrou reviewed Mar 7, 2022
View reviewed changes

lidavidmand others added 2 commits March 8, 2022 08:48

Apply suggestions from code review
9f35efc
Co-authored-by: Antoine Pitrou <pitrou@free.fr>

ARROW-15721: [C++][FlightRPC] Clarify some concepts
2b7d7e5

lidavidm changed the title ~~ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects~~ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojectsMar 8, 2022

pitrou reviewed Mar 9, 2022
View reviewed changes

pitrou approved these changes Mar 9, 2022
View reviewed changes

jduo approved these changes Mar 9, 2022
View reviewed changes

lidavidm closed this in 76b1403Mar 9, 2022

lidavidm deleted the arrow-15721 branch March 11, 2022 14:10

ARROW-15721: [C++][FlightRPC] Add Flight/Flight SQL to subprojects
723a7f4

lidavidm mentioned this pull request Mar 5, 2022
ARROW-15721: [Website] Add Flight/Flight SQL to subprojects apache/arrow-site#197
Closed

apache deleted a comment from github-actionsBotMar 5, 2022

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

lidavidmand others added 2 commits March 7, 2022 08:04

Apply suggestions from code review
e32f971
Co-authored-by: James Duong <duong.james@gmail.com>

ARROW-15721: [C++][FlightRPC] Tweak wording
314d3e0

pitrou reviewed Mar 7, 2022
View reviewed changes

lidavidmand others added 2 commits March 8, 2022 08:48

Apply suggestions from code review
9f35efc
Co-authored-by: Antoine Pitrou <pitrou@free.fr>

ARROW-15721: [C++][FlightRPC] Clarify some concepts
2b7d7e5

lidavidm changed the title ~~ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects~~ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojectsMar 8, 2022

pitrou reviewed Mar 9, 2022
View reviewed changes

pitrou approved these changes Mar 9, 2022
View reviewed changes

jduo approved these changes Mar 9, 2022
View reviewed changes

lidavidm closed this in 76b1403Mar 9, 2022

lidavidm deleted the arrow-15721 branch March 11, 2022 14:10

ARROW-15721: [C++][FlightRPC] Add Flight/Flight SQL to subprojects
723a7f4

lidavidm mentioned this pull request Mar 5, 2022
ARROW-15721: [Website] Add Flight/Flight SQL to subprojects apache/arrow-site#197
Closed

apache deleted a comment from github-actionsBotMar 5, 2022

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

lidavidmand others added 2 commits March 7, 2022 08:04

Apply suggestions from code review
e32f971
Co-authored-by: James Duong <duong.james@gmail.com>

ARROW-15721: [C++][FlightRPC] Tweak wording
314d3e0

pitrou reviewed Mar 7, 2022
View reviewed changes

lidavidmand others added 2 commits March 8, 2022 08:48

Apply suggestions from code review
9f35efc
Co-authored-by: Antoine Pitrou <pitrou@free.fr>

ARROW-15721: [C++][FlightRPC] Clarify some concepts
2b7d7e5

lidavidm changed the title ~~ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects~~ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojectsMar 8, 2022

pitrou reviewed Mar 9, 2022
View reviewed changes

pitrou approved these changes Mar 9, 2022
View reviewed changes

jduo approved these changes Mar 9, 2022
View reviewed changes

lidavidm closed this in 76b1403Mar 9, 2022

lidavidm deleted the arrow-15721 branch March 11, 2022 14:10

ARROW-15721: [C++][FlightRPC] Add Flight/Flight SQL to subprojects
723a7f4

lidavidm mentioned this pull request Mar 5, 2022
ARROW-15721: [Website] Add Flight/Flight SQL to subprojects apache/arrow-site#197
Closed

apache deleted a comment from github-actionsBotMar 5, 2022

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

lidavidmand others added 2 commits March 7, 2022 08:04

Apply suggestions from code review
e32f971
Co-authored-by: James Duong <duong.james@gmail.com>

ARROW-15721: [C++][FlightRPC] Tweak wording
314d3e0

pitrou reviewed Mar 7, 2022
View reviewed changes

lidavidmand others added 2 commits March 8, 2022 08:48

Apply suggestions from code review
9f35efc
Co-authored-by: Antoine Pitrou <pitrou@free.fr>

ARROW-15721: [C++][FlightRPC] Clarify some concepts
2b7d7e5

lidavidm changed the title ~~ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects~~ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojectsMar 8, 2022

pitrou reviewed Mar 9, 2022
View reviewed changes

pitrou approved these changes Mar 9, 2022
View reviewed changes

jduo approved these changes Mar 9, 2022
View reviewed changes

lidavidm closed this in 76b1403Mar 9, 2022

lidavidm deleted the arrow-15721 branch March 11, 2022 14:10

ARROW-15721: [C++][FlightRPC] Add Flight/Flight SQL to subprojects
723a7f4

lidavidm mentioned this pull request Mar 5, 2022
ARROW-15721: [Website] Add Flight/Flight SQL to subprojects apache/arrow-site#197
Closed

apache deleted a comment from github-actionsBotMar 5, 2022

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

lidavidmand others added 2 commits March 7, 2022 08:04

Apply suggestions from code review
e32f971
Co-authored-by: James Duong <duong.james@gmail.com>

ARROW-15721: [C++][FlightRPC] Tweak wording
314d3e0

pitrou reviewed Mar 7, 2022
View reviewed changes

lidavidmand others added 2 commits March 8, 2022 08:48

Apply suggestions from code review
9f35efc
Co-authored-by: Antoine Pitrou <pitrou@free.fr>

ARROW-15721: [C++][FlightRPC] Clarify some concepts
2b7d7e5

lidavidm changed the title ~~ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects~~ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojectsMar 8, 2022

pitrou reviewed Mar 9, 2022
View reviewed changes

pitrou approved these changes Mar 9, 2022
View reviewed changes

jduo approved these changes Mar 9, 2022
View reviewed changes

lidavidm closed this in 76b1403Mar 9, 2022

lidavidm deleted the arrow-15721 branch March 11, 2022 14:10

ARROW-15721: [C++][FlightRPC] Add Flight/Flight SQL to subprojects
723a7f4

lidavidm mentioned this pull request Mar 5, 2022
ARROW-15721: [Website] Add Flight/Flight SQL to subprojects apache/arrow-site#197
Closed

apache deleted a comment from github-actionsBotMar 5, 2022

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

lidavidmand others added 2 commits March 7, 2022 08:04

Apply suggestions from code review
e32f971
Co-authored-by: James Duong <duong.james@gmail.com>

ARROW-15721: [C++][FlightRPC] Tweak wording
314d3e0

pitrou reviewed Mar 7, 2022
View reviewed changes

lidavidmand others added 2 commits March 8, 2022 08:48

Apply suggestions from code review
9f35efc
Co-authored-by: Antoine Pitrou <pitrou@free.fr>

ARROW-15721: [C++][FlightRPC] Clarify some concepts
2b7d7e5

lidavidm changed the title ~~ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects~~ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojectsMar 8, 2022

pitrou reviewed Mar 9, 2022
View reviewed changes

pitrou approved these changes Mar 9, 2022
View reviewed changes

jduo approved these changes Mar 9, 2022
View reviewed changes

lidavidm closed this in 76b1403Mar 9, 2022

lidavidm deleted the arrow-15721 branch March 11, 2022 14:10

ARROW-15721: [C++][FlightRPC] Add Flight/Flight SQL to subprojects
723a7f4

lidavidm mentioned this pull request Mar 5, 2022
ARROW-15721: [Website] Add Flight/Flight SQL to subprojects apache/arrow-site#197
Closed

apache deleted a comment from github-actionsBotMar 5, 2022

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

lidavidmand others added 2 commits March 7, 2022 08:04

Apply suggestions from code review
e32f971
Co-authored-by: James Duong <duong.james@gmail.com>

ARROW-15721: [C++][FlightRPC] Tweak wording
314d3e0

pitrou reviewed Mar 7, 2022
View reviewed changes

lidavidmand others added 2 commits March 8, 2022 08:48

Apply suggestions from code review
9f35efc
Co-authored-by: Antoine Pitrou <pitrou@free.fr>

ARROW-15721: [C++][FlightRPC] Clarify some concepts
2b7d7e5

lidavidm changed the title ~~ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects~~ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojectsMar 8, 2022

pitrou reviewed Mar 9, 2022
View reviewed changes

pitrou approved these changes Mar 9, 2022
View reviewed changes

jduo approved these changes Mar 9, 2022
View reviewed changes

lidavidm closed this in 76b1403Mar 9, 2022

lidavidm deleted the arrow-15721 branch March 11, 2022 14:10

ARROW-15721: [C++][FlightRPC] Add Flight/Flight SQL to subprojects
723a7f4

lidavidm mentioned this pull request Mar 5, 2022
ARROW-15721: [Website] Add Flight/Flight SQL to subprojects apache/arrow-site#197
Closed

apache deleted a comment from github-actionsBotMar 5, 2022

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

Comment threaddocs/source/format/FlightSql.rst Outdated

jduo reviewed Mar 6, 2022
View reviewed changes

lidavidmand others added 2 commits March 7, 2022 08:04

Apply suggestions from code review
e32f971
Co-authored-by: James Duong <duong.james@gmail.com>

ARROW-15721: [C++][FlightRPC] Tweak wording
314d3e0

pitrou reviewed Mar 7, 2022
View reviewed changes

lidavidmand others added 2 commits March 8, 2022 08:48

Apply suggestions from code review
9f35efc
Co-authored-by: Antoine Pitrou <pitrou@free.fr>

ARROW-15721: [C++][FlightRPC] Clarify some concepts
2b7d7e5

lidavidm changed the title ~~ARROW-15721: [Docs][FlightSQL] Add Flight/Flight SQL to subprojects~~ARROW-15721: [Docs][FlightRPC] Add Flight/Flight SQL to subprojectsMar 8, 2022

pitrou reviewed Mar 9, 2022
View reviewed changes

pitrou approved these changes Mar 9, 2022
View reviewed changes

jduo approved these changes Mar 9, 2022
View reviewed changes

lidavidm closed this in 76b1403Mar 9, 2022

lidavidm deleted the arrow-15721 branch March 11, 2022 14:10

Uh oh!

Conversation

lidavidm commented Mar 5, 2022

Uh oh!

github-actionsBot commented Mar 5, 2022

Uh oh!

lidavidm commented Mar 5, 2022

Uh oh!

lidavidm commented Mar 5, 2022

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidavidm commented Mar 7, 2022

Uh oh!

pitrou commented Mar 7, 2022

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidavidm commented Mar 9, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

ursabot commented Mar 9, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

lidavidm commented Mar 5, 2022

Uh oh!

github-actionsBot commented Mar 5, 2022

Uh oh!

lidavidm commented Mar 5, 2022

Uh oh!

lidavidm commented Mar 5, 2022

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidavidm commented Mar 7, 2022

Uh oh!

pitrou commented Mar 7, 2022

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ursabot commented Mar 9, 2022•
edited
Loading

ursabot commented Mar 9, 2022•
edited
Loading