Skip to content

ARROW-10938: [Rust] upgrade dependency "flatbuffers" to 0.8 - #8936

Closed
mqy wants to merge 7 commits into
apache:masterfrom
mqy:flatbuffers-0.8.0
Closed

ARROW-10938: [Rust] upgrade dependency "flatbuffers" to 0.8#8936
mqy wants to merge 7 commits into
apache:masterfrom
mqy:flatbuffers-0.8.0

Conversation

@mqy

@mqymqy commented Dec 16, 2020

Copy link
Copy Markdown
Contributor

flatbuffers 0.8.0 was released on Dec 10, 2020, with some notable changes:

  • new verifier
  • common rust traits to FlatBufferBuilder
  • new VectorIter
  • add FlatBufferBuilder::force_defaults API
  • Optional Scalars
  • up to 2018 edition
  • possible performance speedup
  • ... and minor breaking change to some APIs, for example: remove "get_", return Result.

flatbuffers 0.8.0 requires the latest flatc, the git commit for flatc is updated too.

I deliberately commit all changes step by step to make them clear.

@github-actions

Copy link
Copy Markdown

Comment threadrust/arrow/src/ipc/convert.rs Outdated
Comment threadrust/parquet/src/arrow/schema.rs Outdated
message
.header_as_schema()
.map(arrow::ipc::convert::fb_to_schema)
if let Ok(message) = arrow::ipc::root_as_message(slice) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are ignoring errors here too. Is this the intent?

@mqymqyDec 17, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DONE. Since there is already a eprintln!(), the new error is caught with eprintln!() too.
We may improve them later, right?

BTW, I failed to change the return type of get_arrow_schema_from_metadata() from Option<Schema> to Result<Schema> or Result<Option<Schema>>.

Comment threadrust/arrow-flight/src/utils.rs Outdated
@andygrove

Copy link
Copy Markdown
Member

Thanks @mqy this is looking good and I appreciate you breaking this down into the individual commits. I have some questions on error handling but other than that it LGTM.

@mqy

mqy commented Dec 16, 2020

Copy link
Copy Markdown
ContributorAuthor

Thanks @mqy this is looking good and I appreciate you breaking this down into the individual commits. I have some questions on error handling but other than that it LGTM.

@andygrove that's great! Since i'm pretty new to rust, it may take me some time to completely fix/enhance the "Option -> Result problems", and time to sleep for now.

I'll continue this PR next morning, if ... it is not merged :)

@codecov-io

codecov-io commented Dec 17, 2020

Copy link
Copy Markdown

Codecov Report

Merging #8936 (0dff5ea) into master (6fafedf) will decrease coverage by 0.41%.
The diff coverage is 38.02%.

Impacted file tree graph

@@ Coverage Diff @@## master #8936 +/- ##
==========================================
- Coverage 83.26% 82.85% -0.42% 
==========================================
Files 195 196 +1 Lines 48066 48577 +511 ==========================================
+ Hits 40023 40249 +226 - Misses 8043 8328 +285 
Impacted FilesCoverage Δ
rust/arrow-flight/src/utils.rs0.00% <0.00%> (ø)
rust/arrow/src/ipc/gen/SparseTensor.rs0.00% <0.00%> (ø)
rust/arrow/src/ipc/gen/Tensor.rs0.00% <0.00%> (ø)
rust/arrow/src/ipc/convert.rs91.90% <27.27%> (-1.11%)⬇️
rust/arrow/src/ipc/gen/File.rs40.94% <39.28%> (-2.20%)⬇️
rust/parquet/src/arrow/schema.rs91.36% <40.00%> (-0.45%)⬇️
rust/arrow/src/ipc/gen/Message.rs32.03% <42.10%> (+2.21%)⬆️
rust/arrow/src/ipc/reader.rs82.86% <50.00%> (-0.84%)⬇️
rust/arrow/src/ipc/gen/Schema.rs38.75% <62.55%> (+5.58%)⬆️
rust/datafusion/src/execution/context.rs89.95% <0.00%> (-1.93%)⬇️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6fafedf...0dff5ea. Read the comment docs.

@mqy
mqy requested a review from andygroveDecember 17, 2020 09:57
@alamb

Copy link
Copy Markdown
Contributor

The conda integration test failed due to what appears to be some unrelated error:
Screen Shot 2020-12-17 at 6 49 18 PM

I will restart the check to see if we can get a green CI run

@mqy

mqy commented Dec 18, 2020

Copy link
Copy Markdown
ContributorAuthor

@alamb pushed another commit 0dff5ea1ee

It's the update after fixing flatc rust generator, google/flatbuffers 05192553 FYI.

@nevi-menevi-me left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, we should open a JIRA to move us to the next flatbuffers release

@alamb

Copy link
Copy Markdown
Contributor

Thanks @mqy -- I'll try and look at this carefully tomorrow.

@alamb

Copy link
Copy Markdown
Contributor

LGTM, we should open a JIRA to move us to the next flatbuffers release

@nevi-me -- filed https://issues.apache.org/jira/browse/ARROW-10997

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this commit by commit -- the breakdown of this PR into commits was super helpful. I'll file some tickets to handle follow on improvements but then I plan to merge this one in.

Thanks again @mqy

.header_as_schema()
.map(arrow::ipc::convert::fb_to_schema),
Err(err) => {
// The flatbuffers implementation returns an error on verification error.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I filed https://issues.apache.org/jira/browse/ARROW-10996 to track returning this is a proper error message. I think this PR improves the situation so at least eprintln! is called where prior to this PR no error message is created.

@alamb

Copy link
Copy Markdown
Contributor

I merged this branch into apache/master and ran the tests locally just to be sure things are good. Merging

@alambalamb closed this in f52a100Dec 21, 2020
jorgecarleitao pushed a commit that referenced this pull request Jan 1, 2021
…chema_from_metadata()
#8936 updated crate `flatbuffers` to 0.8.0 , but function `get_arrow_schema_from_metadata` still returning `Option` rather than `Result`. This PR fixes this issue.
Closes#9058 from mqy/get_arrow_schema_from_metadata
Authored-by: mqy <meng.qingyou@gmail.com>
Signed-off-by: Jorge C. Leitao <jorgecarleitao@gmail.com>
alamb pushed a commit to apache/arrow-rs that referenced this pull request Apr 20, 2021
…chema_from_metadata()
apache/arrow#8936 updated crate `flatbuffers` to 0.8.0 , but function `get_arrow_schema_from_metadata` still returning `Option` rather than `Result`. This PR fixes this issue.
Closes#9058 from mqy/get_arrow_schema_from_metadata
Authored-by: mqy <meng.qingyou@gmail.com>
Signed-off-by: Jorge C. Leitao <jorgecarleitao@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@mqy@andygrove@codecov-io@alamb@nevi-me