Skip to content

ARROW-5182: [Rust] Arrow IPC file writer - #6107

Closed
nevi-me wants to merge 2 commits into
apache:masterfrom
nevi-me:ARROW-5182
Closed

ARROW-5182: [Rust] Arrow IPC file writer#6107
nevi-me wants to merge 2 commits into
apache:masterfrom
nevi-me:ARROW-5182

Conversation

@nevi-me

@nevi-menevi-me commented Dec 29, 2019

Copy link
Copy Markdown
Contributor

This is committed on top of the PR for the stream reader, and should be merged after it.

@github-actions

github-actionsBot commented Dec 29, 2019

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-5182

nevi-me
nevi-me commented Jan 1, 2020
View reviewed changes
Comment threadrust/arrow/src/ipc/writer.rs Outdated

@nevi-menevi-meJan 1, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need help here, I couldn't find a way of reusing the functions in crate::ipc::convert

@maxburkemaxburkeJan 4, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pulled this change down and played with it; I'm pretty excited to get the IPC writer for Rust :). I've run into similar challenges before with flatbuffers, and sometimes there's no good way of working around it.

Would it be worth converting schema_to_fb_offset to a macro maybe?

@nevi-menevi-meJan 5, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can try the macro approach if I can't find another workaround, I could do this when working on the StreamWriter given that it'll share most code with the FileWriter.

@andygrove

andygrove commented Jan 1, 2020

Copy link
Copy Markdown
Member

Hi @nevi-me I am back at work tomorrow and will start reviewing this.

@maxburke

maxburke commented Jan 17, 2020

Copy link
Copy Markdown
Contributor

Hi! Is there anything I could be able to do to help with this merge?

@nevi-me

nevi-me commented Jan 18, 2020

Copy link
Copy Markdown
ContributorAuthor

Hey @maxburke, the next release is a few days away, so I doubt this will make the cut. If you need to use the IPC writer for now, you can work from my branch in the short term. I can keep this PR updated with any other Rust changes in master.

Do you also need a StreamWriter? I am holding off from working on it until we review and merge dictionary support and this file writer, but I could work on it if you need it sooner.

nevi-me
nevi-me commented Jan 18, 2020
View reviewed changes
Comment threadrust/arrow/src/ipc/writer.rs Outdated

@nevi-menevi-meJan 18, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ended up not using this, will remove it as it duplicates block_offsets

@maxburke

maxburke commented Jan 19, 2020

Copy link
Copy Markdown
Contributor

The StreamWriter would be great to have but I've mostly worked around it for now with the FileWriter :)

I'm actually currently working from your branch already.

Really appreciate your help and progress on this!

@nevi-menevi-me mentioned this pull request Jan 25, 2020
Closed
andygrove
andygrove reviewed Jan 26, 2020
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking great @nevi-me! What help do you currently need in order to merge this?

@nevi-me

nevi-me commented Jan 26, 2020

Copy link
Copy Markdown
ContributorAuthor

@andygrove I'm just waiting for a review

andygrove
andygrove approved these changes Jan 26, 2020
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andygrove

andygrove commented Jan 26, 2020

Copy link
Copy Markdown
Member

@nevi-me After merging to master, I'm seeing a build failure in master:

failures:
---- ipc::writer::tests::read_and_rewrite_generated_files stdout ----
thread 'ipc::writer::tests::read_and_rewrite_generated_files' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', src/libcore/result.rs:1187:5

@nevi-me

nevi-me commented Jan 26, 2020

Copy link
Copy Markdown
ContributorAuthor

It's also happening on other PRs, I haven't had a chance to look at it. I'll look into it tomorrow morning

andygrove pushed a commit that referenced this pull request Jan 26, 2020
@nevi-me@andygrove
ARROW-7475: [Rust] Arrow IPC Stream writer
b84d823
The stream writer is applied on top of ARROW-5182#6107 (file writer) and should be merged after.
Closes#6281 from nevi-me/ARROW-7475 and squashes the following commits:
11ed168 <Neville Dipale> fix IPC test file location
0f90b3e <Neville Dipale> ARROW-7475: Arrow IPC Stream writer
cb82160 <Neville Dipale> remove redundant header_bytes
Authored-by: Neville Dipale <nevilledips@gmail.com>
Signed-off-by: Andy Grove <andygrove73@gmail.com>
@nevi-me
nevi-me deleted the ARROW-5182 branch October 17, 2020 19:18
@asfimportasfimport mentioned this pull request Jan 26, 2020
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andygroveandygroveandygrove approved these changes
@paddyhoranpaddyhoranAwaiting requested review from paddyhoran
@sunchaosunchaoAwaiting requested review from sunchao
+1 more reviewer
@maxburkemaxburkemaxburke left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Component: Rust

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nevi-me@andygrove@maxburke