ARROW-4219: [Rust] [Parquet] Initial support for arrow reader. - #5523

Closed
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219
Closed

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader.#5523
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219

Conversation

@liurenjie1024

@liurenjie1024liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
Contributor

Initial support of arrow reader, which reads parquet into arrow record batch.

@liurenjie1024

liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
ContributorAuthor

@sunchao@andygrove@nevi-me@paddyhoran Please take a look when you are available.

andygrove
andygrove reviewed Sep 27, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM so far but I have a couple of questions in this review

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a very similar trait in DataFusion, only it inherits Sync + Send too so that it can be passed between threads. Maybe we could do that here too? See https://github.com/apache/arrow/blob/master/rust/datafusion/src/execution/physical_plan/mod.rs#L44-L50

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's safe to make this trait as Send+Sync because the reader used some unsafe data(e.g. MutableBuffer).

@andygroveandygroveSep 30, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, makes sense.

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is returned if there are no more batches? Shouldn't this return Result<Option<RecordBatch>> ?

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently if we reached EOF, it returns zero length array. But I think it's better to change return type to Result<Option<RecordBatch>> . Will fix it.

@liurenjie1024

liurenjie1024 commented Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload some generated files to arrow-testing repo for this integration test. How am I supposed to do that? Should I submit another PR in arrow-testing project?

@github-actions

github-actionsBot commented Sep 29, 2019

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-4219

@andygrove

andygrove commented Sep 30, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 If you are adding Parquet files then you can PR against https://github.com/apache/parquet-testing but please note the comments in the last PR merged there which suggests we look into alternate approaches: apache/parquet-testing#9

@liurenjie1024

liurenjie1024 commented Sep 30, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload not only parquet file, but also json files and proto files. Here is my approach to do integration test: I use protobuf to define schema, then generate both json and parquet file with that schema using java code. Then I use rust code to load parquet and json file to compare them. I guess maybe it's better to upload them into arrow-testing library?

@andygrove

andygrove commented Oct 1, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 I think it would be good to start a discussion about this on the mailing list. I'm not sure what advice to give on this.

@alippai

alippai commented Oct 7, 2019

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024

liurenjie1024 commented Oct 8, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove Sorry for late reply. I've submitted a PR in arrow-testing repo: apache/arrow-testing#11
Let's discuss there.

@liurenjie1024

liurenjie1024 commented Oct 13, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove This is ready for review now.

@liurenjie1024liurenjie1024 mentioned this pull request Oct 14, 2019
Closed
nevi-me
nevi-me approved these changes Oct 14, 2019
View reviewed changes

@nevi-menevi-me left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nevi-me

nevi-me commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024 is this an intentional limitation for now? (

arrow/rust/parquet/src/arrow/array_reader.rs

Lines 219 to 222 in 8b915b3

(arrow_type, _) => Err(general_err!(
"Reading {:?} type from parquet is not supported yet.",
arrow_type
)),
)

Also, my apologies for the late review.

@liurenjie1024

liurenjie1024 commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@nevi-me No, it just takes time to implement. Will implement in future.

andygrove
andygrove approved these changes Oct 14, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @liurenjie1024

@asfimportasfimport mentioned this pull request Oct 14, 2019
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andygroveandygroveandygrove approved these changes
@nevi-menevi-menevi-me approved these changes

Assignees

No one assigned

Labels

Component: Rust

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liurenjie1024@andygrove@alippai@nevi-me
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all \u003cpre\u003e\u003ccode\u003e blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks"); } } catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); } })(); (function(){ try { var __m = "github.com"; var __re = new RegExp('^' + "github\\.com" + '
Skip to content

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader. - #5523

Closed
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219
Closed

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader.#5523
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219

Conversation

@liurenjie1024

@liurenjie1024liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
Contributor

Initial support of arrow reader, which reads parquet into arrow record batch.

@liurenjie1024

liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
ContributorAuthor

@sunchao@andygrove@nevi-me@paddyhoran Please take a look when you are available.

andygrove
andygrove reviewed Sep 27, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM so far but I have a couple of questions in this review

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a very similar trait in DataFusion, only it inherits Sync + Send too so that it can be passed between threads. Maybe we could do that here too? See https://github.com/apache/arrow/blob/master/rust/datafusion/src/execution/physical_plan/mod.rs#L44-L50

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's safe to make this trait as Send+Sync because the reader used some unsafe data(e.g. MutableBuffer).

@andygroveandygroveSep 30, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, makes sense.

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is returned if there are no more batches? Shouldn't this return Result<Option<RecordBatch>> ?

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently if we reached EOF, it returns zero length array. But I think it's better to change return type to Result<Option<RecordBatch>> . Will fix it.

@liurenjie1024

liurenjie1024 commented Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload some generated files to arrow-testing repo for this integration test. How am I supposed to do that? Should I submit another PR in arrow-testing project?

@github-actions

github-actionsBot commented Sep 29, 2019

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-4219

@andygrove

andygrove commented Sep 30, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 If you are adding Parquet files then you can PR against https://github.com/apache/parquet-testing but please note the comments in the last PR merged there which suggests we look into alternate approaches: apache/parquet-testing#9

@liurenjie1024

liurenjie1024 commented Sep 30, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload not only parquet file, but also json files and proto files. Here is my approach to do integration test: I use protobuf to define schema, then generate both json and parquet file with that schema using java code. Then I use rust code to load parquet and json file to compare them. I guess maybe it's better to upload them into arrow-testing library?

@andygrove

andygrove commented Oct 1, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 I think it would be good to start a discussion about this on the mailing list. I'm not sure what advice to give on this.

@alippai

alippai commented Oct 7, 2019

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024

liurenjie1024 commented Oct 8, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove Sorry for late reply. I've submitted a PR in arrow-testing repo: apache/arrow-testing#11
Let's discuss there.

@liurenjie1024

liurenjie1024 commented Oct 13, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove This is ready for review now.

@liurenjie1024liurenjie1024 mentioned this pull request Oct 14, 2019
Closed
nevi-me
nevi-me approved these changes Oct 14, 2019
View reviewed changes

@nevi-menevi-me left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nevi-me

nevi-me commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024 is this an intentional limitation for now? (

arrow/rust/parquet/src/arrow/array_reader.rs

Lines 219 to 222 in 8b915b3

(arrow_type, _) => Err(general_err!(
"Reading {:?} type from parquet is not supported yet.",
arrow_type
)),
)

Also, my apologies for the late review.

@liurenjie1024

liurenjie1024 commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@nevi-me No, it just takes time to implement. Will implement in future.

andygrove
andygrove approved these changes Oct 14, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @liurenjie1024

@asfimportasfimport mentioned this pull request Oct 14, 2019
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andygroveandygroveandygrove approved these changes
@nevi-menevi-menevi-me approved these changes

Assignees

No one assigned

Labels

Component: Rust

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liurenjie1024@andygrove@alippai@nevi-me
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader. - #5523

Closed
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219
Closed

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader.#5523
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219

Conversation

@liurenjie1024

@liurenjie1024liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
Contributor

Initial support of arrow reader, which reads parquet into arrow record batch.

@liurenjie1024

liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
ContributorAuthor

@sunchao@andygrove@nevi-me@paddyhoran Please take a look when you are available.

andygrove
andygrove reviewed Sep 27, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM so far but I have a couple of questions in this review

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a very similar trait in DataFusion, only it inherits Sync + Send too so that it can be passed between threads. Maybe we could do that here too? See https://github.com/apache/arrow/blob/master/rust/datafusion/src/execution/physical_plan/mod.rs#L44-L50

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's safe to make this trait as Send+Sync because the reader used some unsafe data(e.g. MutableBuffer).

@andygroveandygroveSep 30, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, makes sense.

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is returned if there are no more batches? Shouldn't this return Result<Option<RecordBatch>> ?

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently if we reached EOF, it returns zero length array. But I think it's better to change return type to Result<Option<RecordBatch>> . Will fix it.

@liurenjie1024

liurenjie1024 commented Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload some generated files to arrow-testing repo for this integration test. How am I supposed to do that? Should I submit another PR in arrow-testing project?

@github-actions

github-actionsBot commented Sep 29, 2019

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-4219

@andygrove

andygrove commented Sep 30, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 If you are adding Parquet files then you can PR against https://github.com/apache/parquet-testing but please note the comments in the last PR merged there which suggests we look into alternate approaches: apache/parquet-testing#9

@liurenjie1024

liurenjie1024 commented Sep 30, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload not only parquet file, but also json files and proto files. Here is my approach to do integration test: I use protobuf to define schema, then generate both json and parquet file with that schema using java code. Then I use rust code to load parquet and json file to compare them. I guess maybe it's better to upload them into arrow-testing library?

@andygrove

andygrove commented Oct 1, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 I think it would be good to start a discussion about this on the mailing list. I'm not sure what advice to give on this.

@alippai

alippai commented Oct 7, 2019

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024

liurenjie1024 commented Oct 8, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove Sorry for late reply. I've submitted a PR in arrow-testing repo: apache/arrow-testing#11
Let's discuss there.

@liurenjie1024

liurenjie1024 commented Oct 13, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove This is ready for review now.

@liurenjie1024liurenjie1024 mentioned this pull request Oct 14, 2019
Closed
nevi-me
nevi-me approved these changes Oct 14, 2019
View reviewed changes

@nevi-menevi-me left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nevi-me

nevi-me commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024 is this an intentional limitation for now? (

arrow/rust/parquet/src/arrow/array_reader.rs

Lines 219 to 222 in 8b915b3

(arrow_type, _) => Err(general_err!(
"Reading {:?} type from parquet is not supported yet.",
arrow_type
)),
)

Also, my apologies for the late review.

@liurenjie1024

liurenjie1024 commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@nevi-me No, it just takes time to implement. Will implement in future.

andygrove
andygrove approved these changes Oct 14, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @liurenjie1024

@asfimportasfimport mentioned this pull request Oct 14, 2019
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andygroveandygroveandygrove approved these changes
@nevi-menevi-menevi-me approved these changes

Assignees

No one assigned

Labels

Component: Rust

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liurenjie1024@andygrove@alippai@nevi-me
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length \u003e 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader. - #5523

Closed
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219
Closed

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader.#5523
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219

Conversation

@liurenjie1024

@liurenjie1024liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
Contributor

Initial support of arrow reader, which reads parquet into arrow record batch.

@liurenjie1024

liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
ContributorAuthor

@sunchao@andygrove@nevi-me@paddyhoran Please take a look when you are available.

andygrove
andygrove reviewed Sep 27, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM so far but I have a couple of questions in this review

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a very similar trait in DataFusion, only it inherits Sync + Send too so that it can be passed between threads. Maybe we could do that here too? See https://github.com/apache/arrow/blob/master/rust/datafusion/src/execution/physical_plan/mod.rs#L44-L50

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's safe to make this trait as Send+Sync because the reader used some unsafe data(e.g. MutableBuffer).

@andygroveandygroveSep 30, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, makes sense.

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is returned if there are no more batches? Shouldn't this return Result<Option<RecordBatch>> ?

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently if we reached EOF, it returns zero length array. But I think it's better to change return type to Result<Option<RecordBatch>> . Will fix it.

@liurenjie1024

liurenjie1024 commented Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload some generated files to arrow-testing repo for this integration test. How am I supposed to do that? Should I submit another PR in arrow-testing project?

@github-actions

github-actionsBot commented Sep 29, 2019

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-4219

@andygrove

andygrove commented Sep 30, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 If you are adding Parquet files then you can PR against https://github.com/apache/parquet-testing but please note the comments in the last PR merged there which suggests we look into alternate approaches: apache/parquet-testing#9

@liurenjie1024

liurenjie1024 commented Sep 30, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload not only parquet file, but also json files and proto files. Here is my approach to do integration test: I use protobuf to define schema, then generate both json and parquet file with that schema using java code. Then I use rust code to load parquet and json file to compare them. I guess maybe it's better to upload them into arrow-testing library?

@andygrove

andygrove commented Oct 1, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 I think it would be good to start a discussion about this on the mailing list. I'm not sure what advice to give on this.

@alippai

alippai commented Oct 7, 2019

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024

liurenjie1024 commented Oct 8, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove Sorry for late reply. I've submitted a PR in arrow-testing repo: apache/arrow-testing#11
Let's discuss there.

@liurenjie1024

liurenjie1024 commented Oct 13, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove This is ready for review now.

@liurenjie1024liurenjie1024 mentioned this pull request Oct 14, 2019
Closed
nevi-me
nevi-me approved these changes Oct 14, 2019
View reviewed changes

@nevi-menevi-me left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nevi-me

nevi-me commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024 is this an intentional limitation for now? (

arrow/rust/parquet/src/arrow/array_reader.rs

Lines 219 to 222 in 8b915b3

(arrow_type, _) => Err(general_err!(
"Reading {:?} type from parquet is not supported yet.",
arrow_type
)),
)

Also, my apologies for the late review.

@liurenjie1024

liurenjie1024 commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@nevi-me No, it just takes time to implement. Will implement in future.

andygrove
andygrove approved these changes Oct 14, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @liurenjie1024

@asfimportasfimport mentioned this pull request Oct 14, 2019
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andygroveandygroveandygrove approved these changes
@nevi-menevi-menevi-me approved these changes

Assignees

No one assigned

Labels

Component: Rust

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liurenjie1024@andygrove@alippai@nevi-me
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader. - #5523

Closed
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219
Closed

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader.#5523
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219

Conversation

@liurenjie1024

@liurenjie1024liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
Contributor

Initial support of arrow reader, which reads parquet into arrow record batch.

@liurenjie1024

liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
ContributorAuthor

@sunchao@andygrove@nevi-me@paddyhoran Please take a look when you are available.

andygrove
andygrove reviewed Sep 27, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM so far but I have a couple of questions in this review

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a very similar trait in DataFusion, only it inherits Sync + Send too so that it can be passed between threads. Maybe we could do that here too? See https://github.com/apache/arrow/blob/master/rust/datafusion/src/execution/physical_plan/mod.rs#L44-L50

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's safe to make this trait as Send+Sync because the reader used some unsafe data(e.g. MutableBuffer).

@andygroveandygroveSep 30, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, makes sense.

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is returned if there are no more batches? Shouldn't this return Result<Option<RecordBatch>> ?

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently if we reached EOF, it returns zero length array. But I think it's better to change return type to Result<Option<RecordBatch>> . Will fix it.

@liurenjie1024

liurenjie1024 commented Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload some generated files to arrow-testing repo for this integration test. How am I supposed to do that? Should I submit another PR in arrow-testing project?

@github-actions

github-actionsBot commented Sep 29, 2019

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-4219

@andygrove

andygrove commented Sep 30, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 If you are adding Parquet files then you can PR against https://github.com/apache/parquet-testing but please note the comments in the last PR merged there which suggests we look into alternate approaches: apache/parquet-testing#9

@liurenjie1024

liurenjie1024 commented Sep 30, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload not only parquet file, but also json files and proto files. Here is my approach to do integration test: I use protobuf to define schema, then generate both json and parquet file with that schema using java code. Then I use rust code to load parquet and json file to compare them. I guess maybe it's better to upload them into arrow-testing library?

@andygrove

andygrove commented Oct 1, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 I think it would be good to start a discussion about this on the mailing list. I'm not sure what advice to give on this.

@alippai

alippai commented Oct 7, 2019

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024

liurenjie1024 commented Oct 8, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove Sorry for late reply. I've submitted a PR in arrow-testing repo: apache/arrow-testing#11
Let's discuss there.

@liurenjie1024

liurenjie1024 commented Oct 13, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove This is ready for review now.

@liurenjie1024liurenjie1024 mentioned this pull request Oct 14, 2019
Closed
nevi-me
nevi-me approved these changes Oct 14, 2019
View reviewed changes

@nevi-menevi-me left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nevi-me

nevi-me commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024 is this an intentional limitation for now? (

arrow/rust/parquet/src/arrow/array_reader.rs

Lines 219 to 222 in 8b915b3

(arrow_type, _) => Err(general_err!(
"Reading {:?} type from parquet is not supported yet.",
arrow_type
)),
)

Also, my apologies for the late review.

@liurenjie1024

liurenjie1024 commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@nevi-me No, it just takes time to implement. Will implement in future.

andygrove
andygrove approved these changes Oct 14, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @liurenjie1024

@asfimportasfimport mentioned this pull request Oct 14, 2019
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andygroveandygroveandygrove approved these changes
@nevi-menevi-menevi-me approved these changes

Assignees

No one assigned

Labels

Component: Rust

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liurenjie1024@andygrove@alippai@nevi-me
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader. - #5523

Closed
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219
Closed

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader.#5523
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219

Conversation

@liurenjie1024

@liurenjie1024liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
Contributor

Initial support of arrow reader, which reads parquet into arrow record batch.

@liurenjie1024

liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
ContributorAuthor

@sunchao@andygrove@nevi-me@paddyhoran Please take a look when you are available.

andygrove
andygrove reviewed Sep 27, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM so far but I have a couple of questions in this review

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a very similar trait in DataFusion, only it inherits Sync + Send too so that it can be passed between threads. Maybe we could do that here too? See https://github.com/apache/arrow/blob/master/rust/datafusion/src/execution/physical_plan/mod.rs#L44-L50

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's safe to make this trait as Send+Sync because the reader used some unsafe data(e.g. MutableBuffer).

@andygroveandygroveSep 30, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, makes sense.

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is returned if there are no more batches? Shouldn't this return Result<Option<RecordBatch>> ?

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently if we reached EOF, it returns zero length array. But I think it's better to change return type to Result<Option<RecordBatch>> . Will fix it.

@liurenjie1024

liurenjie1024 commented Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload some generated files to arrow-testing repo for this integration test. How am I supposed to do that? Should I submit another PR in arrow-testing project?

@github-actions

github-actionsBot commented Sep 29, 2019

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-4219

@andygrove

andygrove commented Sep 30, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 If you are adding Parquet files then you can PR against https://github.com/apache/parquet-testing but please note the comments in the last PR merged there which suggests we look into alternate approaches: apache/parquet-testing#9

@liurenjie1024

liurenjie1024 commented Sep 30, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload not only parquet file, but also json files and proto files. Here is my approach to do integration test: I use protobuf to define schema, then generate both json and parquet file with that schema using java code. Then I use rust code to load parquet and json file to compare them. I guess maybe it's better to upload them into arrow-testing library?

@andygrove

andygrove commented Oct 1, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 I think it would be good to start a discussion about this on the mailing list. I'm not sure what advice to give on this.

@alippai

alippai commented Oct 7, 2019

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024

liurenjie1024 commented Oct 8, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove Sorry for late reply. I've submitted a PR in arrow-testing repo: apache/arrow-testing#11
Let's discuss there.

@liurenjie1024

liurenjie1024 commented Oct 13, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove This is ready for review now.

@liurenjie1024liurenjie1024 mentioned this pull request Oct 14, 2019
Closed
nevi-me
nevi-me approved these changes Oct 14, 2019
View reviewed changes

@nevi-menevi-me left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nevi-me

nevi-me commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024 is this an intentional limitation for now? (

arrow/rust/parquet/src/arrow/array_reader.rs

Lines 219 to 222 in 8b915b3

(arrow_type, _) => Err(general_err!(
"Reading {:?} type from parquet is not supported yet.",
arrow_type
)),
)

Also, my apologies for the late review.

@liurenjie1024

liurenjie1024 commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@nevi-me No, it just takes time to implement. Will implement in future.

andygrove
andygrove approved these changes Oct 14, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @liurenjie1024

@asfimportasfimport mentioned this pull request Oct 14, 2019
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andygroveandygroveandygrove approved these changes
@nevi-menevi-menevi-me approved these changes

Assignees

No one assigned

Labels

Component: Rust

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liurenjie1024@andygrove@alippai@nevi-me
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader. - #5523

Closed
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219
Closed

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader.#5523
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219

Conversation

@liurenjie1024

@liurenjie1024liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
Contributor

Initial support of arrow reader, which reads parquet into arrow record batch.

@liurenjie1024

liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
ContributorAuthor

@sunchao@andygrove@nevi-me@paddyhoran Please take a look when you are available.

andygrove
andygrove reviewed Sep 27, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM so far but I have a couple of questions in this review

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a very similar trait in DataFusion, only it inherits Sync + Send too so that it can be passed between threads. Maybe we could do that here too? See https://github.com/apache/arrow/blob/master/rust/datafusion/src/execution/physical_plan/mod.rs#L44-L50

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's safe to make this trait as Send+Sync because the reader used some unsafe data(e.g. MutableBuffer).

@andygroveandygroveSep 30, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, makes sense.

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is returned if there are no more batches? Shouldn't this return Result<Option<RecordBatch>> ?

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently if we reached EOF, it returns zero length array. But I think it's better to change return type to Result<Option<RecordBatch>> . Will fix it.

@liurenjie1024

liurenjie1024 commented Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload some generated files to arrow-testing repo for this integration test. How am I supposed to do that? Should I submit another PR in arrow-testing project?

@github-actions

github-actionsBot commented Sep 29, 2019

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-4219

@andygrove

andygrove commented Sep 30, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 If you are adding Parquet files then you can PR against https://github.com/apache/parquet-testing but please note the comments in the last PR merged there which suggests we look into alternate approaches: apache/parquet-testing#9

@liurenjie1024

liurenjie1024 commented Sep 30, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload not only parquet file, but also json files and proto files. Here is my approach to do integration test: I use protobuf to define schema, then generate both json and parquet file with that schema using java code. Then I use rust code to load parquet and json file to compare them. I guess maybe it's better to upload them into arrow-testing library?

@andygrove

andygrove commented Oct 1, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 I think it would be good to start a discussion about this on the mailing list. I'm not sure what advice to give on this.

@alippai

alippai commented Oct 7, 2019

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024

liurenjie1024 commented Oct 8, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove Sorry for late reply. I've submitted a PR in arrow-testing repo: apache/arrow-testing#11
Let's discuss there.

@liurenjie1024

liurenjie1024 commented Oct 13, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove This is ready for review now.

@liurenjie1024liurenjie1024 mentioned this pull request Oct 14, 2019
Closed
nevi-me
nevi-me approved these changes Oct 14, 2019
View reviewed changes

@nevi-menevi-me left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nevi-me

nevi-me commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024 is this an intentional limitation for now? (

arrow/rust/parquet/src/arrow/array_reader.rs

Lines 219 to 222 in 8b915b3

(arrow_type, _) => Err(general_err!(
"Reading {:?} type from parquet is not supported yet.",
arrow_type
)),
)

Also, my apologies for the late review.

@liurenjie1024

liurenjie1024 commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@nevi-me No, it just takes time to implement. Will implement in future.

andygrove
andygrove approved these changes Oct 14, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @liurenjie1024

@asfimportasfimport mentioned this pull request Oct 14, 2019
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andygroveandygroveandygrove approved these changes
@nevi-menevi-menevi-me approved these changes

Assignees

No one assigned

Labels

Component: Rust

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liurenjie1024@andygrove@alippai@nevi-me
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader. - #5523

Closed
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219
Closed

ARROW-4219: [Rust] [Parquet] Initial support for arrow reader.#5523
liurenjie1024 wants to merge 3 commits into
apache:masterfrom
liurenjie1024:arrow-4219

Conversation

@liurenjie1024

@liurenjie1024liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
Contributor

Initial support of arrow reader, which reads parquet into arrow record batch.

@liurenjie1024

liurenjie1024 commented Sep 27, 2019

Copy link
Copy Markdown
ContributorAuthor

@sunchao@andygrove@nevi-me@paddyhoran Please take a look when you are available.

andygrove
andygrove reviewed Sep 27, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM so far but I have a couple of questions in this review

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a very similar trait in DataFusion, only it inherits Sync + Send too so that it can be passed between threads. Maybe we could do that here too? See https://github.com/apache/arrow/blob/master/rust/datafusion/src/execution/physical_plan/mod.rs#L44-L50

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's safe to make this trait as Send+Sync because the reader used some unsafe data(e.g. MutableBuffer).

@andygroveandygroveSep 30, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, makes sense.

Comment threadrust/arrow/src/record_batch.rs Outdated

@andygroveandygroveSep 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is returned if there are no more batches? Shouldn't this return Result<Option<RecordBatch>> ?

@liurenjie1024liurenjie1024Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently if we reached EOF, it returns zero length array. But I think it's better to change return type to Result<Option<RecordBatch>> . Will fix it.

@liurenjie1024

liurenjie1024 commented Sep 29, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload some generated files to arrow-testing repo for this integration test. How am I supposed to do that? Should I submit another PR in arrow-testing project?

@github-actions

github-actionsBot commented Sep 29, 2019

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-4219

@andygrove

andygrove commented Sep 30, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 If you are adding Parquet files then you can PR against https://github.com/apache/parquet-testing but please note the comments in the last PR merged there which suggests we look into alternate approaches: apache/parquet-testing#9

@liurenjie1024

liurenjie1024 commented Sep 30, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove I need to upload not only parquet file, but also json files and proto files. Here is my approach to do integration test: I use protobuf to define schema, then generate both json and parquet file with that schema using java code. Then I use rust code to load parquet and json file to compare them. I guess maybe it's better to upload them into arrow-testing library?

@andygrove

andygrove commented Oct 1, 2019

Copy link
Copy Markdown
Member

@liurenjie1024 I think it would be good to start a discussion about this on the mailing list. I'm not sure what advice to give on this.

@alippai

alippai commented Oct 7, 2019

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024

liurenjie1024 commented Oct 8, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove Sorry for late reply. I've submitted a PR in arrow-testing repo: apache/arrow-testing#11
Let's discuss there.

@liurenjie1024

liurenjie1024 commented Oct 13, 2019

Copy link
Copy Markdown
ContributorAuthor

@andygrove This is ready for review now.

@liurenjie1024liurenjie1024 mentioned this pull request Oct 14, 2019
Closed
nevi-me
nevi-me approved these changes Oct 14, 2019
View reviewed changes

@nevi-menevi-me left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nevi-me

nevi-me commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024 is this an intentional limitation for now? (

arrow/rust/parquet/src/arrow/array_reader.rs

Lines 219 to 222 in 8b915b3

(arrow_type, _) => Err(general_err!(
"Reading {:?} type from parquet is not supported yet.",
arrow_type
)),
)

Also, my apologies for the late review.

@liurenjie1024

liurenjie1024 commented Oct 14, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@nevi-me No, it just takes time to implement. Will implement in future.

andygrove
andygrove approved these changes Oct 14, 2019
View reviewed changes

@andygroveandygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @liurenjie1024

@asfimportasfimport mentioned this pull request Oct 14, 2019
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andygroveandygroveandygrove approved these changes
@nevi-menevi-menevi-me approved these changes

Assignees

No one assigned

Labels

Component: Rust

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liurenjie1024@andygrove@alippai@nevi-me