Uh oh!
There was an error while loading. Please reload this page.
Support fixed_size_list for make_array - #6759
Conversation
There was a problem hiding this comment.
I create fixedsizelist parquet from my own script.
Is there any existing way for us to easily create/update this kind of test case?
There was a problem hiding this comment.
Did you look at the data sets available in https://github.com/apache/parquet-testing (which is a submodule of this repo)
e.g. https://github.com/apache/parquet-testing/tree/a11fc8f148f8a7a89d9281cc0da3eb9d56095fbf/data
There was a problem hiding this comment.
Thank you for this contribution @jayzhan211 -- I think other than the patch to crates.io this PR is ready to go.
I left some suggestions, which I think would improve the PR
| parquet = { version = "42.0.0", features = ["arrow", "async", "object_store"] } | ||
| sqlparser = { version = "0.35", features = ["visitor"] } | ||
| [patch.crates-io] |
There was a problem hiding this comment.
I think we will have to wait for this code to be released to crates.io (in about 10 days) before merging this PR
| ScalarValue::TimestampNanosecond(e, _) => format_option!(f, e)?, | ||
| ScalarValue::Utf8(e) => format_option!(f, e)?, | ||
| ScalarValue::LargeUtf8(e) => format_option!(f, e)?, | ||
| ScalarValue::Binary(e) => match e { |
There was a problem hiding this comment.
Did you look at the data sets available in https://github.com/apache/parquet-testing (which is a submodule of this repo)
e.g. https://github.com/apache/parquet-testing/tree/a11fc8f148f8a7a89d9281cc0da3eb9d56095fbf/data
Uh oh!
There was an error while loading. Please reload this page.
| "Date32" => Token::SimpleType(DataType::Date32), | ||
| "Date64" => Token::SimpleType(DataType::Date64), | ||
| "List" => Token::List, |
alamb
commented
Jun 26, 2023
FYI @izveigor |
alamb
commented
Jun 28, 2023
Marking as a draft as it looks like this PR is not waiting on review -- it is waiting on an upstream release (scheduled early next week) as well as to address some other comments |
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* feat: supports NULL in arrays * feat: supports NULL in array functions * fix: array_fill error * fix: merge * fix: cargo fmt --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
ecc93dd to
c6ba92aCompare| let list_data_type = | ||
| DataType::List(Arc::new(Field::new("item", data_type, false))); | ||
| DataType::List(Arc::new(Field::new("item", data_type, true))); |
There was a problem hiding this comment.
Revert the change from #6662 to avoid failure on the schema mismatch
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
alamb
commented
Jul 5, 2023
cc @izveigor |
* support make_array for fixed_size_list Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * add arrow-typeof in test Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix schema mismatch Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * cleanup code Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * create array data with correct len Signed-off-by: jayzhan211 <jayzhan211@gmail.com> --------- Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* support make_array for fixed_size_list Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * add arrow-typeof in test Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix schema mismatch Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * cleanup code Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * create array data with correct len Signed-off-by: jayzhan211 <jayzhan211@gmail.com> --------- Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
Which issue does this PR close?
Ref #6560
Rationale for this change
Add FixedSizeList support for array methods
What changes are included in this PR?
Only MakeArray is done for this PR.
Are these changes tested?
sqllogictest
array.slt
unit test in type_coercion.rs
Are there any user-facing changes?
Task