Skip to content

feat: support stride in array_slice, change indexes to be1 based - #8829

Merged
alamb merged 10 commits into
apache:mainfrom
Weijun-H:feat-array-slice-stride
Jan 21, 2024
Merged

feat: support stride in array_slice, change indexes to be1 based#8829
alamb merged 10 commits into
apache:mainfrom
Weijun-H:feat-array-slice-stride

Conversation

@Weijun-H

@Weijun-HWeijun-H commented Jan 11, 2024

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes#8784

Rationale for this change

What changes are included in this PR?

support array_slice with stride argument

array_slice(make_array(1, 2, 3, 4, 5), 1, 5, 2) --> [1, 3, 5]

Are these changes tested?

Are there any user-facing changes?

@github-actionsgithub-actionsBot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) labels Jan 11, 2024
@Weijun-HWeijun-H changed the title support 'stride' in 'array_slice'support stride in array_sliceJan 11, 2024
Comment threaddatafusion/physical-expr/src/array_expressions.rs Outdated

query error Execution error: array_slice got invalid stride: 0, it cannot be 0
select array_slice(make_array(1, 2, 3, 4, 5), 1, 5, 0), array_slice(make_array('h', 'e', 'l', 'l', 'o'), 1, 5, 0);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SELECT([1, 2, 3, 4, 5])[4:2:-2]; are also needed. And other examples in duckdb, we should consider them too.

https://duckdb.org/docs/sql/functions/nested#slicing

@Weijun-HWeijun-HJan 12, 2024

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think so. And we could do it in ticket #8830 .

@Weijun-H
Weijun-H marked this pull request as draft January 11, 2024 13:52
@Weijun-H
Weijun-Hforce-pushed the feat-array-slice-stride branch from 6b56383 to d41c3a2CompareJanuary 12, 2024 06:07
@Weijun-H
Weijun-H marked this pull request as ready for review January 12, 2024 06:07
let end = offset_window[1];
let len = end - start;

let stride = if let Some(stride) = stride {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this construction most likely can be replaced by .map/.and_then ?

(start + to + O::usize_as(1)).to_usize().unwrap(),
);
offsets.push(offsets[row_index] + (to - from + O::usize_as(1)));
if let Some(stride) = stride {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this check may be earlier?

@compheadcomphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Weijun-H, just couple of minors, and we need to modify the user documentation?

Comment threaddatafusion/expr/src/expr_fn.rs Outdated
ArraySlice,
array_slice,
array offset length,
array offset length stride,

@jayzhan211jayzhan211Jan 13, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems offset length is incorrect, it should be begin end

@jayzhan211jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Weijun-HWeijun-H changed the title support stride in array_slicefeat: support stride in array_sliceJan 13, 2024
@Weijun-H
Weijun-Hforce-pushed the feat-array-slice-stride branch from d11a0cc to 977f2e1CompareJanuary 13, 2024 06:02
@Weijun-H

Copy link
Copy Markdown
MemberAuthor

stalled until #8847 fixed

};

if let (Some(from), Some(to)) = (from_index, to_index) {
let stride = stride.map(|s| s.value(row_index));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@compheadcomphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Weijun-H for the fixes, its much nicer now. Waiting for the CI

@alamb
alamb marked this pull request as draft January 15, 2024 10:49
@alamb

Copy link
Copy Markdown
Contributor

stalled until #8847 fixed

Marking as draft to make it clear this PR is not waiting on review

@Weijun-H
Weijun-Hforce-pushed the feat-array-slice-stride branch from 977f2e1 to 297ef22CompareJanuary 15, 2024 11:52
@Weijun-H
Weijun-H marked this pull request as ready for review January 15, 2024 13:17
@alambalamb changed the title feat: support stride in array_slicefeat: support stride in array_slice, change indexes to be 1 basedJan 19, 2024
@alambalamb changed the title feat: support stride in array_slice, change indexes to be 1 basedfeat: support stride in array_slice, change indexes to be0 basedJan 19, 2024

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me @Weijun-H -- thank you

Can you confirm the behavior change to be 0 based indexes rather than 1 based indexes is intentional? I updated the PR title but I wanted to double check

@Weijun-H

Copy link
Copy Markdown
MemberAuthor

Looks good to me @Weijun-H -- thank you

Can you confirm the behavior change to be 0 based indexes rather than 1 based indexes is intentional? I updated the PR title but I wanted to double check

array_slice is 1-index, not 0-index, which aligns with PostgreSQL and DuckDB. @alamb

@alambalamb changed the title feat: support stride in array_slice, change indexes to be0 basedfeat: support stride in array_slice, change indexes to be1 basedJan 21, 2024
@alamb

Copy link
Copy Markdown
Contributor

Looks good to me @Weijun-H -- thank you
Can you confirm the behavior change to be 0 based indexes rather than 1 based indexes is intentional? I updated the PR title but I wanted to double check

array_slice is 1-index, not 0-index, which aligns with PostgreSQL and DuckDB. @alamb

Fixed

@alamb
alamb merged commit 0116e2a into apache:mainJan 21, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-exprLogical plan and expressionsphysical-exprChanges to the physical-expr cratessqllogictestSQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support stride for array_slice

4 participants

@Weijun-H@alamb@comphead@jayzhan211