Support mvdedup eval function

[ahkcs]

---

Description

This PR implements the mvdedup eval function for PPL, enabling users to remove duplicate values from multivalue arrays.

---

Behavior

Given the input:

source=index | eval result = mvdedup(array(1, 2, 2, 3, 1, 4))

The function returns:

[1, 2, 3, 4]

---

Key Details

• Empty arrays return empty arrays

• Order-preserving: The first appearance of each value is kept; subsequent duplicates are removed.

---

Example

Input	Output
array(1, 2, 2, 3, 1, 4)	[1, 2, 3, 4]
array()	[]

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dai-chen]

Could you check if this has same semantic as ARRAY_DISTINCT?

2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [1, 2, 3, 4] |
+--------------+
1 row selected (0.018 seconds)
2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[4,1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [4, 1, 2, 3] |
+--------------+

[ahkcs]

Could you check if this has same semantic as ARRAY_DISTINCT?

Hi @dai-chen, thanks for the suggestion! After checking, I think ARRAY_DISTINCT is suitable for our implementation for mvdedup eval function, I have updated the implementation to remove the UDF and use ARRAY_DISTINCT for implementation.

cc @ykmr1224

[dai-chen]

Thanks for the changes!

[dai-chen]

Do we need to backport?

[ahkcs]

Do we need to backport?

Yes

[ahkcs]

---

Description

This PR implements the mvdedup eval function for PPL, enabling users to remove duplicate values from multivalue arrays.

---

Behavior

Given the input:

source=index | eval result = mvdedup(array(1, 2, 2, 3, 1, 4))

The function returns:

[1, 2, 3, 4]

---

Key Details

• Empty arrays return empty arrays

• Order-preserving: The first appearance of each value is kept; subsequent duplicates are removed.

---

Example

Input	Output
array(1, 2, 2, 3, 1, 4)	[1, 2, 3, 4]
array()	[]

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dai-chen]

Could you check if this has same semantic as ARRAY_DISTINCT?

2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [1, 2, 3, 4] |
+--------------+
1 row selected (0.018 seconds)
2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[4,1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [4, 1, 2, 3] |
+--------------+

[ahkcs]

Could you check if this has same semantic as ARRAY_DISTINCT?

Hi @dai-chen, thanks for the suggestion! After checking, I think ARRAY_DISTINCT is suitable for our implementation for mvdedup eval function, I have updated the implementation to remove the UDF and use ARRAY_DISTINCT for implementation.

cc @ykmr1224

[dai-chen]

Thanks for the changes!

[dai-chen]

Do we need to backport?

[ahkcs]

Do we need to backport?

Yes

[ahkcs]

---

Description

This PR implements the mvdedup eval function for PPL, enabling users to remove duplicate values from multivalue arrays.

---

Behavior

Given the input:

source=index | eval result = mvdedup(array(1, 2, 2, 3, 1, 4))

The function returns:

[1, 2, 3, 4]

---

Key Details

• Empty arrays return empty arrays

• Order-preserving: The first appearance of each value is kept; subsequent duplicates are removed.

---

Example

Input	Output
array(1, 2, 2, 3, 1, 4)	[1, 2, 3, 4]
array()	[]

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dai-chen]

Could you check if this has same semantic as ARRAY_DISTINCT?

2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [1, 2, 3, 4] |
+--------------+
1 row selected (0.018 seconds)
2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[4,1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [4, 1, 2, 3] |
+--------------+

[ahkcs]

Could you check if this has same semantic as ARRAY_DISTINCT?

Hi @dai-chen, thanks for the suggestion! After checking, I think ARRAY_DISTINCT is suitable for our implementation for mvdedup eval function, I have updated the implementation to remove the UDF and use ARRAY_DISTINCT for implementation.

cc @ykmr1224

[dai-chen]

Thanks for the changes!

[dai-chen]

Do we need to backport?

[ahkcs]

Do we need to backport?

Yes

[ahkcs]

---

Description

This PR implements the mvdedup eval function for PPL, enabling users to remove duplicate values from multivalue arrays.

---

Behavior

Given the input:

source=index | eval result = mvdedup(array(1, 2, 2, 3, 1, 4))

The function returns:

[1, 2, 3, 4]

---

Key Details

• Empty arrays return empty arrays

• Order-preserving: The first appearance of each value is kept; subsequent duplicates are removed.

---

Example

Input	Output
array(1, 2, 2, 3, 1, 4)	[1, 2, 3, 4]
array()	[]

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dai-chen]

Could you check if this has same semantic as ARRAY_DISTINCT?

2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [1, 2, 3, 4] |
+--------------+
1 row selected (0.018 seconds)
2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[4,1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [4, 1, 2, 3] |
+--------------+

[ahkcs]

Could you check if this has same semantic as ARRAY_DISTINCT?

Hi @dai-chen, thanks for the suggestion! After checking, I think ARRAY_DISTINCT is suitable for our implementation for mvdedup eval function, I have updated the implementation to remove the UDF and use ARRAY_DISTINCT for implementation.

cc @ykmr1224

[dai-chen]

Thanks for the changes!

[dai-chen]

Do we need to backport?

[ahkcs]

Do we need to backport?

Yes

[ahkcs]

---

Description

This PR implements the mvdedup eval function for PPL, enabling users to remove duplicate values from multivalue arrays.

---

Behavior

Given the input:

source=index | eval result = mvdedup(array(1, 2, 2, 3, 1, 4))

The function returns:

[1, 2, 3, 4]

---

Key Details

• Empty arrays return empty arrays

• Order-preserving: The first appearance of each value is kept; subsequent duplicates are removed.

---

Example

Input	Output
array(1, 2, 2, 3, 1, 4)	[1, 2, 3, 4]
array()	[]

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dai-chen]

Could you check if this has same semantic as ARRAY_DISTINCT?

2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [1, 2, 3, 4] |
+--------------+
1 row selected (0.018 seconds)
2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[4,1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [4, 1, 2, 3] |
+--------------+

[ahkcs]

Could you check if this has same semantic as ARRAY_DISTINCT?

Hi @dai-chen, thanks for the suggestion! After checking, I think ARRAY_DISTINCT is suitable for our implementation for mvdedup eval function, I have updated the implementation to remove the UDF and use ARRAY_DISTINCT for implementation.

cc @ykmr1224

[dai-chen]

Thanks for the changes!

[dai-chen]

Do we need to backport?

[ahkcs]

Do we need to backport?

Yes

[ahkcs]

---

Description

This PR implements the mvdedup eval function for PPL, enabling users to remove duplicate values from multivalue arrays.

---

Behavior

Given the input:

source=index | eval result = mvdedup(array(1, 2, 2, 3, 1, 4))

The function returns:

[1, 2, 3, 4]

---

Key Details

• Empty arrays return empty arrays

• Order-preserving: The first appearance of each value is kept; subsequent duplicates are removed.

---

Example

Input	Output
array(1, 2, 2, 3, 1, 4)	[1, 2, 3, 4]
array()	[]

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dai-chen]

Could you check if this has same semantic as ARRAY_DISTINCT?

2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [1, 2, 3, 4] |
+--------------+
1 row selected (0.018 seconds)
2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[4,1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [4, 1, 2, 3] |
+--------------+

[ahkcs]

Could you check if this has same semantic as ARRAY_DISTINCT?

Hi @dai-chen, thanks for the suggestion! After checking, I think ARRAY_DISTINCT is suitable for our implementation for mvdedup eval function, I have updated the implementation to remove the UDF and use ARRAY_DISTINCT for implementation.

cc @ykmr1224

[dai-chen]

Thanks for the changes!

[dai-chen]

Do we need to backport?

[ahkcs]

Do we need to backport?

Yes

[ahkcs]

---

Description

This PR implements the mvdedup eval function for PPL, enabling users to remove duplicate values from multivalue arrays.

---

Behavior

Given the input:

source=index | eval result = mvdedup(array(1, 2, 2, 3, 1, 4))

The function returns:

[1, 2, 3, 4]

---

Key Details

• Empty arrays return empty arrays

• Order-preserving: The first appearance of each value is kept; subsequent duplicates are removed.

---

Example

Input	Output
array(1, 2, 2, 3, 1, 4)	[1, 2, 3, 4]
array()	[]

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dai-chen]

Could you check if this has same semantic as ARRAY_DISTINCT?

2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [1, 2, 3, 4] |
+--------------+
1 row selected (0.018 seconds)
2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[4,1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [4, 1, 2, 3] |
+--------------+

[ahkcs]

Could you check if this has same semantic as ARRAY_DISTINCT?

Hi @dai-chen, thanks for the suggestion! After checking, I think ARRAY_DISTINCT is suitable for our implementation for mvdedup eval function, I have updated the implementation to remove the UDF and use ARRAY_DISTINCT for implementation.

cc @ykmr1224

[dai-chen]

Thanks for the changes!

[dai-chen]

Do we need to backport?

[ahkcs]

Do we need to backport?

Yes

[ahkcs]

---

Description

This PR implements the mvdedup eval function for PPL, enabling users to remove duplicate values from multivalue arrays.

---

Behavior

Given the input:

source=index | eval result = mvdedup(array(1, 2, 2, 3, 1, 4))

The function returns:

[1, 2, 3, 4]

---

Key Details

• Empty arrays return empty arrays

• Order-preserving: The first appearance of each value is kept; subsequent duplicates are removed.

---

Example

Input	Output
array(1, 2, 2, 3, 1, 4)	[1, 2, 3, 4]
array()	[]

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dai-chen]

Could you check if this has same semantic as ARRAY_DISTINCT?

2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [1, 2, 3, 4] |
+--------------+
1 row selected (0.018 seconds)
2: jdbc:calcite:model=src/test/resources/mode> SELECT ARRAY_DISTINCT(ARRAY[4,1,2,1,3,3,4]);
+--------------+
| EXPR$0 |
+--------------+
| [4, 1, 2, 3] |
+--------------+

[ahkcs]

Could you check if this has same semantic as ARRAY_DISTINCT?

Hi @dai-chen, thanks for the suggestion! After checking, I think ARRAY_DISTINCT is suitable for our implementation for mvdedup eval function, I have updated the implementation to remove the UDF and use ARRAY_DISTINCT for implementation.

cc @ykmr1224

[dai-chen]

Thanks for the changes!

[dai-chen]

Do we need to backport?

[ahkcs]

Do we need to backport?

Yes