[SPARK-36980][SQL] Insert support query with CTE by AngersZhuuuu · Pull Request #34252 · apache/spark

AngersZhuuuu · 2021-10-12T04:26:14Z

What changes were proposed in this pull request?

For current SQL grammar

INSERT INTO t_delta (WITH v1(c1) as (values (1)) select 1, 2,3 from v1); OK
INSERT INTO t_delta WITH v1(c1) as (values (1)) select 1, 2,3 from v1; FAILED

It's caused by

insertInto query queryOrganization

Here change to

insert query

can support CTE after insert

Why are the changes needed?

Support CTE after insert query

Does this PR introduce any user-facing change?

How was this patch tested?

Added UT

AngersZhuuuu · 2021-10-12T04:26:33Z

ping @cloud-fan

SparkQA · 2021-10-12T06:03:31Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48599/

SparkQA · 2021-10-12T07:03:16Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48599/

cloud-fan · 2021-10-12T08:18:38Z

can you also update the sql reference doc?

AngersZhuuuu · 2021-10-12T09:02:07Z

can you also update the sql reference doc?

How about current?

SparkQA · 2021-10-12T09:07:57Z

Test build #144122 has finished for PR 34252 at commit 8a1b4e4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-10-12T10:50:26Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48617/

SparkQA · 2021-10-12T11:47:49Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48617/

SparkQA · 2021-10-12T14:11:21Z

Test build #144140 has finished for PR 34252 at commit 39307ba.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2021-10-14T14:58:13Z

ping @cloud-fan

SparkQA · 2021-10-14T16:09:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48744/

SparkQA · 2021-10-14T16:53:16Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48744/

SparkQA · 2021-10-14T20:08:55Z

Test build #144265 has finished for PR 34252 at commit 846f6c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class HistogramPlotBase(NumericPlotBase):
class KdePlotBase(NumericPlotBase):
new_class = type(NameTypeHolder.short_name, (NameTypeHolder,),
class SparkUpgradeException(CapturedException):
public class ExpressionImplUtils
class NoSuchIndexException(message: String, cause: Option[Throwable] = None)
case class AesEncrypt(input: Expression, key: Expression, child: Expression)
case class AesDecrypt(input: Expression, key: Expression, child: Expression)
case class CreateView(
case class ShowCatalogsCommand(pattern: Option[String]) extends LeafRunnableCommand

cloud-fan · 2021-10-15T06:31:10Z

thanks, merging to master!

### What changes were proposed in this pull request? This pr upgrade Apache Arrow from 13.0.0 to 14.0.0. ### Why are the changes needed? The Apache Arrow 14.0.0 release brings a number of enhancements and bug fixes. ‎ In terms of bug fixes, the release addresses several critical issues that were causing failures in integration jobs with Spark([GH-36332](apache/arrow#36332)) and problems with importing empty data arrays([GH-37056](apache/arrow#37056)). It also optimizes the process of appending variable length vectors([GH-37829](apache/arrow#37829)) and includes C++ libraries for MacOS AARCH 64 in Java-Jars([GH-38076](apache/arrow#38076)). ‎ The new features and improvements focus on enhancing the handling and manipulation of data. This includes the introduction of DefaultVectorComparators for large types([GH-25659](apache/arrow#25659)), support for extended expressions in ScannerBuilder([GH-34252](apache/arrow#34252)), and the exposure of the VectorAppender class([GH-37246](apache/arrow#37246)). ‎ The release also brings enhancements to the development and testing process, with the CI environment now using JDK 21([GH-36994](apache/arrow#36994)). In addition, the release introduces vector validation consistent with C++, ensuring consistency across different languages([GH-37702](apache/arrow#37702)). ‎ Furthermore, the usability of VarChar writers and binary writers has been improved with the addition of extra input methods([GH-37705](apache/arrow#37705)), and VarCharWriter now supports writing from `Text` and `String`([GH-37706](apache/arrow#37706)). The release also adds typed getters for StructVector, improving the ease of accessing data([GH-37863](apache/arrow#37863)). The full release notes as follows: - https://arrow.apache.org/release/14.0.0.html ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes#43650 from LuciferYang/arrow-14. Lead-authored-by: yangjie01 <yangjie01@baidu.com> Co-authored-by: YangJie <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

[SPARK-36980][SQL] Insert support query with CTE
8a1b4e4

github-actionsBot added the SQL label Oct 12, 2021

cloud-fan reviewed Oct 12, 2021
View reviewed changes

Comment threadsql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala Outdated

AngersZhuuuu added 2 commits October 12, 2021 17:00

update
82f40ba

Update InsertSuite.scala
39307ba

github-actionsBot added the DOCS label Oct 12, 2021

Merge branch 'master' into SPARK-36980
846f6c8

cloud-fan approved these changes Oct 15, 2021
View reviewed changes

cloud-fan closed this in 9865c09Oct 15, 2021

Uh oh!

Conversation

AngersZhuuuu commented Oct 12, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

AngersZhuuuu commented Oct 12, 2021

Uh oh!

SparkQA commented Oct 12, 2021

Uh oh!

SparkQA commented Oct 12, 2021

Uh oh!

cloud-fan commented Oct 12, 2021

Uh oh!

Uh oh!

AngersZhuuuu commented Oct 12, 2021

Uh oh!

SparkQA commented Oct 12, 2021

Uh oh!

SparkQA commented Oct 12, 2021

Uh oh!

SparkQA commented Oct 12, 2021

Uh oh!

SparkQA commented Oct 12, 2021

Uh oh!

AngersZhuuuu commented Oct 14, 2021

Uh oh!

SparkQA commented Oct 14, 2021

Uh oh!

SparkQA commented Oct 14, 2021

Uh oh!

SparkQA commented Oct 14, 2021

Uh oh!

cloud-fan commented Oct 15, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants