[SPARK-43438][SQL] Error on missing input columns in `INSERT` by MaxGekk · Pull Request #42393 · apache/spark

MaxGekk · 2023-08-08T11:21:28Z

What changes were proposed in this pull request?

In the PR, I propose to raise an error when an user uses V1 INSERT without a list of columns, and the number of inserting columns doesn't match to the number of actual table columns.

At the moment Spark inserts data successfully in such case after the PR #41262 which changed the behaviour of Spark 3.4.x.

Why are the changes needed?

To conform the SQL standard which requires the number of columns must be the same:

Apparently, the insertion below must not succeed:

spark-sql (default)> CREATE TABLE tabtest(c1 INT, c2 INT);
spark-sql (default)>INSERT INTO tabtest SELECT1;

To have the same behaviour as Spark 3.4:

spark-sql (default)>INSERT INTO tabtest SELECT1;
`spark_catalog`.`default`.`tabtest` requires that the data to be inserted have the same number of columns as the target table: target table has 2 column(s) but the inserted data has 1 column(s), including 0 partition column(s) having constant value(s).

Does this PR introduce any user-facing change?

Yes.

After the changes:

spark-sql (default)>INSERT INTO tabtest SELECT1;
[INSERT_COLUMN_ARITY_MISMATCH.NOT_ENOUGH_DATA_COLUMNS] Cannot write to `spark_catalog`.`default`.`tabtest`, the reason is not enough data columns:
Table columns: `c1`, `c2`.
Data columns: `1`.

How was this patch tested?

By running the modified tests:

$ build/sbt "test:testOnly *InsertSuite"
$ build/sbt "test:testOnly *ResolveDefaultColumnsSuite"
$ build/sbt -Phive "test:testOnly *HiveQuerySuite"

cloud-fan · 2023-08-29T08:13:42Z

 }
 withTable("t") {
 sql("create table t(i int, s bigint default 42, x bigint) using parquet")
- sql("insert into t values(1)")


which Spark version added this test?

Added by #41262 which haven't been released yet.

dtenedor

LGTM, there's enough test coverage for INSERT commands with and without user-specified column lists, and with correct and incorrect numbers of attributes from the source table.

MaxGekk · 2023-08-29T20:04:04Z

Merging to master/3.5. Thank you, @cloud-fan and @dtenedor for review.

### What changes were proposed in this pull request? In the PR, I propose to raise an error when an user uses V1 `INSERT` without a list of columns, and the number of inserting columns doesn't match to the number of actual table columns. At the moment Spark inserts data successfully in such case after the PR #41262 which changed the behaviour of Spark 3.4.x. ### Why are the changes needed? 1. To conform the SQL standard which requires the number of columns must be the same: ![Screenshot 2023-08-07 at 11 01 27 AM](https://github.com/apache/spark/assets/1580697/c55badec-5716-490f-a83a-0bb6b22c84c7) Apparently, the insertion below must not succeed: ```sql spark-sql (default)> CREATE TABLE tabtest(c1 INT, c2 INT); spark-sql (default)> INSERT INTO tabtest SELECT 1; ``` 2. To have the same behaviour as **Spark 3.4**: ```sql spark-sql (default)> INSERT INTO tabtest SELECT 1; `spark_catalog`.`default`.`tabtest` requires that the data to be inserted have the same number of columns as the target table: target table has 2 column(s) but the inserted data has 1 column(s), including 0 partition column(s) having constant value(s). ``` ### Does this PR introduce _any_ user-facing change? Yes. After the changes: ```sql spark-sql (default)> INSERT INTO tabtest SELECT 1; [INSERT_COLUMN_ARITY_MISMATCH.NOT_ENOUGH_DATA_COLUMNS] Cannot write to `spark_catalog`.`default`.`tabtest`, the reason is not enough data columns: Table columns: `c1`, `c2`. Data columns: `1`. ``` ### How was this patch tested? By running the modified tests: ``` $ build/sbt "test:testOnly *InsertSuite" $ build/sbt "test:testOnly *ResolveDefaultColumnsSuite" $ build/sbt -Phive "test:testOnly *HiveQuerySuite" ``` Closes#42393 from MaxGekk/fix-num-cols-insert. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Max Gekk <max.gekk@gmail.com> (cherry picked from commit a7eef21) Signed-off-by: Max Gekk <max.gekk@gmail.com>

Error on missing input columns in INSERT
8b07168

github-actionsBot added the SQL label Aug 8, 2023

MaxGekk mentioned this pull request Aug 8, 2023
[SPARK-44577][SQL] Fix INSERT BY NAME returns nonsensical error message #42220
Closed

MaxGekk added 11 commits August 9, 2023 11:37

Add comments and fix tests
c736952

Fix InsertSuite
cd9d3d0

Fix HiveQuerySuite
e565071

Checks column match in TableOutputResolver.resolveOutputColumns
049dbd2

Merge remote-tracking branch 'origin/master' into fix-num-cols-insert
ff0d5d9

Fix DSV2SQLInsertTestSuite.insert by name: mismatch column name
c9b1fb3

Fix DataFrameWriterV2Suite
722685d

Fix V2AppendDataANSIAnalysisSuite
51d3fbf

Remove a test
01f0117

Merge remote-tracking branch 'origin/master' into fix-num-cols-insert
6091fff

Trigger build
96cab93

cloud-fan reviewed Aug 25, 2023
View reviewed changes

Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala Outdated

cloud-fan reviewed Aug 25, 2023
View reviewed changes

Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala

MaxGekk added 4 commits August 25, 2023 16:37

Trigger build
9c3188c

Trigger build
e00fdbe

Move the check
87f7e96

Address review comments
60f9327

MaxGekk changed the title ~~[WIP][SPARK-43438][SQL] Error on missing input columns in INSERT~~[SPARK-43438][SQL] Error on missing input columns in INSERTAug 29, 2023

MaxGekk marked this pull request as ready for review August 29, 2023 07:28