Skip to content

[Spark load][Bug] Spark load's column terminator is not consistent with Broker load #4490

Description

@xy720

Describe the bug
For Broker load:

load label test_db.label13 ( DATA INFILE("hdfs://my-hdfs-host:54310/user/doris/table4") into table test_tbl
COLUMNS TERMINATED BY "|" (k1,k2,name,clicks)
) WITH BROKER "doris" (
"username" = "test", "password" = "test"
);

For spark load:

load label test_db.label13 ( DATA INFILE ("hdfs://my-hdfs-host:54310/user/doris/table4") into table test COLUMNS TERMINATED BY "|" (k1,k2,name,clicks ) ) WITH RESOURCE "spark0";

They have the same table schema、same upstream data source. The former(broker load) run successfully, but the latter(spark load) report an error "quality not good enough to cancel". And if you change the column terminator of spark load to "\|" or "\\|", then it will run successfully.

Expected behavior
We should keep the column terminator's format of spark load consistent with that of broker load.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions