[Feature](partial update) Support flexible partial update in stream load with json files

[bobhan1]

This PR add the ability to update different columns for each row in one stream load
Doc: apache/doris-website#1140

Example

MySQL root@127.1:d1> CREATE TABLE t1 (
->`k`int(11) NULL, ->`v1`BIGINTNULL,
->`v2`BIGINTNULL DEFAULT "9876",
->`v3`BIGINTNOT NULL,
->`v4`BIGINTNOT NULL DEFAULT "1234",
->`v5`BIGINTNULL-> ) UNIQUE KEY(`k`) DISTRIBUTED BY HASH(`k`) BUCKETS 1-> PROPERTIES(
->"replication_num"="1",
->"enable_unique_key_merge_on_write"="true"); Query OK, 0 rows affected
Time: 0.013s
MySQL root@127.1:d1>insert into t1 selectnumber, number, number, number, number, numberfrom numbers("number"="6");
Query OK, 6 rows affected
Time: 0.107s
MySQL root@127.1:d1>select*from t1;
+---+----+----+----+----+----+
| k | v1 | v2 | v3 | v4 | v5 |
+---+----+----+----+----+----+
| 0 | 0 | 0 | 0 | 0 | 0 |
| 1 | 1 | 1 | 1 | 1 | 1 |
| 2 | 2 | 2 | 2 | 2 | 2 |
| 3 | 3 | 3 | 3 | 3 | 3 |
| 4 | 4 | 4 | 4 | 4 | 4 |
| 5 | 5 | 5 | 5 | 5 | 5 |
+---+----+----+----+----+----+

test1.json:

{"k": 1, "v1": 10}
{"k": 2, "v2": 20, "v5": 25}
{"k": 3, "v3": 30}
{"k": 4, "v4": 20, "v1": 43, "v3": 99}
{"k": 5, "v5": null}
{"k": 6, "v1": 999, "v3": 777}
{"k": 2, "v4": 222}
{"k": 1, "v2": 111, "v3": 111}

curl --location-trusted -u root: \
-H "strict_mode:false" \
-H "format:json" \
-H "read_json_by_line:true" \
-H "unique_key_update_mode:UPDATE_FLEXIBLE_COLUMNS" \
-T test1.json \
-XPUT http://<host>:<http_port>/api/d1/t1/_stream_load

MySQL root@127.1:d1>select*from t1;
+---+-----+------+-----+------+--------+
| k | v1 | v2 | v3 | v4 | v5 |
+---+-----+------+-----+------+--------+
| 0 | 0 | 0 | 0 | 0 | 0 |
| 1 | 10 | 111 | 111 | 1 | 1 |
| 2 | 2 | 20 | 2 | 222 | 25 |
| 3 | 3 | 3 | 30 | 3 | 3 |
| 4 | 43 | 4 | 99 | 20 | 4 |
| 5 | 5 | 5 | 5 | 5 | <null> |
| 6 | 999 | 9876 | 777 | 1234 | <null> |
+---+-----+------+-----+------+--------+

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[bobhan1]

run buildall

[github-actions]

clang-tidy made some suggestions

[github-actions]

warning: function '_merge_rows_for_sequence_column' exceeds recommended size/complexity thresholds [readability-function-size]

Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
^

Additional context

be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:830: 91 lines including whitespace and comments (threshold 80)

Status VerticalSegmentWriter::_merge_rows_for_sequence_column(
^

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 37.23% (9638/25888)
Line Coverage: 28.54% (79918/280040)
Region Coverage: 28.01% (41327/147559)
Branch Coverage: 24.62% (21055/85508)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2569de09e81dd504a3267b5e991c552ca678fbb0_2569de09e81dd504a3267b5e991c552ca678fbb0/report/index.html

[bobhan1]

run cloud_p0

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[github-actions]

clang-tidy made some suggestions

[bobhan1]

run buildall

[bobhan1]

run buildall

[github-actions]

clang-tidy made some suggestions

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 37.25% (9645/25895)
Line Coverage: 28.56% (79987/280104)
Region Coverage: 27.99% (41350/147725)
Branch Coverage: 24.60% (21047/85574)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e5b703dbd9f6361ac4bc3456066488cc78091f2f_e5b703dbd9f6361ac4bc3456066488cc78091f2f/report/index.html

[dataroaring]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.

[zhannngchen]

LGTM

[Carl-Zhou-CN]

@dataroaring@zhannngchen hi, is there any plan to release this PR? What risks are involved

[bobhan1]

@Carl-Zhou-CN the feature will probably be in doris 4.0

[Carl-Zhou-CN]

@Carl-Zhou-CN the feature will probably be in doris 4.0

Thank you very much for your response. I currently need this functionality urgently. If I merge and use it myself, what issues should I be aware of?

[bobhan1]

@Carl-Zhou-CN the feature will probably be in doris 4.0

Thank you very much for your response. I currently need this functionality urgently. If I merge and use it myself, what issues should I be aware of?

This PR has problems and some them are fixed in #41701.
This feature may require high iops and io throughput.

[Carl-Zhou-CN]

@Carl-Zhou-CN the feature will probably be in doris 4.0

Thank you very much for your response. I currently need this functionality urgently. If I merge and use it myself, what issues should I be aware of?

This PR has problems and some them are fixed in #41701. This feature may require high iops and io throughput.

Ok. Thank you.