Uh oh!
There was an error while loading. Please reload this page.
[enhancement](fix)change ordinary type null value is \N,complex type null value is null - #24207
Conversation
hubgeter
commented
Sep 11, 2023
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
doris-robot
commented
Sep 11, 2023
(From new machine)TeamCity pipeline, clickbench performance test result: |
90d64b9 to
9039fffComparehubgeter
commented
Sep 12, 2023
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
doris-robot
commented
Sep 12, 2023
(From new machine)TeamCity pipeline, clickbench performance test result: |
doris-robot
commented
Sep 12, 2023
TeamCity be ut coverage result: |
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
| //for map<string,string> type : {"abc","NULL"} , the NULL is string , instead of null values | ||
| if (slice.size == 4 && slice[0] == 'N' && slice[1] == 'U' && slice[2] == 'L' && | ||
| slice[3] == 'L') { | ||
| if (nesting_level >= 2 && slice.size == 4 && slice[0] == 'n' && slice[1] == 'u' && |
There was a problem hiding this comment.
Add comment to describe the logic here. Better give some example
9039fff to
0a5ba7eCompareclang-tidy review says "All clean, LGTM! 👍" |
hubgeter
commented
Sep 14, 2023
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
doris-robot
commented
Sep 14, 2023
TeamCity be ut coverage result: |
doris-robot
commented
Sep 14, 2023
(From new machine)TeamCity pipeline, clickbench performance test result: |
| * for null values in nested types, we use null to represent them, just like the json format. | ||
| * | ||
| * example: | ||
| * If you have three nullable columns |
There was a problem hiding this comment.
make a note: null -> int -> NULL | null -> char family -> "null"
in csv(text) for normal type: we only recognize \N for null
so
for not char family type, like int, if we put null literal , it will parse fail, and make result null,not just because it equals \N
for char family type, like string, if we put null literal, it will parse success, and "null" literal will be stored in doris
hubgeter
commented
Sep 15, 2023
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
doris-robot
commented
Sep 15, 2023
TeamCity be ut coverage result: |
doris-robot
commented
Sep 15, 2023
(From new machine)TeamCity pipeline, clickbench performance test result: |
hubgeter
commented
Sep 16, 2023
run buildall |
clang-tidy review says "All clean, LGTM! 👍" |
doris-robot
commented
Sep 16, 2023
TeamCity be ut coverage result: |
doris-robot
commented
Sep 16, 2023
(From new machine)TeamCity pipeline, clickbench performance test result: |
PR approved by at least one committer and no changes requested. |
Proposed changes
For null values in ordinary types, we use \N to represent them;
for null values in nested types, we use null to represent them, just like the json format.
example:
If you have three nullable columns
a : int, b : string, c : map<string,int>
data:
if you set trim_double_quotes = true
you will get :
if you set trim_double_quotes = false
you will get :
in csv(text) for normal type: we only recognize \N for null , so
for not char family type, like int, if we put null literal ,
it will parse fail, and make result null,not just because it equals \N.
for char family type, like string, if we put null literal, it will parse success,
and "null" literal will be stored in doris.
For strings in the json complex type, we remove double quotes by default.
Because when querying complex types, such as selecting complexColumn from table,
we will add double quotes to the strings in the complex type.
For the map<string,int> column, insert { "abc" : 1, "hello",2 }.
If you do not remove the double quotes, it will display {""abc"":1,""hello"": 2 },
remove the double quotes to display { "abc" : 1, "hello",2 }.
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...