[enhance](hive) Add regression-test cases for hive text ddl and hive text insert and fix reading null string bug by suxiaogang223 · Pull Request #42200 · apache/doris

suxiaogang223 · 2024-10-21T09:23:27Z

Proposed changes

Add regression-test cases for hive text table properties:

Property	Description	Example Value	Supported in Doris
`field.delim`	Defines the delimiter between columns in each row.	`\1`	Yes
`collection.delim`	Defines the delimiter for items in an array (collection type).	`\2`	Yes
`mapkey.delim`	Defines the delimiter between keys and values in a map.	`\3`	Yes
`serialization.null.format`	Defines how `NULL` values are represented in the text file.	`\\N`	Yes
`escape.delim`	Specifies the escape character used for escaping special characters.	`\\`	Yes
`line.delim`	Defines the delimiter between rows or lines in the file.	`\n`	Yes

Explanation:

field.delim: This property is used to specify how columns are separated in a row. For example, \1 indicates the delimiter is a non-printable character.
collection.delim: Used to define how elements in an array or collection are separated. In this case, \2 is used as a separator.
mapkey.delim: This property defines how keys and values in a map are separated. For instance, \3 is used to separate map keys and values.
serialization.null.format: This setting specifies the format for NULL values in the data. \\N is commonly used to represent NULL.
escape.delim: Defines the escape character used in the text file for escaping special characters, such as the delimiter itself. Here, \\ is used as the escape character.
line.delim: This property is used to specify the delimiter between lines or rows. Typically, \n (newline) is used as the line delimiter.
Note: Unlike the other delimiters, line.delim is not escaped. If the content in the table contains the same character as the line delimiter, it may cause query errors. However, the other delimiters (field.delim, collection.delim, mapkey.delim) are escaped, so they will not cause issues.

doris-robot · 2024-10-21T09:23:31Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morningman

LGTM

github-actions · 2024-10-21T12:06:19Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-10-21T12:06:22Z

PR approved by anyone and no changes requested.

suxiaogang223 · 2024-10-21T12:33:44Z

run buildall

suxiaogang223 · 2024-10-22T01:42:57Z

run external

suxiaogang223 · 2024-10-22T01:49:15Z

run buildall

suxiaogang223 · 2024-10-22T07:24:47Z

run buildall

github-actions · 2024-10-22T07:25:55Z

clang-tidy review says "All clean, LGTM! 👍"

morningman

LGTM

github-actions · 2024-10-22T09:29:46Z

PR approved by at least one committer and no changes requested.

…text insert and fix reading null string bug (apache#42200) ## Proposed changes Add regression-test cases for hive text table properties: | **Property** | **Description** | **Example Value** | **Supported in Doris** | |------------------------------|---------------------------------------------------------------------|-------------------|----------------------| | `field.delim` | Defines the delimiter between columns in each row. | `\1` | Yes | | `collection.delim` | Defines the delimiter for items in an array (collection type). | `\2` | Yes | | `mapkey.delim` | Defines the delimiter between keys and values in a map. | `\3` | Yes | | `serialization.null.format` | Defines how `NULL` values are represented in the text file. | `\\N` | Yes | | `escape.delim` | Specifies the escape character used for escaping special characters.| `\\` | Yes | | `line.delim` | Defines the delimiter between rows or lines in the file. | `\n` | Yes | ### Explanation: - **`field.delim`**: This property is used to specify how columns are separated in a row. For example, `\1` indicates the delimiter is a non-printable character. - **`collection.delim`**: Used to define how elements in an array or collection are separated. In this case, `\2` is used as a separator. - **`mapkey.delim`**: This property defines how keys and values in a map are separated. For instance, `\3` is used to separate map keys and values. - **`serialization.null.format`**: This setting specifies the format for `NULL` values in the data. `\\N` is commonly used to represent `NULL`. - **`escape.delim`**: Defines the escape character used in the text file for escaping special characters, such as the delimiter itself. Here, `\\` is used as the escape character. - **`line.delim`**: This property is used to specify the delimiter between lines or rows. Typically, `\n` (newline) is used as the line delimiter. **Note**: Unlike the other delimiters, `line.delim` is not escaped. If the content in the table contains the same character as the line delimiter, it may cause query errors. However, the other delimiters (`field.delim`, `collection.delim`, `mapkey.delim`) are escaped, so they will not cause issues.

…text insert and fix reading null string bug #42200 (#42272) cherry pick from #42200 Co-authored-by: Socrates <suxiaogang223@icloud.com>

…text insert and fix reading null string bug #42200 (#42273) cherry pick from #42200 Co-authored-by: Socrates <suxiaogang223@icloud.com>

## Proposed changes fellow #42200 Null values in text tables are now correctly recognized now.

## Proposed changes fellow apache#42200 Null values in text tables are now correctly recognized now.

## Proposed changes fellow #42200 Null values in text tables are now correctly recognized now.

…text insert and fix reading null string bug (apache#42200) ## Proposed changes Add regression-test cases for hive text table properties: | **Property** | **Description** | **Example Value** | **Supported in Doris** | |------------------------------|---------------------------------------------------------------------|-------------------|----------------------| | `field.delim` | Defines the delimiter between columns in each row. | `\1` | Yes | | `collection.delim` | Defines the delimiter for items in an array (collection type). | `\2` | Yes | | `mapkey.delim` | Defines the delimiter between keys and values in a map. | `\3` | Yes | | `serialization.null.format` | Defines how `NULL` values are represented in the text file. | `\\N` | Yes | | `escape.delim` | Specifies the escape character used for escaping special characters.| `\\` | Yes | | `line.delim` | Defines the delimiter between rows or lines in the file. | `\n` | Yes | ### Explanation: - **`field.delim`**: This property is used to specify how columns are separated in a row. For example, `\1` indicates the delimiter is a non-printable character. - **`collection.delim`**: Used to define how elements in an array or collection are separated. In this case, `\2` is used as a separator. - **`mapkey.delim`**: This property defines how keys and values in a map are separated. For instance, `\3` is used to separate map keys and values. - **`serialization.null.format`**: This setting specifies the format for `NULL` values in the data. `\\N` is commonly used to represent `NULL`. - **`escape.delim`**: Defines the escape character used in the text file for escaping special characters, such as the delimiter itself. Here, `\\` is used as the escape character. - **`line.delim`**: This property is used to specify the delimiter between lines or rows. Typically, `\n` (newline) is used as the line delimiter. **Note**: Unlike the other delimiters, `line.delim` is not escaped. If the content in the table contains the same character as the line delimiter, it may cause query errors. However, the other delimiters (`field.delim`, `collection.delim`, `mapkey.delim`) are escaped, so they will not cause issues.

## Proposed changes fellow apache#42200 Null values in text tables are now correctly recognized now.

morningman previously approved these changes Oct 21, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Oct 21, 2024

suxiaogang223 dismissed morningman’s stale review via 92b7ecbOctober 21, 2024 12:33

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Oct 21, 2024

suxiaogang223 added 5 commits October 22, 2024 09:49

add regression
ed8a25e

gen output
c1c2100

test hive ddl text format for hive2 and hive3
6f80152

test text properties in test_hive_text_write_insert
3e1c3ac

chek hive result
74df02b

suxiaogang223 force-pushed the add_case_for_hive_text branch from 92b7ecb to 74df02bCompare October 22, 2024 01:49

fix csv_reader for null_format
4325abe

suxiaogang223 changed the title ~~[enhance](hive) Add regression-test cases for hive text ddl and hive text insert~~[enhance](hive) Add regression-test cases for hive text ddl and hive text insert and fix reading null string bugOct 22, 2024

morningman approved these changes Oct 22, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 22, 2024

CalvinKirs approved these changes Oct 22, 2024
View reviewed changes

morningman added dev/2.1.x dev/3.0.x labels Oct 22, 2024

morningman merged commit 85f7a5e into apache:masterOct 22, 2024

morningman mentioned this pull request Oct 22, 2024
[enhance](hive) Add regression-test cases for hive text ddl and hive text insert and fix reading null string bug #42200 #42272
Merged

morningman mentioned this pull request Oct 22, 2024
[enhance](hive) Add regression-test cases for hive text ddl and hive text insert and fix reading null string bug #42200 #42273
Merged

morningman added a commit that referenced this pull request Oct 22, 2024
[enhance](hive) Add regression-test cases for hive text ddl and hive …
53ee740
…text insert and fix reading null string bug #42200 (#42272) cherry pick from #42200 Co-authored-by: Socrates <suxiaogang223@icloud.com>

morningman added dev/3.0.3-merged and removed dev/3.0.x labels Oct 22, 2024

morningman added a commit that referenced this pull request Oct 22, 2024
[enhance](hive) Add regression-test cases for hive text ddl and hive …
157d67e
…text insert and fix reading null string bug #42200 (#42273) cherry pick from #42200 Co-authored-by: Socrates <suxiaogang223@icloud.com>

morningman added dev/2.1.7-merged and removed dev/2.1.x labels Oct 22, 2024

suxiaogang223 mentioned this pull request Oct 30, 2024
[fix](regression-test) fix test_hive_serde_prop #42886
Merged

morningman pushed a commit that referenced this pull request Nov 1, 2024
[fix](regression-test) fix test_hive_serde_prop (#42886)
beef40f
## Proposed changes fellow #42200 Null values in text tables are now correctly recognized now.

morningman pushed a commit to morningman/doris that referenced this pull request Nov 1, 2024
[fix](regression-test) fix test_hive_serde_prop (apache#42886)
37782fe
## Proposed changes fellow apache#42200 Null values in text tables are now correctly recognized now.

morningman pushed a commit to morningman/doris that referenced this pull request Nov 1, 2024
[fix](regression-test) fix test_hive_serde_prop (apache#42886)
970fad8
## Proposed changes fellow apache#42200 Null values in text tables are now correctly recognized now.

github-actionsBot pushed a commit that referenced this pull request Nov 1, 2024
[fix](regression-test) fix test_hive_serde_prop (#42886)
600666d
## Proposed changes fellow #42200 Null values in text tables are now correctly recognized now.

github-actionsBot mentioned this pull request Nov 1, 2024
branch-3.0: [fix](regression-test) fix test_hive_serde_prop #43100
Closed

suxiaogang223 deleted the add_case_for_hive_text branch December 12, 2024 09:27

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[fix](regression-test) fix test_hive_serde_prop (apache#42886)
dac984e
## Proposed changes fellow apache#42200 Null values in text tables are now correctly recognized now.

Navigation Menu

Uh oh!

[enhance](hive) Add regression-test cases for hive text ddl and hive text insert and fix reading null string bug - #42200

[enhance](hive) Add regression-test cases for hive text ddl and hive text insert and fix reading null string bug#42200
morningman merged 6 commits into
apache:masterfrom
suxiaogang223:add_case_for_hive_text

suxiaogang223 commented Oct 21, 2024

Uh oh!

doris-robot commented Oct 21, 2024

Uh oh!

morningman left a comment

Uh oh!

github-actionsBot commented Oct 21, 2024

Uh oh!

github-actionsBot commented Oct 21, 2024

Uh oh!

suxiaogang223 commented Oct 21, 2024

Uh oh!

suxiaogang223 commented Oct 22, 2024

Uh oh!

suxiaogang223 commented Oct 22, 2024

Uh oh!

suxiaogang223 commented Oct 22, 2024

Uh oh!

github-actionsBot commented Oct 22, 2024

Uh oh!

morningman left a comment

Uh oh!

github-actionsBot commented Oct 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

suxiaogang223 commented Oct 21, 2024

Proposed changes

Explanation:

Uh oh!

doris-robot commented Oct 21, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Oct 21, 2024

Uh oh!

github-actionsBot commented Oct 21, 2024

Uh oh!

suxiaogang223 commented Oct 21, 2024

Uh oh!

suxiaogang223 commented Oct 22, 2024

Uh oh!

suxiaogang223 commented Oct 22, 2024

Uh oh!

suxiaogang223 commented Oct 22, 2024

Uh oh!

github-actionsBot commented Oct 22, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Oct 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants