(Enhancement)[load-json] support simdjson in new json reader by eldenmoon · Pull Request #16903 · apache/doris

eldenmoon · 2023-02-18T11:53:39Z

Proposed changes

be config:
enable_simdjson_reader=true

related PR #11665
related isue #11663

Issue Number: close #xxx

Problem summary

Describe your changes.

Checklist(Required)

Does it affect the original behavior
Has unit tests been added
Has document been added or modified
Does it need to update dependencies
Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

be config: enable_simdjson_reader=true related PR apache#11665

github-actions · 2023-02-18T11:56:55Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-02-18T11:58:27Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-02-18T12:06:55Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-02-18T12:19:28Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-02-18T12:38:21Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-02-18T13:22:08Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.44 seconds
stream load tsv: 467 seconds loaded 74807831229 Bytes, about 152 MB/s
stream load json: 36 seconds loaded 2358488459 Bytes, about 62 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230220121041_clickbench_pr_100245.html

github-actions · 2023-02-18T14:41:49Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-02-18T14:44:11Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-02-18T15:11:20Z

clang-tidy review says "All clean, LGTM! 👍"

xiaokang · 2023-02-19T13:59:05Z

+ fmt::format("failed to access field as array, field: {}", col));
+
+ HANDLE_SIMDJSON_ERROR(
+ arr.at(index).get(tvalue),


Can simdjson properly process index == -2, which means * ?

no index == -2 is not supported in this function for now

xiaokang · 2023-02-19T14:00:42Z

+ if (UNLIKELY(_err)) { \
+ if (_err == simdjson::NO_SUCH_FIELD || _err == simdjson::INDEX_OUT_OF_BOUNDS) { \
+ return Status::NotFound( \
+ fmt::format("err: {}, msg: {}", simdjson::error_message(_err), _msg)); \


err: to simdjson err:

what's the meaning of err: to simdjson err:?

eldenmoon · 2023-02-19T15:02:16Z

+ fmt::format("failed to access field as array, field: {}", col));
+
+ HANDLE_SIMDJSON_ERROR(
+ arr.at(index).get(tvalue),


no index == -2 is not supported in this function for now

eldenmoon · 2023-02-19T15:03:07Z

+ if (UNLIKELY(_err)) { \
+ if (_err == simdjson::NO_SUCH_FIELD || _err == simdjson::INDEX_OUT_OF_BOUNDS) { \
+ return Status::NotFound( \
+ fmt::format("err: {}, msg: {}", simdjson::error_message(_err), _msg)); \


what's the meaning of err: to simdjson err:?

github-actions · 2023-02-20T05:14:01Z

clang-tidy review says "All clean, LGTM! 👍"

eldenmoon · 2023-02-20T06:11:56Z

run buildall

eldenmoon · 2023-02-20T07:13:45Z

run p0

xiaokang

LGTM

xiaokang · 2023-02-20T08:55:24Z

+ fmt::format("failed to access field as array, field: {}", col));
+
+ HANDLE_SIMDJSON_ERROR(
+ arr.at(index).get(tvalue),


github-actions · 2023-02-20T08:56:20Z

PR approved by anyone and no changes requested.

github-actions · 2023-02-20T10:35:38Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-02-20T10:50:46Z

clang-tidy review says "All clean, LGTM! 👍"

eldenmoon · 2023-02-20T11:02:32Z

run buildall

github-actions · 2023-02-20T11:06:05Z

clang-tidy review says "All clean, LGTM! 👍"

qidaye

LGTM

github-actions · 2023-02-20T12:30:16Z

PR approved by at least one committer and no changes requested.

WinkerDu · 2023-02-24T17:31:09Z

@eldenmoon This patch would cause BE compile error on MacOS with clang below

Apple clang version 13.0.0 (clang-1300.0.27.3)
Target: arm64-apple-darwin21.3.0
Thread model: posix

…16903) be config: enable_simdjson_reader=true related PR apache#11665

* (Enhancement)[load-json] support simdjson in new json reader (apache#16903) be config: enable_simdjson_reader=true related PR apache#11665 * [Optimize](simd json reader) Cached search results for previous row (keyed as index in JSON object) - used as a hint. (apache#17124) * [Optimize](simd json reader) Cached search results for previous row (keyed as index in JSON object) - used as a hint. `_simdjson_set_column_value` could become a hot spot while parsing json in simdjson mode, introduce `_prev_positions` to cache results for previous row (keyed as index in JSON object) due to the json name field order, should be quite the same between each lines * fix case

… (apache#1445) * (Enhancement)[load-json] support simdjson in new json reader (apache#16903) be config: enable_simdjson_reader=true related PR apache#11665 * [Optimize](simd json reader) Cached search results for previous row (keyed as index in JSON object) - used as a hint. (apache#17124) * [Optimize](simd json reader) Cached search results for previous row (keyed as index in JSON object) - used as a hint. `_simdjson_set_column_value` could become a hot spot while parsing json in simdjson mode, introduce `_prev_positions` to cache results for previous row (keyed as index in JSON object) due to the json name field order, should be quite the same between each lines * fix case

eldenmoon changed the title ~~Simdjson~~(Enhancement)[load-json] support simdjson in new json readerFeb 18, 2023

github-actionsBot added area/vectorization kind/test labels Feb 18, 2023

(Enhancement)[load-json] support simdjson in new json reader
9216cc6
be config: enable_simdjson_reader=true related PR apache#11665

eldenmoon force-pushed the simdjson branch from f14a984 to 9216cc6Compare February 18, 2023 11:55

github-actionsBot added the kind/docs Categorizes issue or PR as related to documentation. label Feb 18, 2023

add config doc
0ec87c3

eldenmoon force-pushed the simdjson branch from 0081c6b to 0ec87c3Compare February 18, 2023 12:15

fix ut compile
998495d

add more malformed json cases
9dd6031

eldenmoon force-pushed the simdjson branch from 18c8e3f to a2bfcbcCompare February 18, 2023 15:07

set enable_simdjson_reader fuzzy
3397b89

xiaokang reviewed Feb 19, 2023
View reviewed changes

eldenmoon commented Feb 19, 2023
View reviewed changes

add more cases and fix bug
ac8fc37

eldenmoon force-pushed the simdjson branch from a2bfcbc to ac8fc37Compare February 20, 2023 05:10

eldenmoon requested a review from xiaokang February 20, 2023 05:16

xiaokang previously approved these changes Feb 20, 2023
View reviewed changes

github-actionsBot added the reviewed label Feb 20, 2023

eldenmoon dismissed xiaokang’s stale review via 9d23118February 20, 2023 10:31

eldenmoon force-pushed the simdjson branch from 9d23118 to 62104a8Compare February 20, 2023 10:47

modify
4034e06

eldenmoon force-pushed the simdjson branch from 62104a8 to 4034e06Compare February 20, 2023 11:00

eldenmoon requested a review from xiaokang February 20, 2023 11:27

qidaye approved these changes Feb 20, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 20, 2023

xiaokang approved these changes Feb 20, 2023
View reviewed changes

qidaye merged commit 113023f into apache:masterFeb 21, 2023

yagagagaga pushed a commit to yagagagaga/doris that referenced this pull request Mar 9, 2023
(Enhancement)[load-json] support simdjson in new json reader (apache#…
df2bc5d
…16903) be config: enable_simdjson_reader=true related PR apache#11665

Uh oh!

Conversation

eldenmoon commented Feb 18, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem summary

Checklist(Required)

Further comments

Uh oh!

github-actionsBot commented Feb 18, 2023

Uh oh!

github-actionsBot commented Feb 18, 2023

Uh oh!

github-actionsBot commented Feb 18, 2023

Uh oh!

github-actionsBot commented Feb 18, 2023

Uh oh!

github-actionsBot commented Feb 18, 2023

Uh oh!

hello-stephen commented Feb 18, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Feb 18, 2023

Uh oh!

github-actionsBot commented Feb 18, 2023

Uh oh!

github-actionsBot commented Feb 18, 2023

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actionsBot commented Feb 20, 2023

Uh oh!

eldenmoon commented Feb 20, 2023

Uh oh!

eldenmoon commented Feb 20, 2023

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Feb 20, 2023

Uh oh!

github-actionsBot commented Feb 20, 2023

Uh oh!

github-actionsBot commented Feb 20, 2023

Uh oh!

eldenmoon commented Feb 20, 2023

Uh oh!

github-actionsBot commented Feb 20, 2023

Uh oh!

eldenmoon commented Feb 18, 2023•
edited
Loading

hello-stephen commented Feb 18, 2023•
edited
Loading