Uh oh!
There was an error while loading. Please reload this page.
[Feature][Variant] support support schema for inner sub types in variant type - #40573
[Feature][Variant] support support schema for inner sub types in variant type#40573eldenmoon wants to merge 1 commit into
Conversation
doris-robot
commented
Sep 9, 2024
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
eldenmoon
commented
Sep 9, 2024
run buildall |
doris-robot
commented
Sep 9, 2024
TPC-H: Total hot run time: 38251 ms |
doris-robot
commented
Sep 9, 2024
TPC-DS: Total hot run time: 192755 ms |
doris-robot
commented
Sep 9, 2024
ClickBench: Total hot run time: 31.83 s |
eldenmoon
commented
Sep 10, 2024
run buildall |
doris-robot
commented
Sep 10, 2024
TeamCity be ut coverage result: |
doris-robot
commented
Sep 10, 2024
TPC-H: Total hot run time: 38203 ms |
doris-robot
commented
Sep 10, 2024
TPC-DS: Total hot run time: 191297 ms |
doris-robot
commented
Sep 10, 2024
ClickBench: Total hot run time: 31.46 s |
eldenmoon
commented
Sep 10, 2024
run buildall |
doris-robot
commented
Sep 10, 2024
TeamCity be ut coverage result: |
eldenmoon
commented
Sep 10, 2024
run buildall |
doris-robot
commented
Sep 10, 2024
TeamCity be ut coverage result: |
doris-robot
commented
Sep 10, 2024
TPC-H: Total hot run time: 38429 ms |
doris-robot
commented
Sep 10, 2024
TPC-DS: Total hot run time: 197213 ms |
doris-robot
commented
Sep 10, 2024
ClickBench: Total hot run time: 31.33 s |
eldenmoon
commented
Sep 11, 2024
run buildall |
doris-robot
commented
Sep 11, 2024
TPC-H: Total hot run time: 38214 ms |
doris-robot
commented
Sep 11, 2024
TPC-DS: Total hot run time: 192847 ms |
doris-robot
commented
Sep 11, 2024
TeamCity be ut coverage result: |
doris-robot
commented
Sep 11, 2024
ClickBench: Total hot run time: 31.69 s |
eldenmoon
commented
Sep 13, 2024
run buildall |
| @@ -81,6 +81,15 @@ class DataTypeJsonb final : public IDataType { | |||
| return String(value.value(), value.size()); | |||
eldenmoon
commented
Oct 10, 2024
run buildall |
eldenmoon
commented
Oct 10, 2024
run buildall |
| // Currently the jsonb type should be the top level type, so we should not wrap it in array, | ||
| // see create_array_of_type. | ||
| // TODO we need to support array<jsonb> correctly | ||
| if (UNLIKELY(field.get_type_id() == TypeIndex::JSONB && info->num_dimensions > 0)) { |
There was a problem hiding this comment.
warning: boolean expression can be simplified by DeMorgan's theorem [readability-simplify-boolean-expr]
if (UNLIKELY(field.get_type_id() == TypeIndex::JSONB && info->num_dimensions > 0)) {
^Additional context
be/src/common/compiler_util.h:35: expanded from macro 'UNLIKELY'
#defineUNLIKELY(expr) __builtin_expect(!!(expr), 0)
^| } | ||
| DataTypePtr DataTypeFactory::create_data_type(const TypeIndex& type_index, bool is_nullable) { | ||
| DataTypePtr DataTypeFactory::create_data_type(const TypeIndex& type_index, bool is_nullable, |
There was a problem hiding this comment.
warning: function 'create_data_type' exceeds recommended size/complexity thresholds [readability-function-size]
DataTypePtr DataTypeFactory::create_data_type(const TypeIndex& type_index, bool is_nullable,
^Additional context
be/src/vec/data_types/data_type_factory.cpp:250: 114 lines including whitespace and comments (threshold 80)
DataTypePtr DataTypeFactory::create_data_type(const TypeIndex& type_index, bool is_nullable,
^| @@ -20,6 +20,7 @@ | |||
| #include <glog/logging.h> | |||
There was a problem hiding this comment.
warning: 'glog/logging.h' file not found [clang-diagnostic-error]
#include<glog/logging.h>
^doris-robot
commented
Oct 10, 2024
TeamCity be ut coverage result: |
eldenmoon
commented
Oct 10, 2024
run buildall |
doris-robot
commented
Oct 10, 2024
TeamCity be ut coverage result: |
| validateNestedType(scalarType, fieldType); | ||
| if (!fieldNames.add(field.getName())) { | ||
| throw new AnalysisException("Duplicate field name " + field.getName() | ||
| + " in struct " + scalarType.toSql()); |
| } | ||
| column_with_cast_type.set_type(filed_type); | ||
| column_with_cast_type.set_precision_frac(cast_type.precision, cast_type.scale); | ||
| column_with_cast_type.set_is_decimal(cast_type.precision > 0); |
There was a problem hiding this comment.
It's not rigorous to use cast_type.precision > 0 to judge is_decimal. You can use cast_type.is_decimal_v2_type() and cast_type.is_decimal_v3_type() or add a new is_decimal_type for cast_type.
| auto full_path = full_path_builder.append(parent_column->name_lower_case(), false) | ||
| .append(entry->path.get_parts(), false) | ||
| .build(); | ||
| if (typed_columns.contains(entry->path.get_path())) { |
There was a problem hiding this comment.
What's the purpose of typed_columns?
| Field get_type_field(const IColumn& column, int row) const override { | ||
| Field field; | ||
| column.get(row, field); | ||
| field.set_type_info(get_type_id(), 0, static_cast<int>(get_scale())); |
| assert_cast<const ColumnUInt64&, TypeCheckOnRelease::DISABLE>(column); | ||
| Field field; | ||
| column_data.get(row, field); | ||
| field.set_type_info(get_type_id(), 0, static_cast<int>(get_scale())); |
eldenmoon
commented
Nov 28, 2024
run buildall |
doris-robot
commented
Nov 28, 2024
TPC-H: Total hot run time: 40205 ms |
doris-robot
commented
Nov 28, 2024
TeamCity be ut coverage result: |
doris-robot
commented
Nov 28, 2024
TPC-DS: Total hot run time: 197059 ms |
doris-robot
commented
Nov 28, 2024
ClickBench: Total hot run time: 32.62 s |
eldenmoon
commented
Dec 2, 2024
run buildall |
eldenmoon
commented
Dec 3, 2024
run buildall |
Background
Currently we support auto detect schema info from semi-structure json, but when we encounter something like '2021-01-01' we just store as string type and query like string type, if we use it as date type, it maybe slow.So in this PR, we support to specify subschemas for variant type.
Usage
Design
like struct, we store we predefined fields as static type and no need to do schema merge