Uh oh!
There was an error while loading. Please reload this page.
Decimal multiply kernel should not cause precision loss - #5980
Conversation
Different to #5675, this doesn't add new expression node |
54397f9 to
343ca79Compareviirya
commented
Apr 16, 2023
There is a compilation error. Going to fix it at #6029. |
cb7e326 to
0a88516Compare| Some(99193548387), // 0.99193548387 | ||
| None, | ||
| None, | ||
| Some(100813008130), // 1.0081300813 | ||
| Some(100000000000), // 1.0 | ||
| ], | ||
| 21, | ||
| 11, |
There was a problem hiding this comment.
Previously, this division losses precision. Now we get it back.
| // subtract: decimal array subtract int32 array | ||
| let schema = Arc::new(Schema::new(vec![ | ||
| Field::new("b", DataType::Int32, true), | ||
| Field::new("a", DataType::Decimal128(10, 2), true), |
There was a problem hiding this comment.
Previously the field order is incorrect. But as we did coerce type on both side of the op anyway, so it still worked before. Now we don't coerce the decimal field (which is wrongly bound to Int32Array) before into binary expression, so wrong field causes an error.
| sum(l_extendedprice) as sum_base_price, | ||
| sum(l_extendedprice * (1 - l_discount)) as sum_disc_price, | ||
| sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge, | ||
| sum(cast(l_extendedprice as decimal(12,2)) * (1 - l_discount) * (1 + l_tax)) as sum_charge, |
There was a problem hiding this comment.
| cast(cast(sum(case | ||
| when nation = 'BRAZIL' then volume | ||
| else 0 | ||
| end) as decimal(12,2)) / cast(sum(volume) as decimal(12,2)) as decimal(15,2)) as mkt_share |
There was a problem hiding this comment.
| pub fn i128_to_str(value: i128, precision: &u8, scale: &i8) -> String { | ||
| big_decimal_to_str( | ||
| BigDecimal::from_str(&Decimal::from_i128_with_scale(value, scale).to_string()) | ||
| BigDecimal::from_str(&Decimal128Type::format_decimal(value, *precision, *scale)) |
There was a problem hiding this comment.
This deals with the decimal precision issue without additional |
Dandandan
commented
Apr 18, 2023
I wonder if this already fixes #4024 |
viirya
commented
Apr 18, 2023
Yea, just verified locally that this can pass |
viirya
commented
Apr 18, 2023
Thanks @Dandandan |
Dandandan
commented
Apr 19, 2023
Let's wait ~24hrs so other reviewers can have a chance. |
Dandandan
commented
Apr 19, 2023
FYI @mingmwang@andygrove this PR also has some effect on performance, as casting is changed (mostly reduced). |
Dandandan
commented
Apr 19, 2023
Ran the benchmarks for TPCH(SF=1) in memory. Performance is mostly the same, except a ~30% improvement for q1 compared to main 🚀 |
alamb
commented
Apr 24, 2023
🎉 |
Which issue does this PR close?
Closes#5674.
Closes#3387.
Closes#4024.
Rationale for this change
Currently decimal multiplication in DataFusion silently truncates precision of result. It happens generally for regular decimal multiplication which doesn't overflow. Looks like DataFusion uses incomplete decimal precision coercion rule from Spark to coerce sides of decimal multiplication (and other arithmetic operators). The coerced type on two sides of decimal multiplication is not the resulting decimal type of multiplication. This (and how we computes decimal multiplication in the kernels) leads to truncated precision in the result decimal type.
What changes are included in this PR?
TypeCoercionto physical binary operatorAre these changes tested?
Are there any user-facing changes?