PARQUET-2249: Add IEEE-754 total order and nan count for floating types

[wgtmac]

Rationale for this change

This implements the parquet-format IEEE 754 total order column-order work from apache/parquet-format#514 in parquet-java. The existing type-defined order remains the default.

While adding the new order, this also fixes floating-point NaN handling so parquet-java preserves the exact FLOAT, DOUBLE, and FLOAT16 bit patterns supplied by applications, including NaN sign and payload bits. Filters are updated to avoid false negatives when NaN semantics cannot be answered safely from metadata.

What changes are included in this PR?

• Add IEEE_754_TOTAL_ORDER support for FLOAT, DOUBLE, and FLOAT16, including comparators that distinguish -0/+0 and NaN bit patterns.
• Add floating-point nan_count support in statistics and column indexes, with IEEE-754-specific statistics implementations.
• Preserve raw floating-point bit patterns in parquet-java floating encodings, including dictionary and byte-stream-split paths.
• Update statistics, column-index, dictionary, bloom-filter, and record-level filtering so NaN values are handled conservatively where required.
• Add coverage for statistics, column indexes, dictionary filtering, bloom filtering, record-level filtering, Float16, and parquet-testing interop files.

Are these changes tested?

Yes. This PR adds focused unit and end-to-end tests for the new column order, NaN counts, NaN filtering behavior, raw-bit preservation, and interop with parquet-testing files.

Are there any user-facing changes?

Yes. Users can opt into IEEE_754_TOTAL_ORDER for floating columns. The default column order is unchanged, and existing files remain readable. Filtering around NaN values may be more conservative to avoid dropping data that can still match.

Closes#406

[shangxinli]

Thanks Gang for picking this up and driving it forward!

+1 on the approach. This combined solution addresses both the ordering ambiguity and the NaN pollution concern pragmatically. Looking forward to seeing the arrow-cpp PoC as well.

[etseidl]

@wgtmac thank you for adding the interop test! 🙏 In arrow-rs we've made the decision to only write the new column order for floats, so I can't reproduce the total order columns.

Some things I think need to be added are negative NaNs, as well as examples where the min and/or max are 0. The latter is to make sure that the old rules regarding 0 min being set to -0 and 0 max set +0 are no longer followed with the new ordering.

[wgtmac]

In arrow-rs we've made the decision to only write the new column order for floats, so I can't reproduce the total order columns.

Did you mean arrow-rs will no longer write floats with the legacy TypeDefinedOrder? From the perspective of interoperability test, I think this is fine if it does not fail when reading files produced by other writers.

Some things I think need to be added are negative NaNs, as well as examples where the min and/or max are 0

That's a good suggestion! I have updated the floating-point interop coverage to add explicit ZERO_MIN and ZERO_MAX cases, so we now verify that IEEE-754 total order no longer rewrites +0 min to -0 or -0 max to +0. I also expanded the NaN coverage to include both negative and positive NaN patterns.

While debugging the test, I found that the Java implementation uses Float.floatToIntBits instead of Float.floatToRawIntBits (same for double) which canonicalizes NaN bits and pollutes both values and stats. I fixed the float/double write paths to preserve raw NaN bits instead of canonicalizing them.

[etseidl]

Thank you. I hope to have the rust tests done today.

[wgtmac]

@shangxinli@etseidl Do you want to take a look again? I think now everything is ready on my end. cc @gszadovszky@Fokko

[gszadovszky]

Added a couple of comments.

Also, it would be nice to add coverage for predicate filtering with signed NaN values. Since NaN value are excluded from the statistics, this is only for the value level (ValueInspector). Can be either covered unit or integration level.

[wgtmac]

Thanks @gszadovszky for the review! I think I've addressed all your comments. Let me know what you think.

[gszadovszky]

Thank you, @wgtmac. One more thing.

This PR also changes how NaN values are encoded in dictionaries. I'm not sure if it is tightly related to IEEE-754, and is a breaking change. Before, we have lost the original NaN value during the dictionary encoding, and encoded every NaNs into the same bit pattern (e.g. 0x7ff8000000000000L for doubles). With this change, we preserve the actual bit pattern, whatever it means. We can say this is a bug fix, but maybe, we should discuss this behavioral change with the community first.

If we move to that direction, we also need to fix the dictionary filter. We use a boxed HashSet there, so even if the dictionary itself preserves the original bit patterns, the filter itself does not.

[wgtmac]

@gszadovszky That's a good question! I think the main behavior change is that now original NaN bits are preserved in not only dictionary but also encoded values. I would regard this as a benign bug fix.

For dictionary filter and bloom filter on the read path, they are not aware of column order. Introducing IEEE754 total order is anyway a breaking change to them because raw bits of NaNs must be preserved as is.

Update: I've sent https://lists.apache.org/thread/m6j8lzc09ytyd45wt6pdcyn5qy95f0vt to discuss this.

The discussion is still going on.

[wgtmac]

I updated the patch so the pruning filters and record-level filters have separate semantics:

• Dictionary and Bloom filters now handle NaN literals conservatively and return “might match”. So we don't need to worry about different NaN semantics and boxed NaN values in the dictionary.
• Record-level filtering does not special-case NaN. It uses the column’s PrimitiveComparator, so the behavior follows the declared column order: TYPE_DEFINED_ORDER treats NaNs according to the existing type-defined comparator semantics, while IEEE_754_TOTAL_ORDER compares the raw floating-point bits. The same applies to +0/-0.

I still think that preserving raw bits of NaN values is a bug fix and benign breaking change. Let me know what you think. @gszadovszky

[gszadovszky]

Thanks @wgtmac for the updates. I think one issue remains with NaNs in the dictionary. We "canonicalize" the NaN values in the dictionary filter. So, if there are NaN and -NaN values in the dictionary, and the filter has a range predicate (e.g. > 0), we may drop the row group even if there should be a match based on the total ordering. We probably need to allow a "might match" explicitly if any NaN values are in the dictionary.

[wgtmac]

That's my oversight and thanks @gszadovszky for catching this! Now it should be fixed as well.

[gszadovszky]

Thank you, @wgtmac. LGTM.

[wgtmac]

Merged. Thanks @etseidl@shangxinli@gszadovszky for the review!

[Jiayi-Wang-db]

@wgtmac Sorry for a late question about this pr, why are we still using the old type_order column order for float columns by default? And shouldn't we start writing with IEEE754_total_order as suggested in parquet spec?

[wgtmac]

@Jiayi-Wang-db That's a great question. I was supposed not to break any legacy readers. But I agree that legacy readers should just ignore unknown column order if they are implemented correctly.

[Jiayi-Wang-db]

@wgtmac My concern is that the new column order cannot even be enabled without a code change.

[wgtmac]

Why? Couldn't we explicitly create a Node with a specific column order just like the new unit test does?

[Jiayi-Wang-db]

Yes, but it needs a writer-side code change.
Do you think we need to have a breaking change note before writing new column order by default?

[Jiayi-Wang-db]

Actually, the risker breaking change is reusing type_order while computing min/max over only the non-NaN subset.
A legacy reader that recognizes type_order but not nan_count may use the min/max values without realizing that NaNs are present. It assumes that min/max statistics are omitted whenever NaNs exist, based on the old writer behavior.
With a new column order, there's chance that reader checks the column order, and drop the stats for UNKNOWN column order.
I think we need to have a clear decision on this before having the new release.

[wgtmac]

A legacy reader that recognizes type_order but not nan_count may use the min/max values without realizing that NaNs are present.

I don't think so. NaNs are not written to the min/max for type_order, otherwise it is a bug. Could you double check? BTW, legacy readers cannot assume min/max statistics are omitted whenever NaNs exist. They are just educated to regard stats with NaN values as invalid.

With a new column order, there's chance that reader checks the column order, and drop the stats for UNKNOWN column order.

This is exactly the concern that I have. Readers may have just hard-coded type-order without notice of a new column order.

[Jiayi-Wang-db]

NaNs are not written to min/max under type_order.

Correct. However, the scenario I’m describing is the following:

An older version of parquet-java that predates nan_count—for example, 1.12.x—reads a Parquet file containing a float column that uses type_order, includes nan_count, and contains NaN values.

• nan_count is not present in the older reader’s generated Thrift Statistics class, so the reader never sees it.
• The column uses TYPE_ORDER, so isMinMaxStatsSupported returns true, and the finite min/max values are loaded.
• The NaN guard in DoubleBuilder.build() does not trigger because the bounds themselves are finite, so the statistics remain valid with hasNonNullValue == true.
• StatisticsFilter then uses the max value for pruning.

As a result, rows containing NaN values can be incorrectly pruned.
This is worse than the case where older readers simply cannot use statistics written with the new column order. @wgtmac

[Fokko]

Joining the party late here. I just checked, and for example 1.12.x would ignore the min/max statistics when it encounters an unknown sort order (according to Claude Opus):

So it's safe and forward-compatible: 1.12.x reads all the data correctly but conservatively declines to use the min/max stats it can't interpret. The only "cost" is lost statistics-based pruning for that column — which is the intended, correct behavior, since 1.12.x has no idea how IEEE 754 total order sorts those values and using them under the wrong ordering assumption could produce incorrect query results.

The tricky part is that we don't know how other readers in the wild are handling this, and we want to avoid triggering NPE's (or any other language equivalent).

[wgtmac]

@Jiayi-Wang-db Sorry I don't understand your concern. Old readers are not aware of nan_count and they just have these assumptions of type_defined_order: https://github.com/apache/parquet-format/blob/apache-parquet-format-2.11.0/src/main/thrift/parquet.thrift#L1084-L1098

Because the sorting order is not specified properly for floating
point values (relations vs. total ordering) the following
compatibility rules should be applied when reading statistics:
- If the min is a NaN, it should be ignored.
- If the max is a NaN, it should be ignored.
- If the min is +0, the row group may contain -0 values as well.
- If the max is -0, the row group may contain +0 values as well.
- When looking for NaN values, min and max should be ignored.
When writing statistics the following rules should be followed:
- NaNs should not be written to min or max statistics fields.
- If the computed max value is zero (whether negative or positive),
`+0.0` should be written into the max statistics field.
- If the computed min value is zero (whether negative or positive),
`-0.0` should be written into the min statistics field.

So for columns containing NaN values, either their stats have finite bounds (safe to be used by filter), or contain NaN bounds (should be ignored by filter). If they are implemented correctly then we are safe, if not then that's the problem of old readers not this new version. Please correct me if I was wrong, this change only emits new nan_count but does not change the behavior that NaN values should not be written to min/max bounds in any case for type_defined_order.

[wgtmac]

Digging into this further, I found that old parquet-java writers did write NaN into the bounds, despite the spec guidance. Old readers then discarded the min/max whenever either bound was NaN, which made this safe in practice.

The new writer changes that behavior for TYPE_ORDER: it writes finite bounds computed from the non-NaN values together with nan_count. An old reader ignores nan_count but accepts the finite bounds. This can lead to incorrect pruning for predicates such as eq(x, NaN), in(... NaN ...), notEq(x, finite), and gt/gtEq(x, finite).
For example, [1.0, NaN] now produces min = max = 1.0. An old parquet-java reader may drop the row group for gt(x, 1.0), even though NaN satisfies that predicate according to its comparator.

Strictly speaking, the old readers have a latent bug because they are not conservative enough for spec-conforming TYPE_ORDER statistics. Note that the spec explicitly states that When looking for NaN values, min and max should be ignored. However, this change exposes that bug for files produced by parquet-java, so I agree that it is also a backward-compatibility issue for the new writer. I'm not sure if it is worth fixing provided that the backward-compatible behavior does not look right.

[Jiayi-Wang-db]

Yes, this is exactly my concern.

Since old readers would simply ignore statistics with an unknown sort order (thanks, @Fokko for checking that) and remaining the old sort order could cause correctness issues for old readers, I think moving forward with writing the new sort order is the right approach.

[wgtmac]

@Jiayi-Wang-db Do you want to submit a PR on this? That said, I think old parquet-java readers still have the problem reading a spec-conforming type_defined_order with mixed NaN values written by other implementations.