Bugs in miloyip/nativejson-benchmark: roundtrips

[nlohmann]

I checked the library with the latest https://github.com/miloyip/nativejson-benchmark benchmark suite. There are several errors in the round trip tests:

• Roundtrip,Nlohmann (C++11),roundtrip20,false
• Roundtrip,Nlohmann (C++11),roundtrip21,false
• Roundtrip,Nlohmann (C++11),roundtrip24,false
• Roundtrip,Nlohmann (C++11),roundtrip25,false
• Roundtrip,Nlohmann (C++11),roundtrip26,false
• Roundtrip,Nlohmann (C++11),roundtrip27,false

These are the values:

• [0.0]
• [-0.0]
• [5e-324]
• [2.225073858507201e-308]
• [2.2250738585072014e-308]
• [1.7976931348623157e308]

I will have a deeper look at this later. I just wanted to paste the results here before I forget.

[nlohmann]

I wrote a little tool for the roundtrips:

#include <json.hpp>

using nlohmann::json;

int main()
{
    json j;
    std::cin >> j;
    std::cout << j << std::endl;
}

Here are the results:

• [0.0] -> [0]
• [-0.0] -> [0]
• [5e-324] -> [4.94065645841247e-324]
• [2.225073858507201e-308] -> [2.2250738585072e-308]
• [2.2250738585072014e-308] -> [2.2250738585072e-308]
• [1.7976931348623157e308] -> [1.79769313486232e+308]

Indeed, the output differs from the input.

[nlohmann]

The first two items ([0.0] and [-0.0]) are parsed and stored as integer 0, because no precision is lost. As JSON does not define an internal representation of numbers and the value (zero) is the same, I think it's rather nitpicky to treat the output as an error.

[nlohmann]

FYI: The roundtrips are already part of the unit tests (test case ""compliance tests from nativejson-benchmark", section "roundtrip"), where test case 13, 18-21, and 24-27 are skipped.

[twelsby]

Most of these are silly tests in my opinion and seem pretty contrived.

The exception is -0. There is an argument for saying that -0 should remain a float. This is because while 0 = -0, 1/0 != 1/-0 (LHS is positive infinity and RHS is negative infinity). The 0 case is not so much of a problem because if the user expects a float then it will be cast to 0.0, however -0 would be stored as 0 and cast to 0.0. In rare cases this could cause unexpected behaviour.

Here is what causes them to break:
roundtrip13.json (-1234567890123456789) - this exceeds what can be stored in a double without loss of precision so will not be rendered accurately under the old parse method but will be under the 2.0.0 parse method as it will end up being stored as a int64_t with the default basic_json.

roundtrip18.json (1234567890123456789) - as above except that it will be stored as a uint64_t in version 2.0.0.

roundtrip19.json (9223372036854775807) - as above.

roundtrip20.json (0.0) - cast to integer so ends up dumping as 0 not 0.0. This could be fixed by dropping the cast to integer. In 2.0.0 an attempt is made to parse as integers first so dropping the cast would only affect floating point numbers that are expressed in the form of floating point numbers but are actually integers e.g 1.00, 123e4 etc. It could be argued that if someone has gone the effort of writing a number in the form 1.00 then it is probably intended to be a float so this might be the right way to go anyway. Personally I would like to see us go down that track.

roundtrip21.json (-0.0) - similar to above. A special case would be needed to fix it.

roundtrip24.json (5e-324) - this number is right at the limit of the minimum subnormal range (for double's) and cannot be represented more accurately using a double. Even a GCC long double cannot accurately represent this number although it gets much closer - so there isn't really a good use for this in the real world.

roundtrip25.json (2.225073858507201e-308) - this number is below the minimum normal range so is represented using the subnormal range with loss of precision (it is the loss of precision that causes the error in this case). A GCC long double might work but honestly this test is silly.

roundtrip26.json (2.2250738585072014e-308) - this number is exactly the minimum normal range so can be represented exactly in a double. If the fix suggested in #186 is used then it will parse ok however it won't dump correctly because it needs 17 digits and dump() sets the precision based on std::numeric_limits::digits10, which is 15. The problem is that double can represent 15-17 significant figures depending on the number but std::numeric_limits::digits10 takes a conservative approach. This is a fair test but probably not easily solvable.

roundtrip27.json (1.7976931348623157e308) - as above.

[nlohmann]

Update:

• [0.0] -> [0]
• [-0.0] -> [-0.0]
• [5e-324] -> [4.94065645841247e-324]
• [2.225073858507201e-308] -> [2.2250738585072e-308]
• [2.2250738585072014e-308] -> [2.2250738585072e-308]
• [1.7976931348623157e308] -> [179769313486231570814527423731704356798070567525844996598917476803157260780028538760589558632766878171540458953514382464234321326889464182768467546703537516986049910576551282076245490090389328944075868508455133942304583236903222948165808559332123348274797826204144723168738177180919299881250404026184124858368.0]

[nlohmann]

I reran the benchmark:

Fixed:

• Roundtrip,Nlohmann (C++11),roundtrip21,true [-0.0]

New errors:

• Roundtrip,Nlohmann (C++11),roundtrip13,false [-1234567890123456789]
• Roundtrip,Nlohmann (C++11),roundtrip18,false [1234567890123456789]
• Roundtrip,Nlohmann (C++11),roundtrip19,false [9223372036854775807]

Unchanged:

• Roundtrip,Nlohmann (C++11),roundtrip20,false [0.0]
• Roundtrip,Nlohmann (C++11),roundtrip24,false [5e-324]
• Roundtrip,Nlohmann (C++11),roundtrip25,false [2.225073858507201e-308]
• Roundtrip,Nlohmann (C++11),roundtrip26,false [2.2250738585072014e-308]
• Roundtrip,Nlohmann (C++11),roundtrip27,false [1.7976931348623157e308]

[nlohmann]

• example 13, input -1234567890123456789 output -1234567890123456768
• example 18, input 1234567890123456789 output 1234567890123456768
• example 19, input 9223372036854775807 output 9223372036854775808.0

[twelsby]

Actually I don't think these are new - or at least they shouldn't be. I couldn't get the benchmark to successfully run so I uncommented the above tests in unit.cpp and encountered them, which is why I mentioned them above. They will be automatically fixed by the version 2.0.0 get_number() method as they are a consequence of the 1.0.1 method of parsing as double, which is incapable of holding them without loss of precision. The unit.cpp in #193 has these tests re-enabled and they pass

The reason the bottom one now has a trailing .0 is that the top two are able to fit in a int64_t so they get dumped as integers, although they still lose precision by passing through a double, while the bottom one can only be stored as a double so it gets dumped as a floating point.

To summarise where all of these sit at the moment: in 1.0.1 roundtrip21 is fixed (and the relevant unit test uncommented) but everything else is broken, while in 2.0.0 rountrip13, rountrip18, roundtrip19, and roundtrip20 are all fixed (and the relevant unit tests uncommented).

roundtrip24, roundtrip25 and roundtrip27 are (in my opinion) fundamentally unable to be passed by any library that relies on double's as the floating point data type, because this datatype lacks the necessary accuracy with such small values. I would be interested to see how rapidJSON does it but I suspect they keep the original string representation (would explain why their dumping is so fast).

roundtrip26 could be made to pass but the difficulty is that it requires 17 digit precision and you can't guarantee roundtrips for all numbers if you output at 17 digits. It would be necessary to somehow identify which numbers can support roundtrips with 17 digits and this would be very difficult to do efficiently I think. The worst case scenario is that you would have to do trial round trips at three different levels of precision (15, 16 and 17) and that would only work for double's. There may be a way to do this more directly but I don't even know where to begin - and if such a method did exist it would probably be in the standard library.

[nlohmann]

Now all but roundtrip 24-27 work. I'll rerun the benchmarks just to make sure it also works in the setting of https://github.com/miloyip/nativejson-benchmark.

[nlohmann]

Executing the benchmark, 4 errors remain. It is strange, however, that several other libraries seem to be able to pass the tests.