Call Random.seed! to avoid inconsistent test failure

[dlfivefifty]

PR tests sometimes fail, I believe due to floating point errors. See e.g.

#541

This should hopefully fix it.

[codecov-commenter]

Codecov Report

Merging #544 (47792a2) into master (bb85ea2) will increase coverage by 7.83%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #544 +/- ##
==========================================
+ Coverage 76.82% 84.65% +7.83% 
==========================================
Files 9 9 Lines 604 834 +230 ==========================================
+ Hits 464 706 +242 + Misses 140 128 -12 

Impacted Files	Coverage Δ
src/jacobian.jl	99.29% <0.00%> (+0.34%)	⬆️
src/hessian.jl	97.43% <0.00%> (+1.60%)	⬆️
src/gradient.jl	98.88% <0.00%> (+1.87%)	⬆️
src/apiutils.jl	100.00% <0.00%> (+3.44%)	⬆️
src/derivative.jl	100.00% <0.00%> (+5.00%)	⬆️
src/prelude.jl	36.36% <0.00%> (+5.59%)	⬆️
src/partials.jl	84.34% <0.00%> (+7.60%)	⬆️
src/dual.jl	72.18% <0.00%> (+9.33%)	⬆️
src/config.jl	87.03% <0.00%> (+25.49%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bb85ea2...47792a2. Read the comment docs.

[KristofferC]

While at it, perhaps moving to StableRNGs.jl would make sense?

[dlfivefifty]

But then we’d have to catch every rand call and pass the rng? Seems more trouble than it’s worth since we test on different Julia versions and can change the seed if a future version breaks the tests

[mcabbott]

Is it just one or two tests? If it is, it might be worth figuring out why, instead of avoiding such cases.

In https://github.com/JuliaDiff/ForwardDiff.jl/runs/3340080509?check_suite_focus=true the failure is this, which looks a bit familiar:

450
Error During Test at /home/runner/work/ForwardDiff.jl/ForwardDiff.jl/test/DualTest.jl:400
451
Test threw exception
452
Expression: dual_isapprox(NESTED_FDNUM ^ PRIMAL, exp(PRIMAL * log(NESTED_FDNUM)))
453
DomainError with -2.98738277122415e18:
454
sqrt will only return a complex result if called with a complex argument. Try sqrt(Complex(x)).
455
Stacktrace:
456
[1] throw_complex_domainerror(f::Symbol, x::Float64)
457
@ Base.Math ./math.jl:33
458
[2] sqrt
459
@ ./math.jl:582 [inlined]
460
[3] sqrt
461
@ ~/work/ForwardDiff.jl/ForwardDiff.jl/src/dual.jl:203 [inlined]
462
[4] generic_norm2(x::ForwardDiff.Partials{3, ForwardDiff.Dual{Main.DualTest.TestTag(), Int64, 4}})
463
@ LinearAlgebra /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/generic.jl:509
464
[5] norm2
465
@ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/generic.jl:558 [inlined]
466
[6] norm(itr::ForwardDiff.Partials{3, ForwardDiff.Dual{Main.DualTest.TestTag(), Int64, 4}}, p::Int64)
467
@ LinearAlgebra /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/generic.jl:627
468
[7] norm
469
@ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/generic.jl:625 [inlined]
470
[8] isapprox(x::ForwardDiff.Partials{3, ForwardDiff.Dual{Main.DualTest.TestTag(), Int64, 4}}, y::ForwardDiff.Partials{3, ForwardDiff.Dual{Main.DualTest.TestTag(), Float64, 4}}; atol::Int64, rtol::Float64, nans::Bool, norm::typeof(LinearAlgebra.norm))
471
@ LinearAlgebra /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/generic.jl:1659
472
[9] isapprox
473
@ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/LinearAlgebra/src/generic.jl:1657 [inlined]
474
[10] dual_isapprox(a::ForwardDiff.Dual{Main.DualTest.TestTag(), ForwardDiff.Dual{Main.DualTest.TestTag(), Int64, 4}, 3}, b::ForwardDiff.Dual{Main.DualTest.TestTag(), ForwardDiff.Dual{Main.DualTest.TestTag(), Float64, 4}, 3})
475
@ Main.DualTest ~/work/ForwardDiff.jl/ForwardDiff.jl/test/DualTest.jl:24

[dlfivefifty]

This bug has been around for months. If it was easy to figure out, someone would have. And it’s most likely not even a bug, just an issue if numbers are large producing slightly higher error than expected in the tests.

In any case, it’s standard practice to seed RNG in tests to avoid these issues

[mcabbott]

If isapprox were returning false I'd be happier to write it off to rounding error, as you say. But this is a bit stranger, and I worry a bit it's telling us something is wrong with ^.

[dlfivefifty]

Probably we need to overload norm(::Partials, p) as its falling back to the generic version. It's unclear why

https://github.com/JuliaLang/julia/blob/49e3aecd5966a2af0b064c0314cd61c1338abc00/stdlib/LinearAlgebra/src/generic.jl#L496

might cause sqrt of a negative number

[odow]

Bump. This is pretty painful: #553 (comment)

[odow]

Fixed in #554