Drop support for Base.tail()

[JamesWrigley]

This causes invalidations and should not be needed anymore by ChainRules.jl since the rule that uses it: #576 (comment)
Was deleted in JuliaDiff/ChainRules.jl#680.

I ran this on the ChainRules.jl tests and saw a ton of failures locally (also seen in CI), but none that seem related to this change. Disclaimer: I have almost no understanding of this kind of thing 🙈

Fixes#576.

Fixes these invalidations seen in CurveFit.jl:

 inserting tail(t::ChainRulesCore.Tangent{<:NamedTuple{<:Any, <:Tuple{}}}) @ ChainRulesCore ~/.julia/packages/ChainRulesCore/Vsbj9/src/tangent_types/structural_tangent.jl:378 invalidated: mt_backedges:1: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_isdone(::Tuple{Tuple{Int64}, Vararg{Any}}, ::Any) (0 children) 2: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple{Tuple{Int64}, Vararg{Any}}, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Missing) (0 children) 3: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple{Tuple{Int64}, Vararg{Any}}, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Bool) (0 children) 4: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_interleave(::NamedTuple, ::Any, ::Tuple{Bool, Vararg{Any}}) (0 children) 5: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_interleave(::Any, ::Any, ::Any) (0 children) 6: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_isdone(::Tuple{NTuple{4, Symbol}, Vararg{Any}}, ::Any) (0 children) 7: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple{NTuple{4, Symbol}, Vararg{Any}}, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Missing) (0 children) 8: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Any, ::Any, ::Tuple{Any, Vararg{Any}}, ::Bool) (0 children) 9: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple{NTuple{4, Symbol}, Vararg{Any}}, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Bool) (0 children) 10: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_isdone(::Tuple{Tuple{Symbol}, Vararg{Any}}, ::Any) (0 children) 11: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple{Tuple{Symbol}, Vararg{Any}}, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Missing) (0 children) 12: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Any, ::Any, ::Tuple{Any, Vararg{Any}}, ::Bool) (0 children) 13: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple{Tuple{Symbol}, Vararg{Any}}, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Bool) (0 children) 14: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_isdone(::Tuple{Tuple{Symbol, Symbol, Symbol}, Vararg{Any}}, ::Any) (0 children) 15: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple{Tuple{Symbol, Symbol, Symbol}, Vararg{Any}}, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Missing) (0 children) 16: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Any, ::Any, ::Tuple{Any, Vararg{Any}}, ::Bool) (0 children) 17: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple{Tuple{Symbol, Symbol, Symbol}, Vararg{Any}}, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Bool) (0 children) 18: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_isdone(::Tuple{Tuple{Symbol, Symbol}, Vararg{Any}}, ::Any) (0 children) 19: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple{Tuple{Symbol, Symbol}, Vararg{Any}}, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Missing) (0 children) 20: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Any, ::Any, ::Tuple{Any, Vararg{Any}}, ::Bool) (0 children) 21: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple{Tuple{Symbol, Symbol}, Vararg{Any}}, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Bool) (0 children) 22: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_isdone(::Tuple{Tuple{Int64, Int64}, Vararg{Any}}, ::Any) (0 children) 23: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple{Tuple{Int64, Int64}, Vararg{Any}}, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Missing) (0 children) 24: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Any, ::Any, ::Tuple{Any, Vararg{Any}}, ::Bool) (0 children) 25: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple{Tuple{Int64, Int64}, Vararg{Any}}, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Bool) (0 children) 26: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_isdone(::Tuple{Tuple{}, Vararg{Tuple{}}}, ::Any) (0 children) 27: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::Base.LogicalIndex, ::Any) (0 children) 28: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::AbstractRange{Float64}, ::Any) (0 children) 29: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::AbstractRange{String}, ::Any) (1 children) 30: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_interleave(::NamedTuple, ::Any, ::Any) (1 children) 31: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_interleave(::Any, ::Any, ::NamedTuple) (1 children) 32: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::Base.Iterators.Enumerate{Vector{Any}}, ::Any) (2 children) 33: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::AbstractRange{Compiler.MethodMatchInfo}, ::Any) (4 children) 34: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::AbstractRange{Compiler.ApplyCallInfo}, ::Any) (4 children) 35: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::AbstractRange{Union{Nothing, Compiler.ConstResult}}, ::Any) (4 children) 36: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::AbstractRange{Union{Nothing, Compiler.AbstractIterationInfo}}, ::Any) (4 children) 37: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::AbstractRange{Compiler.CallMeta}, ::Any) (4 children) 38: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::AbstractRange{Compiler.NewPhiCNode2}, ::Any) (4 children) 39: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::AbstractRange{Tuple{LineNumberNode, Expr}}, ::Any) (4 children) 40: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_isdone(::Tuple{Vararg{Tuple{}}}, ::Any) (4 children) 41: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::Base.Iterators.Enumerate, ::Any) (5 children) 42: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_interleave(::NamedTuple, ::NamedTuple, ::Any) (5 children) 43: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::NamedTuple, ::Any, ::Tuple{Bool, Vararg{Any}}, ::Bool) (5 children) 44: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::NamedTuple, ::Any, ::Tuple{Any, Vararg{Any}}, ::Bool) (5 children) 45: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Broadcast.check_broadcast_shape(::Any, ::Tuple{Base.OneTo{Int64}}) (5 children) 46: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foraxes(::Base.ReinterpretArray{Float64, 1, S, A, true}where A<:Union{SubArray{S, N, A, I, true} where {N, A<:DenseArray, I<:Union{Tuple{Vararg{Real}}, Tuple{AbstractUnitRange, Vararg{Any}}}}, DenseArray{S}}) where S (5 children)
47: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::SubArray{T, 1}where T, ::Tuple{Any, Vararg{Any}}) (6 children) 48: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Any, ::NamedTuple, ::Tuple{Bool, Vararg{Any}}, ::Bool) (10 children) 49: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance forsize(::Base.ReinterpretArray{Float64, N, S, A, true}where {N, A<:Union{SubArray{S, N, A, I, true} where {N, A<:DenseArray, I<:Union{Tuple{Vararg{Real}}, Tuple{AbstractUnitRange, Vararg{Any}}}}, DenseArray{S}}}) where S (10 child
ren) 50: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Tuple, ::Any, ::Tuple{Missing, Vararg{Any}}, ::Missing) (12 children) 51: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::SubArray{T, 1}where T, ::Tuple{Any}) (15 children) 52: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base._overflowind(::Any, ::Any, ::NTuple{N, Int64}where N) (15 children) 53: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance foriterate(::AbstractRange{Any}, ::Any) (25 children) 54: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_isdone(::Tuple, ::Any) (38 children) 55: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base.Iterators._zip_iterate_some(::Any, ::NamedTuple, ::Tuple{Any, Vararg{Any}}, ::Bool) (50 children) 56: signature Tuple{typeof(Base.tail), Any} triggered MethodInstance for Base._cshp(::Int64, ::Tuple{Bool}, ::Tuple{Int64}, ::Any) (66 children) 

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.42%. Comparing base (0b2ebfe) to head (6c241cf).

Additional details and impacted files

@@ Coverage Diff @@## main #703 +/- ##
==========================================
+ Coverage 93.38% 93.42% +0.03% 
==========================================
Files 15 15 Lines 998 988 -10 ==========================================
- Hits 932 923 -9 + Misses 66 65 -1 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[JamesWrigley]

The number of failures is a bit disturbing, but they don't seem related to this PR (and they also show up on other PRs).

[devmotion]

Sorry, I mis-clicked. IMO it would be great if we could remove these definitions. There are unfortunately preexisting test failures IIRC. I'll have a closer look tomorrow.

[devmotion]

This looks fine to me. The test errors seem unrelated, in many cases caused by Aqua tests and in particular without any reference to tail.

It would be good to fix the doctest failures though, such that we're able to build docs for the upcoming release. Can you update the doctests (see https://documenter.juliadocs.org/stable/man/doctests/#Fixing-Outdated-Doctests)?

[JamesWrigley]

Ah yeah sure, fixed them in eb83113. BTW one of them was using rand(), I replaced that with a constant to avoid the tests breaking again if the RNG changes.

[devmotion]

I'd suggest

Suggested change

	julia> t =@thunk(3)
	Thunk(var"#4#5"())
	Thunk(var"#2#3"())
	julia> t =@thunk(3);

or a doctest filter. But I think removing the output is simpler.

[JamesWrigley]

Yeah good idea, fixed in c2e51fd.

[devmotion]

Can you update the version number?

[JamesWrigley]

Sure, done in 10e0ec5.

[devmotion]

This is a breaking change - it will definitely break older ChainRules releases and it might break packages that define rules using Base.tail.

Suggested change

	version = "1.26.1"
	version = "2.0.0"

[JamesWrigley]

Ah ok, fixed in 09cb407.

[JamesWrigley]

I also removed the compat bound in docs/Project.toml so that the docs will build. There's quite a few more failing tests now, I guess because downstream packages aren't compatible anymore.

[oxinabox]

Making this a breaking change is really annoying though.
Can we see if anyone is using it?
I suspect they were not, since Base.tail isn't part of Base's public API.

Easy way to see if people are using it is to just release it with this as a nonbreaking change,
and then if people complain tag a bug fix that reverts the change.

Like it will break old ChainRules.jl releases, but only pretty old ones that there is no particular reason to be using, unless they are locked in a Manifest.toml.
But in that case the version of ChainRulesCore.jl is also going to be locked to an old version.

[devmotion]

It's annoying but it's the only way to avoid breaking existing releases, of ChainRules and possibly other packages.

SciML is making non-breaking releases that break existing versions of SciML and non-SciML packages quite regularly in my experience, and I think that's far more annoying than making a breaking release. Even if people would complain and a subsequent bug fix release would be made, there would still be a combination of package versions in the registry that is officially declared to be compatible but in fact isn't.

[oxinabox]

Even if people would complain and a subsequent bug fix release would be made, there would still be a combination of package versions in the registry that is officially declared to be compatible but in fact isn't.

This is why Pkg should refuse to install things other than the latest patch release of a package (at least post 1.0.0)
But can imitate that by yanking prior ones.

I guess arguably Base.tail is part of the public API.
It wasn't when I wrote those overloads.
But it is now documented as public.

At the moment there is 283 direct reverse dependencies.
That is a lot to make do a update.
I guess dependabot will catch them though.

[JamesWrigley]

Am I correct in thinking that all those direct dependents could use package extensions on ChainRules.jl instead?

[oxinabox]

you are not correct.
ChainRules.jl is not something packages writing rules should depend on.
It's huge. It is just a collection of rules for Base + stdlibs.
Not a tool for rules authors.
Only AD packages should depend on that.
Admittedly that is less of a consideration with package extensions now.

In general I would just be shocked to find that any of them depend on this feature.
Calling tail on a ChainRulesCore Tangent is just pretty niche,
expecially if you are writing rules for things other than Tuples.

[JamesWrigley]

(bump)

[JamesWrigley]

Bump, just wondering when v2 is planned? No worries if there are some other things planned for v2 that this would have to wait for.

[ChrisRackauckas]

Yeah these invalidations seem to show up everywhere.

[JamesWrigley]

(bump)

[JamesWrigley]

(bump)

[ChrisRackauckas]

I think to get this in, it would be best to have a PR that first fixes tests for master, then do this. But yes, this is quite required since this invalidation is pretty deadly.

[JamesWrigley]

I opened some PRs to add compat bounds. Could you unblock CI for JuliaArrays/StaticArrays.jl#1339? I'm looking into the FiniteDifferences failures (can't be related since this PR isn't out yet, but still).

[JamesWrigley]

Ok, with a couple more fixes (d59d0ea) CI is in the green again. Caveat is that all of the integration tests are skipped because they don't have compat with v2, we'll have to go through all those after the release.

[ChrisRackauckas]

The fixing should be a separate PR from the breaking PR.

[JamesWrigley]

Why? They're separate commits.

[ChrisRackauckas]

We should probably put the release out which fixes tests as the last v1 release, and that should be reviewed and merged. That's completely separate from a v2 breaking change.

[JamesWrigley]

Ok, I made #706.

[JamesWrigley]

Getting there 😅 Rebased the PR now. We'll want to revert 24f913a once JuliaDiff/FiniteDifferences.jl#248 and JuliaArrays/StaticArrays.jl#1339 are released.

[ChrisRackauckas]

Checks all my boxes.