Add DynSized trait (rebase of #44469)

[mikeyhew]

I managed to rebase @plietar's PR #44469, and get all the tests passing.

This unfortunately conflicts with #44917 (by @Zoxc) in a lot of places, so one of us will have to do a rebase after the other gets merged.

r? @pnkfelix since you were reviewing #44469
cc @arielb1

[Zoxc]

Letting auto trait imply ?Move and ?DynSized might be a good idea. It would at least avoid all the changes to the tests.

@nikomatsakis Any chance that bounds on Self will become sound for auto traits in the future?

[mikeyhew]

A couple of interesting points that weren't mentioned in the conversation on #44469:

• DynSized is an implicit supertrait of all traits, including Sized. This is in contrast to Sized, which is only an implicit bound on type parameters and associated types.

• As a result, this messes up auto traits, which may not have super traits. After this PR, all auto traits must be updated to explicitly remove DynSized as a super trait:

  #![feature(optin_builtin_traits, dynsized)]use core::marker::DynSized;unsafe auto traitAuto: ?DynSized{}

• Whenever another implicit supertrait is added, auto traits will break again. This will happen when Immovable types prototype where the Move trait is builtin #44917 lands, which adds an implicit supertrait Move.

Because of the potential for future breakage, it may make sense to special-case auto traits, so that they don't have any implicit supertraits. As long as auto traits can't have items, it hopefully should be possible to add or remove implicit supertraits without breaking. Or alternatively, if auto traits could be allowed to have supertraits, we could allow that, as @Zoxc has just mentioned

[mikeyhew]

cc @bluss

[mikeyhew]

@Zoxc I saw your comment just before I posted mine, just didn't want to rewrite everything. You basically said everything that I was saying, but in way fewer words :)

[mikeyhew]

ugh... src/test/rustdoc/foreigntype.rs is failing, but I don't have that file in my local branch... guess I need to rebase

[liigo]

dynamically (at runtime) sized types are DynSized, that's ok.
but statically (known at compiletime) sized types are DynSized?? that's a bit weird.
Maybe you should rename DynSized. (ClearSized? VS Opaque types)

[mikeyhew]

@liigo anything that is Sized is trivially also DynSized. If you know the size of something at compile time, then you know it a runtime. That's why DynSized is a supertrait of Sized, though that's not immediately obvious from the declaration of Sized because the DynSized bound is implicit

[mikeyhew]

@iigo I'll try to explain that a bit better: DynSized (in this interpretation) doesn't mean "not statically sized", but rather "has a size that can be calculated at run time".

[shepmaster]

Ping from triage @pnkfelix — will you be able to review this shortly?

[nikomatsakis]

So @withoutboats and I were talking and we had what I think is an interesting insight. We were thinking about when one might want T vs T: ?Sized vs T: ?DynSized and how to relate those to the concepts of ownership / borrowing. We came up with the idea that:

• T -- will use this "by value" (no indirection)
• T: ?Sized -- may take ownership (or at least partial ownership), but only use through a "smart pointer" (e.g., Box, Arc)
  • this falls out because, at least with our current allocators, we can't free memory unless we know its size, so you must have size_of_val
• T: ?DynSized -- no ownership (e.g., only used via &T or *mut T)
  • if something is borrowed, you don't need to know its size

I'm not sure yet what this means but I find it to be an interesting correspondence. Are there exceptions?

[mikeyhew]

• T: ?Dynsized can still be owned, just not using Box. People will probably implement Drop for T by calling some C api function, and use a different owning pointer type that just calls ptr::drop_in_place when dropped.
• The distinction between T and T: ?Sized will be clouded a bit if/when we have by-value DST.

[nikomatsakis]

@mikeyhew

just not using Box

Yes; not using the Rust allocator, basically. Not sure how important that is. (Sometimes I regret designing our allocators so that they supply the size of the value being freed, but I guess it's awfully late to try and reverse course there.)

The distinction between T and T: ?Sized will be clouded a bit if/when we have by-value DST.

Yes, true.

[withoutboats]

(Sometimes I regret designing our allocators so that they supply the size of the value being freed, but I guess it's awfully late to try and reverse course there.)

It seems like the use cases for size_of_val diminish significantly without this requirement. I wonder if we couldn't revisit this?

[nikomatsakis]

OK, I feel bad that this PR is sitting here with no viable means to go forward or backward yet. I remain highly nervous about ? traits like ?DynSized. It still feels to me like this is overall a bad direction for Rust to be going in. But I don't want to get into a litany of these concerns (though it makes sense to write those out and work through them). I would rather focus on the fate of this PR.

i.e., do we want to move forward with ?DynSized even in an experimental capacity? Or should we close the PR?

(It seems clear we have to ensure it is 100% backwards compatible; I've kind of lost track of whether that is the case now.)

@mikeyhew if your main motivation is to explore custom DST, I'd rather just explore that directly, while leaving the questions around DynSized slightly unresolved. I continue to feel they are somewhat orthogonal questions.

[mikeyhew]

• @nikomatsakis and @withoutboats, and anyone else with concerns about ?DynSized and other ?-traits, a good place to discuss them is in More implicit bounds (?Sized, ?DynSized, ?Move) rfcs#2255
• Regarding backward compatibility, this PR is in theory backward compatible, but there is still this issue, which will cause some regressions. Back in October, @Zoxc listed crates that were affected by that issue in this comment
• about custom DST and related stuff, I would also like to move ahead and experiment with that. What I can do is open a thread on internals and discuss the design that I have in mind, using the DynSized trait, and then you can poke and prod and we can discuss alternatives

[nikomatsakis]

@mikeyhew

a good place to discuss them is in rust-lang/rfcs#2255

agreed, this PR doesn't feel like the right place!

this PR is in theory backward compatible, but there is still this issue, which will cause some regressions

OK, I see, I remember that now. Thanks for the clarification.

What I can do is open a thread on internals and discuss the design that I have in mind, using the DynSized trait, and then you can poke and prod and we can discuss alternatives

Sounds like a plan.

[kennytm]

FYI I've submitted rust-lang/rfcs#2310 as an alternative.

[nikomatsakis]

@rfcbot fcp close

We discussed this in the @rust-lang/lang meeting today. As a result of that conversation, I'm going to move to close this PR. The general feeling was that the "jury is still out" on what approach to use for handling extern types and the related problems -- as @withoutboats and I have shared here, there is a general concern about the complexity costs of a ?DynSized-based solution (and ?-trait-bounds in general), particularly when weighed against the relative obscurity of extern types (it's a lot of language baggage for a corner case). On the other hand, @mikeyhew and others make a strong case for static guarantees.

Given this uncertainty, I at least am reluctant to land deep changes to the compiler at this time, and in particular not to the trait system (which I think is due for some overhauls). Past experience (e.g., with the ! type) shows that this often has unexpected consequences, causing regressions and other maintenance burden, which then distracts from higher priority goals.

Speaking personally: I do intend to leave a comment on rust-lang/rfcs#2255 "real soon now" trying to spell out some of the specific confusion that can arise from ?DynSized. I am also still intrigued by the possibility of finding an "ownership-based" way of thinking about optional traits along the lines I outlined before, though I don't know if that really leads anywhere.

[rfcbot]

Team member @nikomatsakis has proposed to close this. The next step is review by the rest of the tagged teams:

• @Zoxc
• @arielb1
• @aturon
• @cramertj
• @eddyb
• @estebank
• @jseyfried
• @michaelwoerister
• @nikomatsakis
• @nrc
• @petrochenkov
• @pnkfelix
• @withoutboats

No concerns currently listed.

Once these reviewers reach consensus, this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[shepmaster]

Ping for ticky boxes @Zoxc, @arielb1, @jseyfried, @michaelwoerister!

[nikomatsakis]

I tagged @arielb1 (who should be moved to alumni status, really) as well as @michaelwoerister (who is on vacation afaik) and @jseyfried (who probably also should be moved to alumni status).

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[aturon]

@nikomatsakis you can modify the FCP list here: https://github.com/anp/rfcbot-rs/blob/master/teams.toml

[rfcbot]

The final comment period is now complete.

[nikomatsakis]

OK, going to close. Thanks @mikeyhew

[Ericson2314]

@nikomatsakis The language team decision against ?DynSize looked to me like "jurry is still out, so the fact that the code is in flux (chalk, etc) we use as a tie-breaker". That sounds to me more like "not now" than "no never": is there a possibility for landing this on an experimental basis once the type system overhauls are done?

The problem seems to be ?-traits syntax more than semantics that people want. I say landing this on an experimental basis is best way to find a different syntax / explore downstream problems like custom DSTs. Otherwise too many chained problems are hard to untangle.

[mikeyhew]

@Ericson2314 I agree with you, I think that ultimately having a trait or traits like DynSizedis desired, and that the decision to close this PR wasn't a rejection of the underlying concept, but rather a "let's postpone this for now, and get a better idea of what we want to do first".

I've been meaning to comment on https://internals.rust-lang.org/t/pre-erfc-lets-fix-dsts/6663 and keep the discussion going, and I still think that is the best way forward – we should get an eRFC merged and start working on things experimentally.

[Ericson2314]

@mikeyhew See the end of #43467 . Looks like DynSized is off the table for now, but hopefully it could come back as a non-default bound with {size,align}_of_bound being deprecated.