Default for arrays via const generics

[MikailBag]

Sorry, I created PR too early :)
It is WIP currently.
UPD: no longer WIP, but blocked on codegen issues.

[rust-highfive]

r? @dtolnay

(rust_highfive has picked a reviewer for you, use r? to override)

[MikailBag]

Well, I think this PR works and can be reviewed.
Two tests are failing, but I don't know why. Their error message does not mention 'Default' at all.

[dtolnay]

I haven't reviewed closely yet, but the generated code looks quite a bit worse than before so I would be concerned about the performance and code size implications of this.

https://rust.godbolt.org/z/3dxGE5

externcrate core;use core::mem::{self,MaybeUninit};use core::ptr;typeT = String;constN:usize = 2;pubfnbefore() -> [T;N]{Default::default()}pubfnafter() -> [T;N]{structWrapper{data:MaybeUninit<[T;N]>,init:usize,}implDropforWrapper{#[inline]fndrop(&mutself){debug_assert!(self.init <= N);let ptr = self.data.as_mut_ptr()as*mutT;let initialized_part = ptr::slice_from_raw_parts_mut(ptr,self.init);unsafe{
ptr::drop_in_place(initialized_part);}}}letmut w = Wrapper{data:MaybeUninit::uninit(),init:0};let array_pointer = w.data.as_mut_ptr()as*mutT;for i in0..N{assert!(N <= isize::MAXasusize);unsafe{let elem_ptr = array_pointer.add(i);
elem_ptr.write(T::default());}
w.init += 1;}let data = unsafe{ w.data.as_ptr().read()};
mem::forget(w);
data
}

example::before:movrax,rdimov qword ptr [rdi],1 vxorps xmm0,xmm0,xmm0vmovups xmmword ptr [rdi+8],xmm0mov qword ptr [rdi+24],1vmovups xmmword ptr [rdi+32],xmm0retexample::after:subrsp,56mov qword ptr [rsp],1 vxorps xmm0,xmm0,xmm0vmovups xmmword ptr [rsp+8],xmm0mov qword ptr [rsp+24],1vmovups xmmword ptr [rsp+32],xmm0movrax,rdimov qword ptr [rsp+48],2movrcx, qword ptr [rsp]mov qword ptr [rdi],rcxvmovupsxmm0, xmmword ptr [rsp+8]vmovups xmmword ptr [rdi+8],xmm0movrcx, qword ptr [rsp+24]mov qword ptr [rdi+24],rcxmovrcx, qword ptr [rsp+16]mov qword ptr [rdi+16],rcxmovrcx, qword ptr [rsp+24]mov qword ptr [rdi+24],rcxvmovupsxmm0, xmmword ptr [rsp+32]vmovups xmmword ptr [rdi+32],xmm0addrsp,56ret

[dtolnay]

@bors try

[bors]

⌛ Trying commit 3d3a7bd02207ae5d2be3bd3ac776b4b0ea15e1e1 with merge 8205f8afb8c63490368efc0d9db81e4e2b5830d8...

[bors]

☀️ Try build successful - checks-actions, checks-azure
Build commit: 8205f8afb8c63490368efc0d9db81e4e2b5830d8 (8205f8afb8c63490368efc0d9db81e4e2b5830d8)

[lcnr]

The test failures seems like a problem with winnowing. The bound in default_array has two candidates now

[T, 0] and [T; N]: NonZero, we then winnow this down to just [T; 0] which does not implement Foo.
This seems fairly interesting... will try this out locally

[kpp]

The previous version did not have inline attr

[lcnr]

Minimized:

fndefault_array<T,constN:usize>() -> [T;N]where[T;N]:Default{Default::default()}fnmain(){let _:[u8;4] = default_array();}

[shepmaster]

This is effectively implementing FromIterator for arrays. I think extracting this making it reusable across other places would make sense.

[MikailBag]

Well, I imagine something like:

// in core::mempub(crate)structPartialBuffer<T,constN>{buf:[MaybeUninit<T>;N],init:usize}implPartialBuffer<_,_>{pub(crate)fnnew() -> Self{..}pub(crate)unsafefnpush(&mutself,value:T){..}// or maybe pub(crate) unsafe fn next_place(&mut self) -> &mut MaybeUninit<T> {..} to reduce copyingpub(crate)unsafefnassume_init(self) -> [T;N]{..}pub(crate)fnget_ref(&self) -> &[T]{..}pub(crate)fnget_mut(&mutself) -> &mut[T]{..}}

[shepmaster]

Why not make the guard type itself hold the MaybeUninit?

[MikailBag]

Previous version did so.
I tried keeping MaybeUninit outside of Guard to see whether it simplifies code and whether it optimizes generated ASM.

[dtolnay]

Newest version of the code, in godbolt: https://rust.godbolt.org/z/Pc6175.

example::before:movrax,rdimov qword ptr [rdi],1 vxorps xmm0,xmm0,xmm0vmovups xmmword ptr [rdi+8],xmm0mov qword ptr [rdi+24],1vmovups xmmword ptr [rdi+32],xmm0retexample::after:subrsp,48movrax,rdimov qword ptr [rsp],1 vxorps xmm0,xmm0,xmm0vmovups xmmword ptr [rsp+8],xmm0mov qword ptr [rsp+24],1vmovups xmmword ptr [rsp+32],xmm0movrcx, qword ptr [rsp]mov qword ptr [rdi],rcxvmovupsxmm0, xmmword ptr [rsp+8]vmovups xmmword ptr [rdi+8],xmm0movrcx, qword ptr [rsp+24]mov qword ptr [rdi+24],rcxmovrcx, qword ptr [rsp+16]mov qword ptr [rdi+16],rcxmovrcx, qword ptr [rsp+24]mov qword ptr [rdi+24],rcxvmovupsxmm0, xmmword ptr [rsp+32]vmovups xmmword ptr [rdi+32],xmm0addrsp,48ret

It appears to me that the difference is entirely a consequence of "return value optimization" not kicking in for the MaybeUninit for some reason, which seems fixable in the compiler. Changing the following two lines fixes the performance issue and gets exactly the same machine code as before (but isn't correct).

- let mut data: MaybeUninit<[T; N]> = MaybeUninit::uninit();+ let mut data: [T; N] = unsafe { mem::uninitialized() }; // unsound- unsafe { data.assume_init() }+ data

example::before:movrax,rdimov qword ptr [rdi],1 vxorps xmm0,xmm0,xmm0vmovups xmmword ptr [rdi+8],xmm0mov qword ptr [rdi+24],1vmovups xmmword ptr [rdi+32],xmm0retexample::after:movrax,rdimov qword ptr [rdi],1 vxorps xmm0,xmm0,xmm0vmovups xmmword ptr [rdi+8],xmm0mov qword ptr [rdi+24],1vmovups xmmword ptr [rdi+32],xmm0ret

[lcnr]

@MikailBag thanks for this PR, we ended up talking about this implementation and the winnowing problem on zulip: https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/good.20example.20for.20const.20generics/near/203642562
and I ended up with a solution which ended up evading that problem entirely about which I personally feel a lot more comfortable.

Do you want to try and incorporate that impl in your PR or should I open a new one for it?

/// A trait implemented by all arrays which are either empty or contain a type implementing `Default`.#[unstable(feature = "array_default_internals", reason = "implementation detail", issue = "none")]#[marker]pubtraitArrayDefault{}#[unstable(feature = "array_default_internals", reason = "implementation detail", issue = "none")]impl<T>ArrayDefaultfor[T;0]{}#[unstable(feature = "array_default_internals", reason = "implementation detail", issue = "none")]impl<T:Default,constN:usize>ArrayDefaultfor[T;N]{}traitDefaultHack{fndefault_hack() -> Self;}impl<T>DefaultHackforT{defaultfndefault_hack() -> Self{unreachable!();}}impl<T:Default>DefaultHackforT{fndefault_hack() -> Self{Default::default()}}#[stable(since = "1.4.0", feature = "array_default")]impl<T,constN:usize>Defaultfor[T;N]where[T;N]:ArrayDefault{// ...}

[MikailBag]

@dtolnay thanks for your wonderful investigation!
I know pretty little about LLVM, codegen and assembly, so I don't think I can solve 74267 mysef.
I see following ways forward:

1. Wait until MaybeUninit performance problems are solved.
2. Ignore performance regression.
3. Combine macros with const generics: for N <= 32 generate impls via macros, and use const generics for all other lengthes.
   That way, we have no performance regressions.

[dtolnay]

3 seems best if possible.

[shepmaster]

3. Combine macros with const generics: for N <= 32 generate impls via macros, and use const generics for all other lengthes.

If we are going to add back the LengthAtMost32 bound, then the const generic version will never be used 🙂

[MikailBag]

@dtolnay I implemented third approach, tests pass (wow 😄 ).
@shepmaster I marked const-generics-based impl as unstable. Should it work?

[lcnr]

I do not feel comfortable with the current approach.

• this instantly stabilizes the const generic Default impls without giving them the imo required test coverage as most of the existing code was written with the 32 elements limitation, therefore not using the new impl.
• it uses negative impls to implement (something even stronger than) specialization, which we aren't doing anywhere else afaik. This previously resulted in Default for arrays via const generics #74254 (comment) failing and in general results in strange type inference issues. The current implementation fails for:

#![feature(const_generics)]fnbreak_me<T,constN:usize>() -> [T;N]where[T;N]:Default{Default::default()}fnmain(){let _:[u8;35] = break_me();}

I personally would like to try and perf what happens if we just use const generics for all impls and
use the approach outlined in #74254 (comment)

@shepmaster I marked const-generics-based impl as unstable. Should it work?

While these impls are bound on an unstable trait, they are still instantly stable.

[lcnr]

The current impl also fails for

#![feature(const_generics)]structFoo<constN:usize>([u8;N]);impl<constN:usize>Foo<N>{fnnew() -> Self{Foo(Default::default())}}fnmain(){}

with

error[E0277]: the trait bound `[u8; N]: std::default::Default` is not satisfied
--> /home/lcnr/rust7/src/test/ui/__check/fk.rs:7:13
|
LL | Foo(Default::default())
| ^^^^^^^^^^^^^^^^^^ the trait `std::default::Default` is not implemented for `[u8; N]`
|
= help: the following implementations were found:
<&[T] as std::default::Default>
<&mut [T] as std::default::Default>
<[T; 0] as std::default::Default>
<[T; 10] as std::default::Default>
and 32 others
= note: required by `std::default::Default::default`
error: aborting due to previous error; 1 warning emitted

[MikailBag]

@lcnr thanks, it's really serious argument against this combined "macro + const-generics" approach, especially the last example.

Let me summarize how I see situation.
Only unbroken (i.e. playing well with compiler, generics and so on) approach is marker trait.
Only efficient approach is one that proposed now (using macros for small lengths).

[shepmaster]

1. Wait until MaybeUninit performance problems are solved.

I feel like this is the best final solution, and it would probably benefit multiple other cases.

[MikailBag]

Unfortunately, PR does not play well with min_specialization:

error: cannot specialize on trait `Default`
--> library/core/src/array/mod.rs:392:5
|
392 | / impl<T: Default> DefaultHack for T {
393 | | fn default_hack() -> Self {
394 | | Default::default()
395 | | }
396 | | }
| |_____^

I currently changed the feature from min_specialization to specialization.

[RalfJung]

I currently changed the feature from min_specialization to specialization.

Please do not. specialization is unsound. Some people worked hard to remove all uses of specialization from rustc and the stdlib.

[MikailBag]

Yeah, I understand that using non-min specialization is only workaround.

I have zero experience with trait system, so I can be wrong here.
As I see, min_specialization disallows any trait which is not marked with special attribute.
Docs say:

// We allow specializing on explicitly marked traits with no associated// items.

And Default does not have associated types.

That's why I now think I should add this specialization marker to Default and min_specialization should work again.
I'll try this just now.

UPD: no, build fails:

 error: cannot specialize on trait `ArrayDefault`
--> library/core/src/array/mod.rs:397:5
|
397 | / impl<T, const N: usize> Default for [T; N]
398 | | where
399 | | [T; N]: ArrayDefault,
400 | | {
... |
403 | | }
404 | | }
| |_____^
error: specializing impl repeats parameter `T`
--> library/core/src/lazy.rs:374:1
|
374 | / impl<T: Default> Default for Lazy<T> {
375 | | /// Creates a new lazy value using `Default` as the initializing function.
376 | | fn default() -> Lazy<T> {
377 | | Lazy::new(T::default)
378 | | }
379 | | }
| |_^

[RalfJung]

It is backwards-incompatible to add this marker to an existing trait. And it is unsound to do specialization on a trait unless the trait has that marker -- a trait that is used for specializing other traits needs to pass some stricter checks, which most existing traits will not pass. I don't know of a good way out of this, but I am also not a specialization expert. Cc @matthewjasper

[Dylan-DPC-zz]

This is backwards incompatible and looks highly likely won't be merged as is. If you find a different way to solve this, it is better to create a new PR. Thanks for taking the time to contribute

[MikailBag]

@Dylan-DPC can you please describe in what way this PR breaks compatibility?
I know there is a problem that PR currently uses an unsound specialization feature, but it is not backcompat concern AFAIU.

I don't know if the desired behavior can be achieved with min_specialization. I hope that it is possible: specialization usage here does not look too complex. I was waiting for WG-traits input on this PR.

[nbdd0121]

Can't we just introduce a new MarkerDefault trait with #[rustc_unsafe_specialization_marker]? It's similar to the MarkerEq trait used in slice/cmp.rs:

#[rustc_unsafe_specialization_marker]traitMarkerDefault:Default{}impl<T:Default>MarkerDefaultforT{}traitDefaultHack{fndefault_hack() -> Self;}impl<T>DefaultHackforT{defaultfndefault_hack() -> T{unreachable!();}}impl<T:MarkerDefault>DefaultHackforT{fndefault_hack() -> T{Default::default()}}

[RalfJung]

We should IMO not introduce more rustc_unsafe_specialization_marker until we know how to make them safe.

[crlf0710]

There's a specialization-free approach, waiting for the feature const_evaluatable_checked. See sample code in #86259 .