Make function pointer types look like borrowed pointer types for forwards compatability

[Ericson2314]

Rendered

[glaebhoerl]

Broken link, should be e.g.: [custom pointer types][1].

[Ericson2314]

In this case I wanted an actual footnote. But maybe there is a way I can make that clearer / look better.

[glaebhoerl]

Oh right. There's usually a way to do superscripts but I think it's nonstandard, foo^1 on reddit and foo<sup>1</sup> on GH. (let's^see... yup.)

[Ericson2314]

Done! How awful is it that allow tags in their markdown....

[scialex]

What is the benefit of this over simply using the Fn* family of types? It seems like those would allow you to do all the same things you put down here while letting us keep the common case easy and clean.

[Ericson2314]

@scialex the Fn* family of traits is meant to for closures not function pointers. Neither any concrete type implementing those traits nor trait objects of those traits have the same ABI as a function pointer, because in both cases it is assumed that the method is defined statically.

The best hack in the existing language is to create a new type of a function pointer with a phantom lifetime for borrowed pointers, and a newtype with a drop instance for owned pointers.

[bombless]

Then how do we handle extern-functions?
They are in similar system.

And can we remove that optional argument name? #838

I'll try making a PoC soon.
(a little bit off-topic, sorry)

[ftxqxd]

I like this RFC, but I think it should be adjusted to require &function to coerce a function into a &'static fn(). Like with other DSTs, functions have a fixed-sized type (there is no way of writing such types in Rust, but the important thing is that each function has its own type), and in this proposal there is also a dynamically-sized type (fn(Args..) -> Return) that ‘contains’ any fixed-size function type. However, the coercions for functions under this RFC would not match those of other DSTs: normally, the fixed-size type (e.g., [u8; 4]) must be behind a pointer for the coercion to occur (e.g., &[u8; 4] coerces to &[u8]). Under this RFC, typeof(foo) (where foo is a function) coerces to &'static fn(), instead of &typeof(foo) coercing to &'static fn().

[petrochenkov]

+1 to the RFC with @P1start's amendment
Even without considering dynamically loaded code and null function pointers in FFI, it would be much simpler, teachable and aesthetically pleasing to treat pointers to functions the same way as pointers to everything else.

fn f() {}
// f has type fn(){f}, unsized and unnameable
// &f has type &fn(){f}, pointer to the unsized and unnameable type
// fn() is generalized function type, unsized
// &fn() is pointer to the unsized generalized function type
fn main() {
let g = f; // ERROR: can't copy/move unsized type
let g = &f; // OK
let g: &fn() = f; // ERROR: can't coerce fn(){f} to &fn()
let g: fn() = &f; // ERROR: can't create object of unsized type on stack
let g: &fn() = &f; // OK; pointers to concrete functions coerce to pointers to generalized function
}

Unfortunately, IIRC, a similar idea was already discussed and rejected.

[CloudiDust]

+1 to this with amendment.

@P1start, do you mean typeof(&foo) not &typeof(foo)?

@petrochenkov, was that discussion before the implementation of DST?

[Florob]

Particularly with the amendment, I really like this. It could alleviate the fallout from the recent-ish change that each function has its own type.

I.e. where we now need:

fnf(x:Ty){ ...}let f:fn(Ty) = f;// coerce to fn pointer
foo.map(f)

we could probably do:

fn f(x:Ty){ ...}foo.map(&f)

[petrochenkov]

Some thoughts about the proposal.
Disclaimer: The statements below are mostly just guesses, so please correct me if I'm wrong.

1. As far as I know the current DST machinery always considers &UnsizedType to be a fat pointer, but in case of functions it is wrong and such pointers would just waste space.
2. The unique unnameable function types fn(){f} can be zero sized and not unsized, because all the information about them can be encoded in type system and there's no need to carry any pointers around at runtime.
3. Can the generalized function types fn() be zero sized instead of unsized? I guess yes, because pointers to zero-sized types are still meaningful (right?) and &fn() can successfully carry the code address around at runtime. Can external function types be zero-sized as well? I guess yes, for the same reasons, but I'm less sure about it.
   So, the conclusion is that the proposal shouldn't involve DSTs in any way, but make the function types zero-sized.

Edit:
But if the function types are (normal) zero-sized types, then they become even more crazy. They can be moved around by value losing their own identity, "empty" objects of function types can be created.
So, they somehow should be zero-sized and have the limitations of dynamically-sized types at the same time.

[Ericson2314]

@bombless If you are wondering whether &'static extern fn() would be valid syntax, the answer is yes. I will update the detailed changes to to make that clear.

@P1start Are you saying we should get rid of the secret_unboxed_function -> function pointer conversion and replace it with &secret_unboxed_function -> function for consistency? If so I think that is a great idea.

@petrochenkov@Florob if you don't mind, I think both your examples would look great in the RFC to explain the coercion changes.

@petrochenkov To be clear these semantic decisions would be left for the future, but I'm happy to speculate on how thing might turn out in the long run:

• For memory management purposes, it would be cool to have [fn; n]() sized types (just making up a syntax) and [fn]() would be it's DST equivalent with fat pointers. Functions could then be compiled to the proper [fn; n]() sized type rather than generating unique types for each of them.
• We should have a notion of unsized types distinct from dynamically sized types. This would be good for C strings, and unboxed functions. A pointer to a unsized type would be a thin pointer, solving our ABI problems.

[ftxqxd]

@Ericson2314 Yeah. In other words, a slight change in coercion rules that means that <specific function type> no longer coerces to fn(), and instead &<specific function type> coerces to &fn().

@CloudiDusttypeof(&foo) and &typeof(foo) are the same type, so it doesn’t matter either way. I guess typeof(&foo) might be a little clearer.

@Florob Why do you need to coerce to a function pointer anyway? Assuming that .map is a function like Option::map or IteratorExt::map that takes a F: Fn(A) -> B, you shouldn’t need to coerce it to a function pointer anyway. And if the function takes a fn(), the coercion should happen anyway without the need for a temporary. In the rare case that you do explicitly need a function pointer (e.g., when you wish to pass a function pointer to a generic function), unfortunately the amendment would not help that (because &f would still be assumed to be a &<specific function type> rather than &fn()).

@petrochenkov The anonymous function item types should already be zero-sized, but unfortunately aren’t yet (cc rust-lang/rust#19925). Regarding the fn() types, I think it still makes sense for them to be DSTs because they would be completely useless not behind a pointer if they were zero-sized types. I think what makes the most sense is (as @Ericson2314 notes) to have in the future a new category of DSTs that aren’t fat when behind pointers.

[Florob]

@P1start Indeed you may disregard that comment. It's a mixture of outdated information and lack of coherent thinking at the time.

That said I am still generally in favour of this proposal. It adds a nice explicitness to things.

[Ericson2314]

Ok, added the coercion rules. The response seems pretty positive here, any team member want to comment?

[CloudiDust]

This can be seen as an amendment to RFC 401, the Function type polymorphism section.

Every function item has its own fresh type. This type cannot be written by the programmer (i.e., it is expressible but not denotable).

No need to change.

Conceptually, for each fresh function type, there is an automatically generated implementation of the Fn, FnMut, and FnOnce traits.

No need to change.

All function types are implicitly coercible to a fn() type with the corresponding parameter types.

All function pointers are implicitly coercible to a &fn() type with the corresponding parameter types.

Conceptually, there is an implementation of Fn, FnMut, and FnOnce for every fn() type.

No need to change, but we will not be passing fn() values around, the important thing here is &fn should be coercible to &Fn(), &FnMut() and &FnOnce().

Fn, FnMut, or FnOnce trait objects and references to type parameters bounded by these traits may be considered to have the corresponding unboxed closure type. This is a desugaring (alias), rather than a coercion. This is an existing part of the unboxed closures work.

No need to change.

Examples:

fnfoo(){ ...}// `foo` has a fresh and non-denotable type. (`fn() {foo}`)fnmain(){let x:fn() = foo;// error (currently valid)let x:&fn() = &foo;// ok, `&foo` is coerced to `&fn()`. (currently invalid)let y:&Fn() = x;// ok, `x` is coerced to `&Fn` (a closure object),// legal because `x` is a `&fn()` here, and `fn()` implements `Fn()`.let z:&Fn() = foo;// error.let z:&Fn() = &foo;// ok, `&foo` is coerced to `&Fn()`,}

I think one of the reasons that we want functions have unique types that implement closure traits directly, is to avoid virtual calls. Is this correct?

cc @nick29581@nikomatsakis

EDIT: some corrections.

[CloudiDust]

@petrochenkov, after reading RFC 401, I think one of the reasons that "function values" (that have unique types) exist now, is because virtual calls can be avoided when function values are used as closures, while the same cannot be said about function pointers.

If my understanding is correct, then it makes sense to pass function values around, and the lack of & signifies the lack of indirection, which means:

fnf(){}// f has type fn(){f}, zero sized (ideally) and unnameable// &f has type &fn(){f}, pointer to the zero sized and unnameable type// fn() is the unsized and generalized function type.// &fn() is the pointer type to the unsized and generalized function typefnmain(){let g = f;// OK, copying a "function value".let g = &f;// OK, g is a `&fn(){f}`let g:&fn() = f;// ERROR: can't coerce fn(){f} to &fn()let g:fn() = &f;// ERROR: can't create object of unsized type on stacklet g:&fn() = &f;// OK; pointers to concrete functions coerce to pointers to generalized function}

[nikomatsakis]

On Wed, Mar 04, 2015 at 03:04:47AM -0800, Huon Wilson wrote:

@Ericson2314 could I ask you to update/clarify the RFC text to reflect the recent discussion? (It would be great to be able to land this RFC as soon as possible, if we land it at all, so getting it in to a more final form will help the core team who haven't been following decide what they think about it.)

Thanks huon, I was going to ask for the same. At this point, I can
summarize my feeling as follows. I consider this a "nice to have"
feature, meaning that I think it's an improvement over the state of
the art and, if we agree on a design and it gets implemented, I'd like
to see it land. However, I don't think it's strictly necessary and if
it doesn't land for 1.0 I could live with that. The reason I think it
is an improvement is that the current system of fn types is
definitely oriented around static code and does not scale to jitted
code etc. The reason I think we can surive without it is that this
need can easily be addressed via unsafely implemented types that wrap
the jitted code and implement Fn (and a JIT or dynamic code loading
system will have plenty of other unsafe code anyway). A side benefit
is that fn pointers will also use the & type, making it clear that
they are, indeed, pointers.

I think my preferred proposal is as follows:

• a fn item has a zero-sized type F;
• fn(...) is considered an unsized type representing the actual bytes,
  with an "auxiliary data" type of () (hence the fat pointer is the same
  size as a thin pointer).
• &F is coercible to &fn(...) as another kind of unsize coercion,
  analogous to &[T;N] => &[T] or &T => &Trait.

I believe this is more-or-less what @huonw presented as "option 2",
though it's been a day or two since I read it and I guess I will have
to re-read it.

[nikomatsakis]

I was thinking about this in the shower and I realized that what I wrote doesn't actually work. In particular, the use of an unsize coercion doesn't make sense -- it only makes sense if we assume that the only pointers of type &F where F is a fn type are always actually pointing in static memory. But if we ever gained the ability to name a fn type (which seems like it will eventually become possible, perhaps through typeof foo or some similar construct), that would not necessarily be true. So I think I would have to revise what I wrote to the following:

• a fn item has a zero-sized type F;
• fn(...) is considered an unsized type representing the actual bytes, with an "auxiliary data" type of () (hence the fat pointer is the same size as a thin pointer).
• F is coercible to &fn(...) as another kind of unsize coercion, analogous to &[T;N] => &[T] or &T => &Trait.

This is basically exactly what we have today except that the type of a fn pointer is &fn -- more-or-less what the original RFC wrote, except that we do the integration in a somewhat more "real" way rather than just having a "fake" reference like &'static fn. (Incidentally, I am opposed to that idea, because it would not match against an impl of &T and so forth.)

I think that going through this exercise leaves me more-or-less where I was before, but perhaps mildly more negative. The use case being addressed here feels marginal -- we're making the implementations of a dynamic linking utility library or jit a smidgen more safe. (Am I missing some other advantage?)

I guess it ultimately depends on whether you prefer to see &fn(i32) or fn(i32) to represent a fn pointer. The advantage of &fn(i32) is that it obviously a pointer (it has an &!). The disadvantage is that, in struct types at least, you'll have to write stuff like &'static extern "C" fn(i32), which is longer and more verbose. (In fn bodies you should be able to lean on inference though, at least.)

[CloudiDust]

@nikomatsakis I think "pointers should look like pointers" outweighs the disadvantage, especially when there can be function pointers that have non-static lifetimes in the future.

I notice that you assumed the F -> &'static fn(...) coercion which was what this RFC originally proposed but got replaced by &F -> &fn(...).

I'd prefer &F -> &fn(...), but this really depends on how one interprets the function proxy types. Are they like pointers so much that it is acceptable to coerce them to function pointers?

(I expect F -> &'static fn() to be easier to implement than &F -> &fn(), though.)

[Kimundi]

@nikomatsakis One issue that I see with the "custom type implementing a Fn() trait" approach is that the function types are quite complex and can not currently be recreated fully - things like ABI and variacity of the arguments for example.

In other words, its not actually possible right now to define a custom function pointer type that can be used as let f: FnPtr<Unsafe, "C", A, B, Z, R> = ..., while its possible to have a let f: unsafe extern "C" fn(A, B, Z) -> R.

By having function types actually just describe the actual code, unrelated to having a pointer to it, it would be possible to off-load all that complexity to the primitve, compiler-provided function types: let f: FnPtr<unsafe extern "C" fn(A, B, Z) -> R> = ...

[CloudiDust]

@nikomatsakis, I realized that I might have missed a point of your posts.

Were you implying that even if we wouldn't do F -> &'static fn(), it still wouldn't make sense to do &F -> &fn(), because &F might not have to be pointing into static memory in the future?

If this is the case, then I'd settle with simply doing F -> &'static fn().

[Ericson2314]

Sorry everybody, I've been busy with school. I'll try to fix this up this weekend---and go with the proxy approach.

@Kimundi Excellent point

@nikomatsakis In case you missed it, I mentioned that (using your identifiers) F could impl Deref<Target=fn()>. Then we would have &F -> &fn() as a deref coercion for free. This has a number of advantages:

• &F wouldn't be expected to point anywhere in particular, just as with all deref coercions.
• It is enough to implement the call operator traits on the unboxed function types, as Deref causes the methods to be propagated. F can be a completely normal singleton type with just an unsafe auto-generated Deref impl.
• F, as a function proxy type, is a reference type in some sense.

[CloudiDust]

@Ericson2314, &fn()s should still implement the closure traits, because currently we can pass fn()s around, but if fn()s become unsized, then they will not be passable as closures. We'll have to pass &fn()s instead.

I suspect dereferencing to unsized types need special-casing anyway, so if &F -> &fn() doesn't work as an unsize coercion, it won't work as a deref coercion.

That said, @nikomatsakis, I wonder why &F -> &fn() won't work if &F doesn't point into static memory?

[nikomatsakis]

On Thu, Mar 05, 2015 at 06:24:58PM -0800, Richard Zhang wrote:

Were you implying that even if we wouldn't do F -> &'static fn(), it still wouldn't make sense to do &F -> &fn(), because &F might not have to be pointing into static memory in the future?

I was implying that, but I think I was wrong. I thought about it some
more today and realized that a pointer to a zero-sized type, at
present, has an undefined value (but it can't be null). So in fact we
can in principle permit this coercion -- but it's a bit different than
other unsize coercions. The other coercions only attach a vtable-- but
this one also would transform the ptr from an undefined value to a
defined value. (But we can determine that value solely from the type,
so we're ok.)

[nikomatsakis]

On Thu, Mar 05, 2015 at 11:35:36PM -0800, Richard Zhang wrote:

That said, @nikomatsakis, I wonder why &F -> &fn() won't work if &F doesn't point into static memory?

To be clear, any instance of type F would ALWAYS point into static
memory. The whole idea is that F is a singleton type referring to a
specific fn item.

[nikomatsakis]

To be clear, any instance of type F would ALWAYS point into static memory. The whole idea is that F is a singleton type referring to a specific fn item.

Even though I wrote "to be clear", I think I was anything but. F doesn't really point anywhere, it's a zero-sized type corresponding to a specific fn item that the compiler knows about. This means the compiler can synthesize a pointer to that fn item on demand. But it also means that all instances of one of these types I am referring to as F are always static fn items.

[CloudiDust]

@nikomatsakis, thanks, so actually you were to mean: F (not &F) always "points" into static memory and &F would return an undefined value if F becomes zero-sized.

Currently, as F is not zero sized, I think a &F points to a F value just like any other normal reference, right?

(So, &F is not limited to be pointing to static memory currently, and where it points to won't matter in the future. EDIT: I for some reason was under the impression that pointers to zero-sized function values will store the functions' addresses in static memory, but actually they don't have to, as this piece of information is "encoded" in their types. I should have realized this earlier.)

And most importantly &F -> &fn() is doable.

[nikomatsakis]

On Fri, Mar 06, 2015 at 04:57:41PM -0800, Richard Zhang wrote:

Currently, as F is not zero sized, I think a &F points to a F value just like any other normal reference, right?

Yes.

(So, &F is not limited to be pointing to static memory currently, and where it points to won't matter in the future.)

Yes.

[CloudiDust]

So here is a summary of what we may want to do (assuming that we are going the "functions are almost statics" route):

1. Make the current function pointer types (fn()) unsized and represent the actual code bytes of functions.
2. Make the current function item types (F) zero sized. (This is an optimization and can wait.)
3. F should implement the closure traits just like today.
4. fn() will no longer implement the closure traits. (Closure traits don't work on unsized types, right?)
5. &fn() should implement the closure traits, old appearances of fn() should all be replaced by &fn().
6. Make F -> fn() illegal, while &F -> &fn() will be legal as an unsize coercion and/or a deref coercion.

Note:

1. Implementing Deref means implementing &F -> &fn() (as indicated by the signature of the deref method) and enabling the *F expression. I think *F is unnecessary for everyday usage. But if we are to stress the fact that F is pointer-like, then we may implement Deref to do so.
2. After the above changes, &fn() would be passable as closures (for implementing the closure traits) but not closure trait objects (for fn() no longer implements closure traits).
3. But &F would be passable as closures (after being coerced to &fn()) and closure trait objects (for being references to F, which implements closure traits).

There seems to be some asymmetry. @nikomatsakis, will fn() still be able to implement closure traits even when it becomes unsized?

[huonw]

fn() will no longer implement the closure traits. (Closure traits don't work on unsized types, right?)

They should work. Although, it may require the planned changes that ensure that FnOnce is properly object-safe to work fully.

[Ericson2314]

Here is a summery of what I am thinking (based off of @nikomatsakis and @CloudiDust's:

1. a fn item has a zero-sized type F. (Can be made zero-sized post 1.0.) F impls Copy as its just a proxy.
2. fn(...) is considered an unsized type representing the actual bytes, with an "auxiliary data" type of () (hence the fat pointer is the same size as a thin pointer).
3. impl Deref<Target=fn(...)> for F
4. Remove the F -> fn(...) coercion. Use a deref coercion instead.
5. F should implement the closure traits just like today.
6. fn(..) should also implement the closure traits like today. [In addition to the object safety problems, Fn for X -> FnOnce for &Y would be nice to avoid needlessly consuming the original, what is the status/verdict on this again?]
7. &fn(...) shouldn't need to implement Fn and FnMut (like today) as those take self by reference. Hopefully it wouldn't need to manually implement FnOnce either (see above).

[CloudiDust]

@Ericson2314,

&fn(...) shouldn't need to implement Fn and FnMut (like today) as those take self by reference. Hopefully it wouldn't need to manually implement FnOnce either (see above).

&fn(...) should implement the closure traits directly. Note how closures (types directly implementing closure traits) and closure trait objects (coercible from references to closures) are usable in different contexts because they have different types. If something doesn't directly implement Fn or FnMut or FnOnce, it simply cannot be used as a closure.

For &fn(...), these implementations will be special cased by the compiler and avoid extra indirections, I suppose.

[Ericson2314]

@CloudiDust I think I see what you mean. While impls on the unsized functions are better for trait objects, impls on the pointers are better for calling polymorphic functions.

Frankly though, I see the problem not specific to the closure traits, but more broadly part of a trade off between supporting in polymorphic code non-Copy types (or values that could be coppied but should be mutated in place) and cutting down on extraneous indirection. The controversy of whether iterators should be Copy, and the ByRef adapter iterator I'd wager stems in part from this.

An associated type defined to a borrowed pointer for non-copy types and a the type itself non copy ones would help in many places in Rust. The counter-argument is optimization should do most of that work, and it would make a whole lot of a code much less readable. And of course, negative bounds are needed to implement this.

Probably the best lower-impact solution is to make some blanket impls. This would also make "reusing" sugar-made closures more ergonomic, to give an example of a different benefit.

[CloudiDust]

@Ericson2314,

An associated type defined to a borrowed pointer for non-copy types and a the type itself non copy ones would help in many places in Rust.

I think you mean:

An associated type defined to:

• a borrowed pointer (to the type itself) for non-copy types,
• the type itself for copy ones,

would help in many places in Rust.

Right?

I'm thinking, generally, for non-Copy types, we have to pass/use references or the value will be consumed, but for Copy types, it is hard to say whether we should pass by-ref or by-value for efficiency. This depends a lot on the size of the type.

Probably the best lower-impact solution is to make some blanket impls. This would also make "reusing" sugar-made closures more ergonomic, to give an example of a different benefit.

I think "blanket impls" would do nothing? The impls would delegate to the underlying functions.

Also, would you please update the RFC? If you are still busy I'd like to help.

[eddyb]

Most of what I see here looks good, but there's a certain detail I've been pondering on - the use of "unsized" terminology.
One way I could see it is that "unsized X" means "sized X after unsizing has been applied", where "unsizing" could be "turning static sizing information into dynamic sizing", and not necessarily "removing sizing information".

All our (current) DSTs keep around enough information to query size and alignment, so we could have a Sizeable trait expressing this property.
It would be using methods instead of Sized's planned/proposed/hypothetical associated size/align constants, and it could be automatically implemented for the current trait object (via the vtable) and slices (multiplying element size with slice length).

Then "unsizing" can be defined as P<T> -> P<U> where T: Sized, U: !Sized + Sizeable.
After the changes proposed by this RFC, fn(..A) -> R would not implement Sizeable and could be used in generics with T: ?Sizeable.

One might also wonder what semantics would manual implementations of Sizeable have.
I like to think it would be an useful tool in designing custom DST layouts - I have even tried to represent slices in a library, e.g:

traitArray<T>{constLEN:usize;}// "Just" needs generics over values.impl<T,constN:usize>Array<T>for[T;N]{constLEN = N;}impl<T>SizeableforArray<T>{fnsize_of(&self) -> usize{self.LEN*size_of::<T>()}fnmin_align_of(&self) -> usize{min_align_of::<T>()}}// This is the tricky part, that might require backwards incompatible// changes to Drop.impl<T:Drop>DropforArray<T>{fndrop(&move self){for _ inself.into_iter(){}}}

I didn't mean to post that in its entirety, but there you go. More creative usecases left as an exercise for the reader.

[Ericson2314]

@CloudiDust Yeah that sounds good. A simpler way to look at the problem than my ranty post above is that in many cases we consume of an abstract type bounded on a trait that only borrows self. (e.g. consuming a FnMut in IteratorExt::map.) This seems tragic.

Also, size isn't important because it it trivial to always compile pass-by-value to pass-by-reference (we already do that) for specific, while the converse breaks code that depends on the pointer value itself (e.g. pointer equality).

@eddyb Yeah I want that for HAMTs, where traditionally nodes has a sparse array of a dynamically sized array with a bitmap to show which elements are present to save space. The size is thus the hamming weight of the bitmap. When @nikomatsakis said "... with an "auxiliary data" type of () ..." I assumed he was referring to some mechanism that is in place though? So Sizable isn't strictly necessary for this RFC.

[eddyb]

There is no mechanism in place and such a type is not valid today, as all types are required to allow their size and alignment to be queried, dynamically in the case of DSTs.

[CloudiDust]

@Ericson2314, would you please update the RFC soon? If you are still busy, I'd like to post an alternative RFC tomorrow (local time) to describe what we have come up in the discussions. The beta is approaching fast, so I think we should move forward now.

[Ericson2314]

@CloudiDust Yeah sorry. I've been busy, and for the upcoming week I won't have internet access, so I think you better do that. Thanks for taking the initiative!

[CloudiDust]

@Ericson2314, no problem and thanks for this RFC.

The new RFC is #996.

[nikomatsakis]

Should we close this RFC in favor of #996?

[nikomatsakis]

It seems that this RFC is basically deprecated in favor of #996. In any case, I'm going to close this as postponed under issue #1037. Thanks very much for the proposal and discussion.