Start documenting autodiff activities

[ZuseZ4]

View all comments

Some initial documentation of the autodiff macros and usage examples

[kevinyamauchi]

Thanks, @ZuseZ4 ! I will have a look tomorrow.

[kevinyamauchi]

Thanks @ZuseZ4 ! This already clarifies some things. Knowing which activity types are valid for forward/reverse mode is very helpful.

For me, it is still a bit unclear how the input/output annotations work in the reverse mode case where the output of the function is via a mutable reference function.

For example if we have a function f(x) = C * x^2 and we want to compute df / dx, is this how we would apply the macro?

#[autodiff_reverse(d_foo,Const,Active,Duplicated)]fnfoo(c:f32: x:f32,out:&mutf32){
c * x * x
}

If so, is this how we call the function?

letC:f32 = 4.0;let x:f32 = 3.0;// store the result of fooletmut foo_result:f32 = 0.0;// shadow variable to store the dF/dx valueletmut dFoo_dx = 1.0;d_foo(C, x,&mut foo_result,&mut dFoo_dx);// I would expect dFoo_dx to be 2* 4 * 3 = 24*[View changes since this review](https://triagebot.infra.rust-lang.org/gh-changes-since/rust-lang/rust/148201/df984edf44203c862e01b5a20c8092d5614d872e..a2dce774bc35a7fbafbe2d191a4eedf99023e17a)*

[kevinyamauchi]

To me, it is unclear what "this case" refers to. Based on the following text, I think this refers to the case when the output is a mutable reference or pointer argument. Is that right? If so, maybe something like the following would be more clear

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. In the latter case, the

or

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. If the output is stored in a mutable reference or pointer argument, the

[kevinyamauchi]

It is a bit unclear to me how the 1.0 and 0.0 are used to specify the partial derivatives. I would assume the output given 1.0 has it's derivative evaluated and the one 0.0 not, but I think it would be good to be explicit in the docs.

[ZuseZ4]

I think you're right with asking for a concrete example, the enzyme.jl (julia) docs have some as well: https://enzyme.mit.edu/julia/stable/#Reverse-mode

I'll add two (in place and returning) examples.

[kevinyamauchi]

Awesome. Thank you!

[ZuseZ4]

@kevinyamauchi sorry, it took me a bit. I added examples and addressed your nit, does that help? Any parts which you still find hard to understand?

[ZuseZ4]

r? @jieyouxu Since you seem to be quite active on cleaning up tests.
I'd like to have code examples in my docs, but as you see above, they require an extra feature and will only work sometimes on nightly, if we have autodiff enabled. (@needs-enzyme for other type of tests) Is there a good way to express this? I'm also fine to mark them as allowed to fail, just not sure how (the dev guide is a bit sparse on doc tests).

[jieyouxu]

Do you want to only express them as "examples", or do you mean actual doctests?

[ZuseZ4]

I'd prefer them to be doctests, just to guarantee that our docs are always up to date.
If that's too hard however, we can also just have them as examples.

[jieyouxu]

As doctests, that'd have to be gated with a cfg when building libcore (like --cfg bootstrap_autodiff_enabled) and then use that to

#[cfg_attr(bootstrap_autodiff_enabled, doc = " ```rust")]#[cfg_attr(not(bootstrap_autodiff_enabled), doc = " ```rust,compile_fail")]/// use meow;

[ZuseZ4]

It seems like bjorn has already added such a feature here: #146598
So I'd just need to make sure it's also set for libcore, thanks.

[ZuseZ4]

@jieyouxu I've added those checks, however things are still failing due to autodiff depending on RUSTFLAGS=-Zautodiff=Enable, as well as -Clto=fat. Any suggestions on how to handle this?

[jieyouxu]

Hm, I think it's easiest to leave as examples only then. Otherwise you'd need to build the doctests against a compiler/library toolchain that has autodiff enabled with fat LTO IIUC

[ZuseZ4]

Sad, but sure. I marked them as ignore, I guess that's what we want?

[jieyouxu]

bootstrap side looks fine

View changes since this review

[jieyouxu]

r? libs

[oli-obk]

@bors r+ rollup

Can iterate more in the future but let's ship this initial version so users can have sth to refer to

[bors]

📌 Commit f5892da has been approved by oli-obk

It is now in the queue for this repository.

[ZuseZ4]

View all comments

Some initial documentation of the autodiff macros and usage examples

[kevinyamauchi]

Thanks, @ZuseZ4 ! I will have a look tomorrow.

[kevinyamauchi]

Thanks @ZuseZ4 ! This already clarifies some things. Knowing which activity types are valid for forward/reverse mode is very helpful.

For me, it is still a bit unclear how the input/output annotations work in the reverse mode case where the output of the function is via a mutable reference function.

For example if we have a function f(x) = C * x^2 and we want to compute df / dx, is this how we would apply the macro?

#[autodiff_reverse(d_foo,Const,Active,Duplicated)]fnfoo(c:f32: x:f32,out:&mutf32){
c * x * x
}

If so, is this how we call the function?

letC:f32 = 4.0;let x:f32 = 3.0;// store the result of fooletmut foo_result:f32 = 0.0;// shadow variable to store the dF/dx valueletmut dFoo_dx = 1.0;d_foo(C, x,&mut foo_result,&mut dFoo_dx);// I would expect dFoo_dx to be 2* 4 * 3 = 24*[View changes since this review](https://triagebot.infra.rust-lang.org/gh-changes-since/rust-lang/rust/148201/df984edf44203c862e01b5a20c8092d5614d872e..a2dce774bc35a7fbafbe2d191a4eedf99023e17a)*

[kevinyamauchi]

To me, it is unclear what "this case" refers to. Based on the following text, I think this refers to the case when the output is a mutable reference or pointer argument. Is that right? If so, maybe something like the following would be more clear

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. In the latter case, the

or

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. If the output is stored in a mutable reference or pointer argument, the

[kevinyamauchi]

It is a bit unclear to me how the 1.0 and 0.0 are used to specify the partial derivatives. I would assume the output given 1.0 has it's derivative evaluated and the one 0.0 not, but I think it would be good to be explicit in the docs.

[ZuseZ4]

I think you're right with asking for a concrete example, the enzyme.jl (julia) docs have some as well: https://enzyme.mit.edu/julia/stable/#Reverse-mode

I'll add two (in place and returning) examples.

[kevinyamauchi]

Awesome. Thank you!

[ZuseZ4]

@kevinyamauchi sorry, it took me a bit. I added examples and addressed your nit, does that help? Any parts which you still find hard to understand?

[ZuseZ4]

r? @jieyouxu Since you seem to be quite active on cleaning up tests.
I'd like to have code examples in my docs, but as you see above, they require an extra feature and will only work sometimes on nightly, if we have autodiff enabled. (@needs-enzyme for other type of tests) Is there a good way to express this? I'm also fine to mark them as allowed to fail, just not sure how (the dev guide is a bit sparse on doc tests).

[jieyouxu]

Do you want to only express them as "examples", or do you mean actual doctests?

[ZuseZ4]

I'd prefer them to be doctests, just to guarantee that our docs are always up to date.
If that's too hard however, we can also just have them as examples.

[jieyouxu]

As doctests, that'd have to be gated with a cfg when building libcore (like --cfg bootstrap_autodiff_enabled) and then use that to

#[cfg_attr(bootstrap_autodiff_enabled, doc = " ```rust")]#[cfg_attr(not(bootstrap_autodiff_enabled), doc = " ```rust,compile_fail")]/// use meow;

[ZuseZ4]

It seems like bjorn has already added such a feature here: #146598
So I'd just need to make sure it's also set for libcore, thanks.

[ZuseZ4]

@jieyouxu I've added those checks, however things are still failing due to autodiff depending on RUSTFLAGS=-Zautodiff=Enable, as well as -Clto=fat. Any suggestions on how to handle this?

[jieyouxu]

Hm, I think it's easiest to leave as examples only then. Otherwise you'd need to build the doctests against a compiler/library toolchain that has autodiff enabled with fat LTO IIUC

[ZuseZ4]

Sad, but sure. I marked them as ignore, I guess that's what we want?

[jieyouxu]

bootstrap side looks fine

View changes since this review

[jieyouxu]

r? libs

[oli-obk]

@bors r+ rollup

Can iterate more in the future but let's ship this initial version so users can have sth to refer to

[bors]

📌 Commit f5892da has been approved by oli-obk

It is now in the queue for this repository.

[ZuseZ4]

View all comments

Some initial documentation of the autodiff macros and usage examples

[kevinyamauchi]

Thanks, @ZuseZ4 ! I will have a look tomorrow.

[kevinyamauchi]

Thanks @ZuseZ4 ! This already clarifies some things. Knowing which activity types are valid for forward/reverse mode is very helpful.

For me, it is still a bit unclear how the input/output annotations work in the reverse mode case where the output of the function is via a mutable reference function.

For example if we have a function f(x) = C * x^2 and we want to compute df / dx, is this how we would apply the macro?

#[autodiff_reverse(d_foo,Const,Active,Duplicated)]fnfoo(c:f32: x:f32,out:&mutf32){
c * x * x
}

If so, is this how we call the function?

letC:f32 = 4.0;let x:f32 = 3.0;// store the result of fooletmut foo_result:f32 = 0.0;// shadow variable to store the dF/dx valueletmut dFoo_dx = 1.0;d_foo(C, x,&mut foo_result,&mut dFoo_dx);// I would expect dFoo_dx to be 2* 4 * 3 = 24*[View changes since this review](https://triagebot.infra.rust-lang.org/gh-changes-since/rust-lang/rust/148201/df984edf44203c862e01b5a20c8092d5614d872e..a2dce774bc35a7fbafbe2d191a4eedf99023e17a)*

[kevinyamauchi]

To me, it is unclear what "this case" refers to. Based on the following text, I think this refers to the case when the output is a mutable reference or pointer argument. Is that right? If so, maybe something like the following would be more clear

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. In the latter case, the

or

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. If the output is stored in a mutable reference or pointer argument, the

[kevinyamauchi]

It is a bit unclear to me how the 1.0 and 0.0 are used to specify the partial derivatives. I would assume the output given 1.0 has it's derivative evaluated and the one 0.0 not, but I think it would be good to be explicit in the docs.

[ZuseZ4]

I think you're right with asking for a concrete example, the enzyme.jl (julia) docs have some as well: https://enzyme.mit.edu/julia/stable/#Reverse-mode

I'll add two (in place and returning) examples.

[kevinyamauchi]

Awesome. Thank you!

[ZuseZ4]

@kevinyamauchi sorry, it took me a bit. I added examples and addressed your nit, does that help? Any parts which you still find hard to understand?

[ZuseZ4]

r? @jieyouxu Since you seem to be quite active on cleaning up tests.
I'd like to have code examples in my docs, but as you see above, they require an extra feature and will only work sometimes on nightly, if we have autodiff enabled. (@needs-enzyme for other type of tests) Is there a good way to express this? I'm also fine to mark them as allowed to fail, just not sure how (the dev guide is a bit sparse on doc tests).

[jieyouxu]

Do you want to only express them as "examples", or do you mean actual doctests?

[ZuseZ4]

I'd prefer them to be doctests, just to guarantee that our docs are always up to date.
If that's too hard however, we can also just have them as examples.

[jieyouxu]

As doctests, that'd have to be gated with a cfg when building libcore (like --cfg bootstrap_autodiff_enabled) and then use that to

#[cfg_attr(bootstrap_autodiff_enabled, doc = " ```rust")]#[cfg_attr(not(bootstrap_autodiff_enabled), doc = " ```rust,compile_fail")]/// use meow;

[ZuseZ4]

It seems like bjorn has already added such a feature here: #146598
So I'd just need to make sure it's also set for libcore, thanks.

[ZuseZ4]

@jieyouxu I've added those checks, however things are still failing due to autodiff depending on RUSTFLAGS=-Zautodiff=Enable, as well as -Clto=fat. Any suggestions on how to handle this?

[jieyouxu]

Hm, I think it's easiest to leave as examples only then. Otherwise you'd need to build the doctests against a compiler/library toolchain that has autodiff enabled with fat LTO IIUC

[ZuseZ4]

Sad, but sure. I marked them as ignore, I guess that's what we want?

[jieyouxu]

bootstrap side looks fine

View changes since this review

[jieyouxu]

r? libs

[oli-obk]

@bors r+ rollup

Can iterate more in the future but let's ship this initial version so users can have sth to refer to

[bors]

📌 Commit f5892da has been approved by oli-obk

It is now in the queue for this repository.

[ZuseZ4]

View all comments

Some initial documentation of the autodiff macros and usage examples

[kevinyamauchi]

Thanks, @ZuseZ4 ! I will have a look tomorrow.

[kevinyamauchi]

Thanks @ZuseZ4 ! This already clarifies some things. Knowing which activity types are valid for forward/reverse mode is very helpful.

For me, it is still a bit unclear how the input/output annotations work in the reverse mode case where the output of the function is via a mutable reference function.

For example if we have a function f(x) = C * x^2 and we want to compute df / dx, is this how we would apply the macro?

#[autodiff_reverse(d_foo,Const,Active,Duplicated)]fnfoo(c:f32: x:f32,out:&mutf32){
c * x * x
}

If so, is this how we call the function?

letC:f32 = 4.0;let x:f32 = 3.0;// store the result of fooletmut foo_result:f32 = 0.0;// shadow variable to store the dF/dx valueletmut dFoo_dx = 1.0;d_foo(C, x,&mut foo_result,&mut dFoo_dx);// I would expect dFoo_dx to be 2* 4 * 3 = 24*[View changes since this review](https://triagebot.infra.rust-lang.org/gh-changes-since/rust-lang/rust/148201/df984edf44203c862e01b5a20c8092d5614d872e..a2dce774bc35a7fbafbe2d191a4eedf99023e17a)*

[kevinyamauchi]

To me, it is unclear what "this case" refers to. Based on the following text, I think this refers to the case when the output is a mutable reference or pointer argument. Is that right? If so, maybe something like the following would be more clear

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. In the latter case, the

or

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. If the output is stored in a mutable reference or pointer argument, the

[kevinyamauchi]

It is a bit unclear to me how the 1.0 and 0.0 are used to specify the partial derivatives. I would assume the output given 1.0 has it's derivative evaluated and the one 0.0 not, but I think it would be good to be explicit in the docs.

[ZuseZ4]

I think you're right with asking for a concrete example, the enzyme.jl (julia) docs have some as well: https://enzyme.mit.edu/julia/stable/#Reverse-mode

I'll add two (in place and returning) examples.

[kevinyamauchi]

Awesome. Thank you!

[ZuseZ4]

@kevinyamauchi sorry, it took me a bit. I added examples and addressed your nit, does that help? Any parts which you still find hard to understand?

[ZuseZ4]

r? @jieyouxu Since you seem to be quite active on cleaning up tests.
I'd like to have code examples in my docs, but as you see above, they require an extra feature and will only work sometimes on nightly, if we have autodiff enabled. (@needs-enzyme for other type of tests) Is there a good way to express this? I'm also fine to mark them as allowed to fail, just not sure how (the dev guide is a bit sparse on doc tests).

[jieyouxu]

Do you want to only express them as "examples", or do you mean actual doctests?

[ZuseZ4]

I'd prefer them to be doctests, just to guarantee that our docs are always up to date.
If that's too hard however, we can also just have them as examples.

[jieyouxu]

As doctests, that'd have to be gated with a cfg when building libcore (like --cfg bootstrap_autodiff_enabled) and then use that to

#[cfg_attr(bootstrap_autodiff_enabled, doc = " ```rust")]#[cfg_attr(not(bootstrap_autodiff_enabled), doc = " ```rust,compile_fail")]/// use meow;

[ZuseZ4]

It seems like bjorn has already added such a feature here: #146598
So I'd just need to make sure it's also set for libcore, thanks.

[ZuseZ4]

@jieyouxu I've added those checks, however things are still failing due to autodiff depending on RUSTFLAGS=-Zautodiff=Enable, as well as -Clto=fat. Any suggestions on how to handle this?

[jieyouxu]

Hm, I think it's easiest to leave as examples only then. Otherwise you'd need to build the doctests against a compiler/library toolchain that has autodiff enabled with fat LTO IIUC

[ZuseZ4]

Sad, but sure. I marked them as ignore, I guess that's what we want?

[jieyouxu]

bootstrap side looks fine

View changes since this review

[jieyouxu]

r? libs

[oli-obk]

@bors r+ rollup

Can iterate more in the future but let's ship this initial version so users can have sth to refer to

[bors]

📌 Commit f5892da has been approved by oli-obk

It is now in the queue for this repository.

[ZuseZ4]

View all comments

Some initial documentation of the autodiff macros and usage examples

[kevinyamauchi]

Thanks, @ZuseZ4 ! I will have a look tomorrow.

[kevinyamauchi]

Thanks @ZuseZ4 ! This already clarifies some things. Knowing which activity types are valid for forward/reverse mode is very helpful.

For me, it is still a bit unclear how the input/output annotations work in the reverse mode case where the output of the function is via a mutable reference function.

For example if we have a function f(x) = C * x^2 and we want to compute df / dx, is this how we would apply the macro?

#[autodiff_reverse(d_foo,Const,Active,Duplicated)]fnfoo(c:f32: x:f32,out:&mutf32){
c * x * x
}

If so, is this how we call the function?

letC:f32 = 4.0;let x:f32 = 3.0;// store the result of fooletmut foo_result:f32 = 0.0;// shadow variable to store the dF/dx valueletmut dFoo_dx = 1.0;d_foo(C, x,&mut foo_result,&mut dFoo_dx);// I would expect dFoo_dx to be 2* 4 * 3 = 24*[View changes since this review](https://triagebot.infra.rust-lang.org/gh-changes-since/rust-lang/rust/148201/df984edf44203c862e01b5a20c8092d5614d872e..a2dce774bc35a7fbafbe2d191a4eedf99023e17a)*

[kevinyamauchi]

To me, it is unclear what "this case" refers to. Based on the following text, I think this refers to the case when the output is a mutable reference or pointer argument. Is that right? If so, maybe something like the following would be more clear

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. In the latter case, the

or

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. If the output is stored in a mutable reference or pointer argument, the

[kevinyamauchi]

It is a bit unclear to me how the 1.0 and 0.0 are used to specify the partial derivatives. I would assume the output given 1.0 has it's derivative evaluated and the one 0.0 not, but I think it would be good to be explicit in the docs.

[ZuseZ4]

I think you're right with asking for a concrete example, the enzyme.jl (julia) docs have some as well: https://enzyme.mit.edu/julia/stable/#Reverse-mode

I'll add two (in place and returning) examples.

[kevinyamauchi]

Awesome. Thank you!

[ZuseZ4]

@kevinyamauchi sorry, it took me a bit. I added examples and addressed your nit, does that help? Any parts which you still find hard to understand?

[ZuseZ4]

r? @jieyouxu Since you seem to be quite active on cleaning up tests.
I'd like to have code examples in my docs, but as you see above, they require an extra feature and will only work sometimes on nightly, if we have autodiff enabled. (@needs-enzyme for other type of tests) Is there a good way to express this? I'm also fine to mark them as allowed to fail, just not sure how (the dev guide is a bit sparse on doc tests).

[jieyouxu]

Do you want to only express them as "examples", or do you mean actual doctests?

[ZuseZ4]

I'd prefer them to be doctests, just to guarantee that our docs are always up to date.
If that's too hard however, we can also just have them as examples.

[jieyouxu]

As doctests, that'd have to be gated with a cfg when building libcore (like --cfg bootstrap_autodiff_enabled) and then use that to

#[cfg_attr(bootstrap_autodiff_enabled, doc = " ```rust")]#[cfg_attr(not(bootstrap_autodiff_enabled), doc = " ```rust,compile_fail")]/// use meow;

[ZuseZ4]

It seems like bjorn has already added such a feature here: #146598
So I'd just need to make sure it's also set for libcore, thanks.

[ZuseZ4]

@jieyouxu I've added those checks, however things are still failing due to autodiff depending on RUSTFLAGS=-Zautodiff=Enable, as well as -Clto=fat. Any suggestions on how to handle this?

[jieyouxu]

Hm, I think it's easiest to leave as examples only then. Otherwise you'd need to build the doctests against a compiler/library toolchain that has autodiff enabled with fat LTO IIUC

[ZuseZ4]

Sad, but sure. I marked them as ignore, I guess that's what we want?

[jieyouxu]

bootstrap side looks fine

View changes since this review

[jieyouxu]

r? libs

[oli-obk]

@bors r+ rollup

Can iterate more in the future but let's ship this initial version so users can have sth to refer to

[bors]

📌 Commit f5892da has been approved by oli-obk

It is now in the queue for this repository.

[ZuseZ4]

View all comments

Some initial documentation of the autodiff macros and usage examples

[kevinyamauchi]

Thanks, @ZuseZ4 ! I will have a look tomorrow.

[kevinyamauchi]

Thanks @ZuseZ4 ! This already clarifies some things. Knowing which activity types are valid for forward/reverse mode is very helpful.

For me, it is still a bit unclear how the input/output annotations work in the reverse mode case where the output of the function is via a mutable reference function.

For example if we have a function f(x) = C * x^2 and we want to compute df / dx, is this how we would apply the macro?

#[autodiff_reverse(d_foo,Const,Active,Duplicated)]fnfoo(c:f32: x:f32,out:&mutf32){
c * x * x
}

If so, is this how we call the function?

letC:f32 = 4.0;let x:f32 = 3.0;// store the result of fooletmut foo_result:f32 = 0.0;// shadow variable to store the dF/dx valueletmut dFoo_dx = 1.0;d_foo(C, x,&mut foo_result,&mut dFoo_dx);// I would expect dFoo_dx to be 2* 4 * 3 = 24*[View changes since this review](https://triagebot.infra.rust-lang.org/gh-changes-since/rust-lang/rust/148201/df984edf44203c862e01b5a20c8092d5614d872e..a2dce774bc35a7fbafbe2d191a4eedf99023e17a)*

[kevinyamauchi]

To me, it is unclear what "this case" refers to. Based on the following text, I think this refers to the case when the output is a mutable reference or pointer argument. Is that right? If so, maybe something like the following would be more clear

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. In the latter case, the

or

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. If the output is stored in a mutable reference or pointer argument, the

[kevinyamauchi]

It is a bit unclear to me how the 1.0 and 0.0 are used to specify the partial derivatives. I would assume the output given 1.0 has it's derivative evaluated and the one 0.0 not, but I think it would be good to be explicit in the docs.

[ZuseZ4]

I think you're right with asking for a concrete example, the enzyme.jl (julia) docs have some as well: https://enzyme.mit.edu/julia/stable/#Reverse-mode

I'll add two (in place and returning) examples.

[kevinyamauchi]

Awesome. Thank you!

[ZuseZ4]

@kevinyamauchi sorry, it took me a bit. I added examples and addressed your nit, does that help? Any parts which you still find hard to understand?

[ZuseZ4]

r? @jieyouxu Since you seem to be quite active on cleaning up tests.
I'd like to have code examples in my docs, but as you see above, they require an extra feature and will only work sometimes on nightly, if we have autodiff enabled. (@needs-enzyme for other type of tests) Is there a good way to express this? I'm also fine to mark them as allowed to fail, just not sure how (the dev guide is a bit sparse on doc tests).

[jieyouxu]

Do you want to only express them as "examples", or do you mean actual doctests?

[ZuseZ4]

I'd prefer them to be doctests, just to guarantee that our docs are always up to date.
If that's too hard however, we can also just have them as examples.

[jieyouxu]

As doctests, that'd have to be gated with a cfg when building libcore (like --cfg bootstrap_autodiff_enabled) and then use that to

#[cfg_attr(bootstrap_autodiff_enabled, doc = " ```rust")]#[cfg_attr(not(bootstrap_autodiff_enabled), doc = " ```rust,compile_fail")]/// use meow;

[ZuseZ4]

It seems like bjorn has already added such a feature here: #146598
So I'd just need to make sure it's also set for libcore, thanks.

[ZuseZ4]

@jieyouxu I've added those checks, however things are still failing due to autodiff depending on RUSTFLAGS=-Zautodiff=Enable, as well as -Clto=fat. Any suggestions on how to handle this?

[jieyouxu]

Hm, I think it's easiest to leave as examples only then. Otherwise you'd need to build the doctests against a compiler/library toolchain that has autodiff enabled with fat LTO IIUC

[ZuseZ4]

Sad, but sure. I marked them as ignore, I guess that's what we want?

[jieyouxu]

bootstrap side looks fine

View changes since this review

[jieyouxu]

r? libs

[oli-obk]

@bors r+ rollup

Can iterate more in the future but let's ship this initial version so users can have sth to refer to

[bors]

📌 Commit f5892da has been approved by oli-obk

It is now in the queue for this repository.

[ZuseZ4]

View all comments

Some initial documentation of the autodiff macros and usage examples

[kevinyamauchi]

Thanks, @ZuseZ4 ! I will have a look tomorrow.

[kevinyamauchi]

Thanks @ZuseZ4 ! This already clarifies some things. Knowing which activity types are valid for forward/reverse mode is very helpful.

For me, it is still a bit unclear how the input/output annotations work in the reverse mode case where the output of the function is via a mutable reference function.

For example if we have a function f(x) = C * x^2 and we want to compute df / dx, is this how we would apply the macro?

#[autodiff_reverse(d_foo,Const,Active,Duplicated)]fnfoo(c:f32: x:f32,out:&mutf32){
c * x * x
}

If so, is this how we call the function?

letC:f32 = 4.0;let x:f32 = 3.0;// store the result of fooletmut foo_result:f32 = 0.0;// shadow variable to store the dF/dx valueletmut dFoo_dx = 1.0;d_foo(C, x,&mut foo_result,&mut dFoo_dx);// I would expect dFoo_dx to be 2* 4 * 3 = 24*[View changes since this review](https://triagebot.infra.rust-lang.org/gh-changes-since/rust-lang/rust/148201/df984edf44203c862e01b5a20c8092d5614d872e..a2dce774bc35a7fbafbe2d191a4eedf99023e17a)*

[kevinyamauchi]

To me, it is unclear what "this case" refers to. Based on the following text, I think this refers to the case when the output is a mutable reference or pointer argument. Is that right? If so, maybe something like the following would be more clear

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. In the latter case, the

or

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. If the output is stored in a mutable reference or pointer argument, the

[kevinyamauchi]

It is a bit unclear to me how the 1.0 and 0.0 are used to specify the partial derivatives. I would assume the output given 1.0 has it's derivative evaluated and the one 0.0 not, but I think it would be good to be explicit in the docs.

[ZuseZ4]

I think you're right with asking for a concrete example, the enzyme.jl (julia) docs have some as well: https://enzyme.mit.edu/julia/stable/#Reverse-mode

I'll add two (in place and returning) examples.

[kevinyamauchi]

Awesome. Thank you!

[ZuseZ4]

@kevinyamauchi sorry, it took me a bit. I added examples and addressed your nit, does that help? Any parts which you still find hard to understand?

[ZuseZ4]

r? @jieyouxu Since you seem to be quite active on cleaning up tests.
I'd like to have code examples in my docs, but as you see above, they require an extra feature and will only work sometimes on nightly, if we have autodiff enabled. (@needs-enzyme for other type of tests) Is there a good way to express this? I'm also fine to mark them as allowed to fail, just not sure how (the dev guide is a bit sparse on doc tests).

[jieyouxu]

Do you want to only express them as "examples", or do you mean actual doctests?

[ZuseZ4]

I'd prefer them to be doctests, just to guarantee that our docs are always up to date.
If that's too hard however, we can also just have them as examples.

[jieyouxu]

As doctests, that'd have to be gated with a cfg when building libcore (like --cfg bootstrap_autodiff_enabled) and then use that to

#[cfg_attr(bootstrap_autodiff_enabled, doc = " ```rust")]#[cfg_attr(not(bootstrap_autodiff_enabled), doc = " ```rust,compile_fail")]/// use meow;

[ZuseZ4]

It seems like bjorn has already added such a feature here: #146598
So I'd just need to make sure it's also set for libcore, thanks.

[ZuseZ4]

@jieyouxu I've added those checks, however things are still failing due to autodiff depending on RUSTFLAGS=-Zautodiff=Enable, as well as -Clto=fat. Any suggestions on how to handle this?

[jieyouxu]

Hm, I think it's easiest to leave as examples only then. Otherwise you'd need to build the doctests against a compiler/library toolchain that has autodiff enabled with fat LTO IIUC

[ZuseZ4]

Sad, but sure. I marked them as ignore, I guess that's what we want?

[jieyouxu]

bootstrap side looks fine

View changes since this review

[jieyouxu]

r? libs

[oli-obk]

@bors r+ rollup

Can iterate more in the future but let's ship this initial version so users can have sth to refer to

[bors]

📌 Commit f5892da has been approved by oli-obk

It is now in the queue for this repository.

[ZuseZ4]

View all comments

Some initial documentation of the autodiff macros and usage examples

[kevinyamauchi]

Thanks, @ZuseZ4 ! I will have a look tomorrow.

[kevinyamauchi]

Thanks @ZuseZ4 ! This already clarifies some things. Knowing which activity types are valid for forward/reverse mode is very helpful.

For me, it is still a bit unclear how the input/output annotations work in the reverse mode case where the output of the function is via a mutable reference function.

For example if we have a function f(x) = C * x^2 and we want to compute df / dx, is this how we would apply the macro?

#[autodiff_reverse(d_foo,Const,Active,Duplicated)]fnfoo(c:f32: x:f32,out:&mutf32){
c * x * x
}

If so, is this how we call the function?

letC:f32 = 4.0;let x:f32 = 3.0;// store the result of fooletmut foo_result:f32 = 0.0;// shadow variable to store the dF/dx valueletmut dFoo_dx = 1.0;d_foo(C, x,&mut foo_result,&mut dFoo_dx);// I would expect dFoo_dx to be 2* 4 * 3 = 24*[View changes since this review](https://triagebot.infra.rust-lang.org/gh-changes-since/rust-lang/rust/148201/df984edf44203c862e01b5a20c8092d5614d872e..a2dce774bc35a7fbafbe2d191a4eedf99023e17a)*

[kevinyamauchi]

To me, it is unclear what "this case" refers to. Based on the following text, I think this refers to the case when the output is a mutable reference or pointer argument. Is that right? If so, maybe something like the following would be more clear

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. In the latter case, the

or

Suggested change

	/// be a scalar return value, or a mutable reference or pointer argument. In this case, the
	/// be a scalar return value, or a mutable reference or pointer argument. If the output is stored in a mutable reference or pointer argument, the

[kevinyamauchi]

It is a bit unclear to me how the 1.0 and 0.0 are used to specify the partial derivatives. I would assume the output given 1.0 has it's derivative evaluated and the one 0.0 not, but I think it would be good to be explicit in the docs.

[ZuseZ4]

I think you're right with asking for a concrete example, the enzyme.jl (julia) docs have some as well: https://enzyme.mit.edu/julia/stable/#Reverse-mode

I'll add two (in place and returning) examples.

[kevinyamauchi]

Awesome. Thank you!

[ZuseZ4]

@kevinyamauchi sorry, it took me a bit. I added examples and addressed your nit, does that help? Any parts which you still find hard to understand?

[ZuseZ4]

r? @jieyouxu Since you seem to be quite active on cleaning up tests.
I'd like to have code examples in my docs, but as you see above, they require an extra feature and will only work sometimes on nightly, if we have autodiff enabled. (@needs-enzyme for other type of tests) Is there a good way to express this? I'm also fine to mark them as allowed to fail, just not sure how (the dev guide is a bit sparse on doc tests).

[jieyouxu]

Do you want to only express them as "examples", or do you mean actual doctests?

[ZuseZ4]

I'd prefer them to be doctests, just to guarantee that our docs are always up to date.
If that's too hard however, we can also just have them as examples.

[jieyouxu]

As doctests, that'd have to be gated with a cfg when building libcore (like --cfg bootstrap_autodiff_enabled) and then use that to

#[cfg_attr(bootstrap_autodiff_enabled, doc = " ```rust")]#[cfg_attr(not(bootstrap_autodiff_enabled), doc = " ```rust,compile_fail")]/// use meow;

[ZuseZ4]

It seems like bjorn has already added such a feature here: #146598
So I'd just need to make sure it's also set for libcore, thanks.

[ZuseZ4]

@jieyouxu I've added those checks, however things are still failing due to autodiff depending on RUSTFLAGS=-Zautodiff=Enable, as well as -Clto=fat. Any suggestions on how to handle this?

[jieyouxu]

Hm, I think it's easiest to leave as examples only then. Otherwise you'd need to build the doctests against a compiler/library toolchain that has autodiff enabled with fat LTO IIUC

[ZuseZ4]

Sad, but sure. I marked them as ignore, I guess that's what we want?

[jieyouxu]

bootstrap side looks fine

View changes since this review

[jieyouxu]

r? libs

[oli-obk]

@bors r+ rollup

Can iterate more in the future but let's ship this initial version so users can have sth to refer to

[bors]

📌 Commit f5892da has been approved by oli-obk

It is now in the queue for this repository.