Build wasmtime with wasmtime-c-api-impl crate

[leonm1]

Always uses prefixed wasmtime-c-api-impl, otherwise the prefixed implementation bitrots. Prefixes the headers in the repo rule to allow for globbing, which is not possible in a genrule.

The crates_vendor-provided wasmtime-c-api-impl provided build allows us to upgrade wasmtime solely by changing the version number in the repositories.bzl and Cargo.toml files.

Build off of #496

[PiotrSikora]

A few high-level comments:

1. Why? Looking at the changes, the benefits don't seem very obvious.
2. Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?
3. Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

[leonm1]

Why? Looking at the changes, the benefits don't seem very obvious.

Two things:

1. Updating wasmtime resulted in our rust_static_library in wasmtime.BUILD failing to build due to missing dependencies. With this current approach, the dependency tree is resolved by Cargo and we simply wrap it in rust_static_library. Updates which pull in new dependencies are trivial after this change and (in theory) will never require hunting down missing dependencies.
2. For prefixing itself, using parts of the wasmtime_c_api (as opposed to the raw wasm_c_api in wasm.h / required for limits) involves a whole tree of c header files in the c-api crate. The approach we used previously, prefixing individual files via genrule, doesn't work with globs for all the required files (as bazel requires declaring all genrule outputs), so the toil and complexity of the old build would be increased significantly to support limits.

Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

Ack. I'm considering this a "change how we build wasmtime" PR, which makes sense semantically. While they are separable, due to higher priority tasks on my todo list, I simply don't have a couple hours to spare to split this.

[PiotrSikora]

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

I don't believe that was ever the case, and I cannot find a code that would support that.

Do you have a link? If not, could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime - it can be either this PR or off the last PR in the stacked series.

[leonm1]

could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime

envoyproxy/envoy#43920, made some additional minor changes to fix the build.

[PiotrSikora]

LGTM (assuming that all Envoy tests pass), but could you shorten the commit subject? Perhaps drop the "crates_universe-provided" part?

Also, after taking another look at it, I'd probably remove the prefixed version altogether, since I don't think anybody is shipping with multiple engines (but maybe I'm wrong?).

[PiotrSikora]

Circling back on the prefixed variant and use of multiengine in real products - it looks that ATS supports multiple Wasm engines, but it builds them using standard tools, and not our Bazel build process, so it doesn't get the prefixed version anyway (and it has a warning about conflict between Wasmtime & WAMR due to the same exported symbols).

[leonm1]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

[PiotrSikora]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

The only value from prefixed variant (assuming nobody is using it in production) is that we could do:

bazelisk test --define engine=multiengine //test/...

and it would run tests across all the Wasm engines, instead of requiring per-engine invocation.

[leonm1]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

[PiotrSikora]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

Oh, no... it's not a blocker or a requirement (I already approved this PR) - I was just wondering if it provides any value, since there is clearly some minimal maintenance overhead.

[leonm1]

Always uses prefixed wasmtime-c-api-impl, otherwise the prefixed implementation bitrots. Prefixes the headers in the repo rule to allow for globbing, which is not possible in a genrule.

The crates_vendor-provided wasmtime-c-api-impl provided build allows us to upgrade wasmtime solely by changing the version number in the repositories.bzl and Cargo.toml files.

Build off of #496

[PiotrSikora]

A few high-level comments:

1. Why? Looking at the changes, the benefits don't seem very obvious.
2. Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?
3. Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

[leonm1]

Why? Looking at the changes, the benefits don't seem very obvious.

Two things:

1. Updating wasmtime resulted in our rust_static_library in wasmtime.BUILD failing to build due to missing dependencies. With this current approach, the dependency tree is resolved by Cargo and we simply wrap it in rust_static_library. Updates which pull in new dependencies are trivial after this change and (in theory) will never require hunting down missing dependencies.
2. For prefixing itself, using parts of the wasmtime_c_api (as opposed to the raw wasm_c_api in wasm.h / required for limits) involves a whole tree of c header files in the c-api crate. The approach we used previously, prefixing individual files via genrule, doesn't work with globs for all the required files (as bazel requires declaring all genrule outputs), so the toil and complexity of the old build would be increased significantly to support limits.

Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

Ack. I'm considering this a "change how we build wasmtime" PR, which makes sense semantically. While they are separable, due to higher priority tasks on my todo list, I simply don't have a couple hours to spare to split this.

[PiotrSikora]

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

I don't believe that was ever the case, and I cannot find a code that would support that.

Do you have a link? If not, could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime - it can be either this PR or off the last PR in the stacked series.

[leonm1]

could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime

envoyproxy/envoy#43920, made some additional minor changes to fix the build.

[PiotrSikora]

LGTM (assuming that all Envoy tests pass), but could you shorten the commit subject? Perhaps drop the "crates_universe-provided" part?

Also, after taking another look at it, I'd probably remove the prefixed version altogether, since I don't think anybody is shipping with multiple engines (but maybe I'm wrong?).

[PiotrSikora]

Circling back on the prefixed variant and use of multiengine in real products - it looks that ATS supports multiple Wasm engines, but it builds them using standard tools, and not our Bazel build process, so it doesn't get the prefixed version anyway (and it has a warning about conflict between Wasmtime & WAMR due to the same exported symbols).

[leonm1]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

[PiotrSikora]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

The only value from prefixed variant (assuming nobody is using it in production) is that we could do:

bazelisk test --define engine=multiengine //test/...

and it would run tests across all the Wasm engines, instead of requiring per-engine invocation.

[leonm1]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

[PiotrSikora]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

Oh, no... it's not a blocker or a requirement (I already approved this PR) - I was just wondering if it provides any value, since there is clearly some minimal maintenance overhead.

[leonm1]

Always uses prefixed wasmtime-c-api-impl, otherwise the prefixed implementation bitrots. Prefixes the headers in the repo rule to allow for globbing, which is not possible in a genrule.

The crates_vendor-provided wasmtime-c-api-impl provided build allows us to upgrade wasmtime solely by changing the version number in the repositories.bzl and Cargo.toml files.

Build off of #496

[PiotrSikora]

A few high-level comments:

1. Why? Looking at the changes, the benefits don't seem very obvious.
2. Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?
3. Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

[leonm1]

Why? Looking at the changes, the benefits don't seem very obvious.

Two things:

1. Updating wasmtime resulted in our rust_static_library in wasmtime.BUILD failing to build due to missing dependencies. With this current approach, the dependency tree is resolved by Cargo and we simply wrap it in rust_static_library. Updates which pull in new dependencies are trivial after this change and (in theory) will never require hunting down missing dependencies.
2. For prefixing itself, using parts of the wasmtime_c_api (as opposed to the raw wasm_c_api in wasm.h / required for limits) involves a whole tree of c header files in the c-api crate. The approach we used previously, prefixing individual files via genrule, doesn't work with globs for all the required files (as bazel requires declaring all genrule outputs), so the toil and complexity of the old build would be increased significantly to support limits.

Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

Ack. I'm considering this a "change how we build wasmtime" PR, which makes sense semantically. While they are separable, due to higher priority tasks on my todo list, I simply don't have a couple hours to spare to split this.

[PiotrSikora]

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

I don't believe that was ever the case, and I cannot find a code that would support that.

Do you have a link? If not, could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime - it can be either this PR or off the last PR in the stacked series.

[leonm1]

could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime

envoyproxy/envoy#43920, made some additional minor changes to fix the build.

[PiotrSikora]

LGTM (assuming that all Envoy tests pass), but could you shorten the commit subject? Perhaps drop the "crates_universe-provided" part?

Also, after taking another look at it, I'd probably remove the prefixed version altogether, since I don't think anybody is shipping with multiple engines (but maybe I'm wrong?).

[PiotrSikora]

Circling back on the prefixed variant and use of multiengine in real products - it looks that ATS supports multiple Wasm engines, but it builds them using standard tools, and not our Bazel build process, so it doesn't get the prefixed version anyway (and it has a warning about conflict between Wasmtime & WAMR due to the same exported symbols).

[leonm1]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

[PiotrSikora]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

The only value from prefixed variant (assuming nobody is using it in production) is that we could do:

bazelisk test --define engine=multiengine //test/...

and it would run tests across all the Wasm engines, instead of requiring per-engine invocation.

[leonm1]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

[PiotrSikora]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

Oh, no... it's not a blocker or a requirement (I already approved this PR) - I was just wondering if it provides any value, since there is clearly some minimal maintenance overhead.

[leonm1]

Always uses prefixed wasmtime-c-api-impl, otherwise the prefixed implementation bitrots. Prefixes the headers in the repo rule to allow for globbing, which is not possible in a genrule.

The crates_vendor-provided wasmtime-c-api-impl provided build allows us to upgrade wasmtime solely by changing the version number in the repositories.bzl and Cargo.toml files.

Build off of #496

[PiotrSikora]

A few high-level comments:

1. Why? Looking at the changes, the benefits don't seem very obvious.
2. Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?
3. Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

[leonm1]

Why? Looking at the changes, the benefits don't seem very obvious.

Two things:

1. Updating wasmtime resulted in our rust_static_library in wasmtime.BUILD failing to build due to missing dependencies. With this current approach, the dependency tree is resolved by Cargo and we simply wrap it in rust_static_library. Updates which pull in new dependencies are trivial after this change and (in theory) will never require hunting down missing dependencies.
2. For prefixing itself, using parts of the wasmtime_c_api (as opposed to the raw wasm_c_api in wasm.h / required for limits) involves a whole tree of c header files in the c-api crate. The approach we used previously, prefixing individual files via genrule, doesn't work with globs for all the required files (as bazel requires declaring all genrule outputs), so the toil and complexity of the old build would be increased significantly to support limits.

Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

Ack. I'm considering this a "change how we build wasmtime" PR, which makes sense semantically. While they are separable, due to higher priority tasks on my todo list, I simply don't have a couple hours to spare to split this.

[PiotrSikora]

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

I don't believe that was ever the case, and I cannot find a code that would support that.

Do you have a link? If not, could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime - it can be either this PR or off the last PR in the stacked series.

[leonm1]

could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime

envoyproxy/envoy#43920, made some additional minor changes to fix the build.

[PiotrSikora]

LGTM (assuming that all Envoy tests pass), but could you shorten the commit subject? Perhaps drop the "crates_universe-provided" part?

Also, after taking another look at it, I'd probably remove the prefixed version altogether, since I don't think anybody is shipping with multiple engines (but maybe I'm wrong?).

[PiotrSikora]

Circling back on the prefixed variant and use of multiengine in real products - it looks that ATS supports multiple Wasm engines, but it builds them using standard tools, and not our Bazel build process, so it doesn't get the prefixed version anyway (and it has a warning about conflict between Wasmtime & WAMR due to the same exported symbols).

[leonm1]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

[PiotrSikora]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

The only value from prefixed variant (assuming nobody is using it in production) is that we could do:

bazelisk test --define engine=multiengine //test/...

and it would run tests across all the Wasm engines, instead of requiring per-engine invocation.

[leonm1]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

[PiotrSikora]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

Oh, no... it's not a blocker or a requirement (I already approved this PR) - I was just wondering if it provides any value, since there is clearly some minimal maintenance overhead.

[leonm1]

Always uses prefixed wasmtime-c-api-impl, otherwise the prefixed implementation bitrots. Prefixes the headers in the repo rule to allow for globbing, which is not possible in a genrule.

The crates_vendor-provided wasmtime-c-api-impl provided build allows us to upgrade wasmtime solely by changing the version number in the repositories.bzl and Cargo.toml files.

Build off of #496

[PiotrSikora]

A few high-level comments:

1. Why? Looking at the changes, the benefits don't seem very obvious.
2. Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?
3. Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

[leonm1]

Why? Looking at the changes, the benefits don't seem very obvious.

Two things:

1. Updating wasmtime resulted in our rust_static_library in wasmtime.BUILD failing to build due to missing dependencies. With this current approach, the dependency tree is resolved by Cargo and we simply wrap it in rust_static_library. Updates which pull in new dependencies are trivial after this change and (in theory) will never require hunting down missing dependencies.
2. For prefixing itself, using parts of the wasmtime_c_api (as opposed to the raw wasm_c_api in wasm.h / required for limits) involves a whole tree of c header files in the c-api crate. The approach we used previously, prefixing individual files via genrule, doesn't work with globs for all the required files (as bazel requires declaring all genrule outputs), so the toil and complexity of the old build would be increased significantly to support limits.

Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

Ack. I'm considering this a "change how we build wasmtime" PR, which makes sense semantically. While they are separable, due to higher priority tasks on my todo list, I simply don't have a couple hours to spare to split this.

[PiotrSikora]

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

I don't believe that was ever the case, and I cannot find a code that would support that.

Do you have a link? If not, could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime - it can be either this PR or off the last PR in the stacked series.

[leonm1]

could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime

envoyproxy/envoy#43920, made some additional minor changes to fix the build.

[PiotrSikora]

LGTM (assuming that all Envoy tests pass), but could you shorten the commit subject? Perhaps drop the "crates_universe-provided" part?

Also, after taking another look at it, I'd probably remove the prefixed version altogether, since I don't think anybody is shipping with multiple engines (but maybe I'm wrong?).

[PiotrSikora]

Circling back on the prefixed variant and use of multiengine in real products - it looks that ATS supports multiple Wasm engines, but it builds them using standard tools, and not our Bazel build process, so it doesn't get the prefixed version anyway (and it has a warning about conflict between Wasmtime & WAMR due to the same exported symbols).

[leonm1]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

[PiotrSikora]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

The only value from prefixed variant (assuming nobody is using it in production) is that we could do:

bazelisk test --define engine=multiengine //test/...

and it would run tests across all the Wasm engines, instead of requiring per-engine invocation.

[leonm1]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

[PiotrSikora]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

Oh, no... it's not a blocker or a requirement (I already approved this PR) - I was just wondering if it provides any value, since there is clearly some minimal maintenance overhead.

[leonm1]

Always uses prefixed wasmtime-c-api-impl, otherwise the prefixed implementation bitrots. Prefixes the headers in the repo rule to allow for globbing, which is not possible in a genrule.

The crates_vendor-provided wasmtime-c-api-impl provided build allows us to upgrade wasmtime solely by changing the version number in the repositories.bzl and Cargo.toml files.

Build off of #496

[PiotrSikora]

A few high-level comments:

1. Why? Looking at the changes, the benefits don't seem very obvious.
2. Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?
3. Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

[leonm1]

Why? Looking at the changes, the benefits don't seem very obvious.

Two things:

1. Updating wasmtime resulted in our rust_static_library in wasmtime.BUILD failing to build due to missing dependencies. With this current approach, the dependency tree is resolved by Cargo and we simply wrap it in rust_static_library. Updates which pull in new dependencies are trivial after this change and (in theory) will never require hunting down missing dependencies.
2. For prefixing itself, using parts of the wasmtime_c_api (as opposed to the raw wasm_c_api in wasm.h / required for limits) involves a whole tree of c header files in the c-api crate. The approach we used previously, prefixing individual files via genrule, doesn't work with globs for all the required files (as bazel requires declaring all genrule outputs), so the toil and complexity of the old build would be increased significantly to support limits.

Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

Ack. I'm considering this a "change how we build wasmtime" PR, which makes sense semantically. While they are separable, due to higher priority tasks on my todo list, I simply don't have a couple hours to spare to split this.

[PiotrSikora]

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

I don't believe that was ever the case, and I cannot find a code that would support that.

Do you have a link? If not, could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime - it can be either this PR or off the last PR in the stacked series.

[leonm1]

could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime

envoyproxy/envoy#43920, made some additional minor changes to fix the build.

[PiotrSikora]

LGTM (assuming that all Envoy tests pass), but could you shorten the commit subject? Perhaps drop the "crates_universe-provided" part?

Also, after taking another look at it, I'd probably remove the prefixed version altogether, since I don't think anybody is shipping with multiple engines (but maybe I'm wrong?).

[PiotrSikora]

Circling back on the prefixed variant and use of multiengine in real products - it looks that ATS supports multiple Wasm engines, but it builds them using standard tools, and not our Bazel build process, so it doesn't get the prefixed version anyway (and it has a warning about conflict between Wasmtime & WAMR due to the same exported symbols).

[leonm1]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

[PiotrSikora]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

The only value from prefixed variant (assuming nobody is using it in production) is that we could do:

bazelisk test --define engine=multiengine //test/...

and it would run tests across all the Wasm engines, instead of requiring per-engine invocation.

[leonm1]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

[PiotrSikora]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

Oh, no... it's not a blocker or a requirement (I already approved this PR) - I was just wondering if it provides any value, since there is clearly some minimal maintenance overhead.

[leonm1]

Always uses prefixed wasmtime-c-api-impl, otherwise the prefixed implementation bitrots. Prefixes the headers in the repo rule to allow for globbing, which is not possible in a genrule.

The crates_vendor-provided wasmtime-c-api-impl provided build allows us to upgrade wasmtime solely by changing the version number in the repositories.bzl and Cargo.toml files.

Build off of #496

[PiotrSikora]

A few high-level comments:

1. Why? Looking at the changes, the benefits don't seem very obvious.
2. Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?
3. Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

[leonm1]

Why? Looking at the changes, the benefits don't seem very obvious.

Two things:

1. Updating wasmtime resulted in our rust_static_library in wasmtime.BUILD failing to build due to missing dependencies. With this current approach, the dependency tree is resolved by Cargo and we simply wrap it in rust_static_library. Updates which pull in new dependencies are trivial after this change and (in theory) will never require hunting down missing dependencies.
2. For prefixing itself, using parts of the wasmtime_c_api (as opposed to the raw wasm_c_api in wasm.h / required for limits) involves a whole tree of c header files in the c-api crate. The approach we used previously, prefixing individual files via genrule, doesn't work with globs for all the required files (as bazel requires declaring all genrule outputs), so the toil and complexity of the old build would be increased significantly to support limits.

Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

Ack. I'm considering this a "change how we build wasmtime" PR, which makes sense semantically. While they are separable, due to higher priority tasks on my todo list, I simply don't have a couple hours to spare to split this.

[PiotrSikora]

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

I don't believe that was ever the case, and I cannot find a code that would support that.

Do you have a link? If not, could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime - it can be either this PR or off the last PR in the stacked series.

[leonm1]

could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime

envoyproxy/envoy#43920, made some additional minor changes to fix the build.

[PiotrSikora]

LGTM (assuming that all Envoy tests pass), but could you shorten the commit subject? Perhaps drop the "crates_universe-provided" part?

Also, after taking another look at it, I'd probably remove the prefixed version altogether, since I don't think anybody is shipping with multiple engines (but maybe I'm wrong?).

[PiotrSikora]

Circling back on the prefixed variant and use of multiengine in real products - it looks that ATS supports multiple Wasm engines, but it builds them using standard tools, and not our Bazel build process, so it doesn't get the prefixed version anyway (and it has a warning about conflict between Wasmtime & WAMR due to the same exported symbols).

[leonm1]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

[PiotrSikora]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

The only value from prefixed variant (assuming nobody is using it in production) is that we could do:

bazelisk test --define engine=multiengine //test/...

and it would run tests across all the Wasm engines, instead of requiring per-engine invocation.

[leonm1]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

[PiotrSikora]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

Oh, no... it's not a blocker or a requirement (I already approved this PR) - I was just wondering if it provides any value, since there is clearly some minimal maintenance overhead.

[leonm1]

Always uses prefixed wasmtime-c-api-impl, otherwise the prefixed implementation bitrots. Prefixes the headers in the repo rule to allow for globbing, which is not possible in a genrule.

The crates_vendor-provided wasmtime-c-api-impl provided build allows us to upgrade wasmtime solely by changing the version number in the repositories.bzl and Cargo.toml files.

Build off of #496

[PiotrSikora]

A few high-level comments:

1. Why? Looking at the changes, the benefits don't seem very obvious.
2. Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?
3. Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

[leonm1]

Why? Looking at the changes, the benefits don't seem very obvious.

Two things:

1. Updating wasmtime resulted in our rust_static_library in wasmtime.BUILD failing to build due to missing dependencies. With this current approach, the dependency tree is resolved by Cargo and we simply wrap it in rust_static_library. Updates which pull in new dependencies are trivial after this change and (in theory) will never require hunting down missing dependencies.
2. For prefixing itself, using parts of the wasmtime_c_api (as opposed to the raw wasm_c_api in wasm.h / required for limits) involves a whole tree of c header files in the c-api crate. The approach we used previously, prefixing individual files via genrule, doesn't work with globs for all the required files (as bazel requires declaring all genrule outputs), so the toil and complexity of the old build would be increased significantly to support limits.

Always using prefixed variant seems fine (although, I think there was a reason why it wasn't the default - perhaps something with Envoy build?), but is that something that's at all relevant with the move to Wasmtime C++ API in #503?

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

Using wasmtime-c-api-impl and always using prefixed variant are pretty separate changes, so might be good to split them off (but if they are too intertwined and would result in a throwaway work, then we can keep this as-is).

Ack. I'm considering this a "change how we build wasmtime" PR, which makes sense semantically. While they are separable, due to higher priority tasks on my todo list, I simply don't have a couple hours to spare to split this.

[PiotrSikora]

Envoy build itself ran the multi runtime tests, so I don't think it required the non-prefixed version.

I don't believe that was ever the case, and I cannot find a code that would support that.

Do you have a link? If not, could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime - it can be either this PR or off the last PR in the stacked series.

[leonm1]

could you open a draft PR against Envoy (you'll need to do that anyway at some point) and verify that it builds fine with Wasmtime

envoyproxy/envoy#43920, made some additional minor changes to fix the build.

[PiotrSikora]

LGTM (assuming that all Envoy tests pass), but could you shorten the commit subject? Perhaps drop the "crates_universe-provided" part?

Also, after taking another look at it, I'd probably remove the prefixed version altogether, since I don't think anybody is shipping with multiple engines (but maybe I'm wrong?).

[PiotrSikora]

Circling back on the prefixed variant and use of multiengine in real products - it looks that ATS supports multiple Wasm engines, but it builds them using standard tools, and not our Bazel build process, so it doesn't get the prefixed version anyway (and it has a warning about conflict between Wasmtime & WAMR due to the same exported symbols).

[leonm1]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

[PiotrSikora]

The commits still exist, so I am okay if we drop the prefixed variant in this PR and can discuss adding it back in conjunction with supporting it in Envoy.

The only value from prefixed variant (assuming nobody is using it in production) is that we could do:

bazelisk test --define engine=multiengine //test/...

and it would run tests across all the Wasm engines, instead of requiring per-engine invocation.

[leonm1]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

[PiotrSikora]

I currently use this regularly:

bazelisk test --define engine=multi //test/...

However, in the interest of getting the wasmtime update merged, I'm happy to drop the prefixed variant for now, and reintroduce it "properly" later.

Oh, no... it's not a blocker or a requirement (I already approved this PR) - I was just wondering if it provides any value, since there is clearly some minimal maintenance overhead.