Support gazelle plugin when using bzlmod

[aignas]

🚀 feature request

Creating this issue to encompass all of the PRs related to this feature, as the single research PR (#961) got a little bit too big for easy review.

Relevant Rules

gazelle Python plugin.

Description

Currently the gazelle plugin does not work if using bzlmod with bazel >= 6.0.0. This is due to:

• Gazelle and go toolchain is not present in MODULE.bazel of rules_python, which means that the plugin reference fails.
• The all_whl_requirements have labels, that do not work in bzlmod due to repository remapping. Hence, the modules_mapping rule fails unless we do something about it.
• Gazelle will generate dependency labels in the targets that do not work in bzlmod due to repository remapping.

Describe the solution you'd like

As researched in #961, we can fix these three issues separately:

1. Adding gazelle and go as bzlmod deps.
2. Changing the contents of the all_whl_requirements and all_requirements based on wether bzlmod is used.
3. Have extra declaratives for changing the generated labels to work with bzlmod.
4. Have extra aliases in the @pip repo that work with both, bzlmod and classic gazelle. This might be similar to feat: add alias generation for pip parse #814, however this is not a requirement to make gazelle with bzlmod work.

Describe alternatives you've considered

Rejected alternatives:

• Having a flag in gazelle_python.yaml manifest, which would say that we are using bzlmod. The manifest should be the same irrespective of whether we are using bzlmod or not due to the same reason why we may want to have aliases in @pip repo -- because we want to enable a smooth transition to bzlmod as described in the migration guide

[aignas]

Some ideas that came from the comment in #970.

Regarding the metadata for gazelle to consume from the requirements.bzl file, I see that there are 2 pieces for gazelle to work:

• modules_mapping creates a mapping between import names and Python package names. It downloads the whl files to get this information, so it depends on the said whl targets.
• gazelle_python.yaml manifest generation depends on the mapping and a few extra variables, which help it map the Python package name back to the bazel label for the pkg target.

Thinking from outside-in, I'd like to use something like below in my BUILD.bazel files:

# BUILD.bazel contents
load("@pip//:gazelle.bzl", "gazelle_python_manifest")
load("@gazelle//:def.bzl", "gazelle", "gazelle_binary")
load("@rules_python//gazelle:defs.bzl", "GAZELLE_PYTHON_RUNTIME_DEPS")

# This macro could read the metadata within the '@pip//:gazelle.bzl' which would have the
# following info:
# - The whl library list needed for gazelle.
# - The name of the pip repository.
# - The pip repo naming convention, thus eliminating the need for some of the
# PRs in the series.
# - The '//:requirements_lock.txt' could be also infered as we pass it to `pip_parse` anyway.
gazelle_python_manifest(
 name = "gazelle_python_manifest",
 extend_exclude_patterns = [
 "(\\tests)+", # There are many packages on PyPI with tests in their wheels
 ],
)

gazelle_binary(
 name = "gazelle_bin",
 extend_languages = [
 "@rules_python//gazelle",
 ],
)

gazelle(
 name = "gazelle",
 data = GAZELLE_PYTHON_RUNTIME_DEPS,
 binary = ":gazelle_bin",
)

I'll play around with the idea of having gazelle macro come from the output of pip_parse.

[alexeagle]

For your #1, rules_python must not introduce a Go dependency for all users. The situation is the same as for bazel-skylib, which has a Gazelle extension for creating bzl_library targets. The solution there is a submodule introduced in bazelbuild/bazel-skylib@60abca8#diff-7490f090fcbdb5d3b486333531265e0f0ad157dd5cadc964b3c98698021f3ced so I suspect rules_python should have something similar.

[aignas]

Thanks @alexeagle, will change the approach in #968. I was under an impression, that the dependencies will be lazy-fetched anyway, so there is no harm.