A Better rules_go

Table of Contents

I’ve recently been working with Pulumi¹ in my Bazel monorepo, using Go as the runtime language and realized that the out of the box experience with the go_library, go_binary, etc, rules sort of sucks.

So I made it better.

TLDR
#

Before:

_deps = [
    "//src/go/comptime",
    "@com_github_pulumi_pulumi_sdk_v3//go/pulumi",
    "@com_github_pulumi_pulumi_sdk_v3//go/pulumi/config",
    "@com_github_pulumi_pulumi_vault_sdk_v7//go/vault",
    "@com_github_pulumi_pulumi_vault_sdk_v7//go/vault/pkisecret",
]

go_binary(
    name        = "openbao-pulumi-runtime",
    srcs        = ["main.go"],
    deps        = _deps + [
        ":config",
        ":pki",
    ],
)

go_library(
    name       = "config",
    srcs       = glob(["config/*.go"]),
    deps       = _deps,
    importpath = "my.internal/pulumi/openbao/config",
)

go_library(
    name       = "pki",
    srcs       = glob(["pki/*.go"]),
    deps       = _deps,
    importpath = "my.internal/pulumi/openbao/pki",
)

After:

_deps = [
    "/comptime",
    "github.com/pulumi/pulumi/sdk/v3:go/pulumi",
    "github.com/pulumi/pulumi/sdk/v3:go/pulumi/config",
    "github.com/pulumi/pulumi-vault/sdk/v7:go/vault",
    "github.com/pulumi/pulumi-vault/sdk/v7:go/vault/pkisecret",
]

go.binary(
    name = "openbao-pulumi-runtime",
    srcs = ["main.go"],
    deps = _deps + [
        ":config",
        ":pki",
    ],
)

go.library(
    name = "config",
    srcs = glob(["config/*.go"]),
    deps = _deps,
)

go.library(
    name = "pki",
    srcs = glob(["pki/*.go"]),
    deps = _deps,
)

Major points being:

Usage of normal(ish) go module strings over trying to guess how exactly gazelle is naming your Bazel repo
(And also some shorthands for in-monorepo go deps)
No need to specify importpaths for each go_library!

How it works
#

So this is basically a combination of three underlying functions:

Translating a given go module string into the correct Bazel label
Adding a bit of structured string manipulation to deps resolution
Calculating the importpath via Label resolution

Translate
#

Starting with the first point, we’re basically reimplementing ImportPathToBazelRepoName from gazelle², but in starlark:

def _go_pkg_to_repo_name(pkg):
    """
    Emulate what @gazelle does to map go pkg paths to bazel repository names
    """.strip()
    pkg = pkg.lower()
    parts = pkg.split("/")
    labels = parts[0].split(".")

    rlabels = []
    for i in range(len(labels)):
        rlabels.append(labels[len(labels) - i - 1])

    return _strip_nonwords(".".join(rlabels + parts[1:]))

_replacements=[
    ".",
    "-",
    "/",
    "@",
    "~",
    "+",
]
def _strip_nonwords(s):
    out = s

    for r in _replacements:
        out = out.replace(r, "_")

    for _ in range(10):
        if "__" not in out:
            break
        out = out.replace("__", "_")

    return out

Dep strings
#

Next, we do some string manipulation (a lot of this is omitted for brevity)

def _go_expand_dep_string(dep, aliases):
    # An absolute or package local repository label -- pass this through directly
    if dep.startswith("@//") or dep.startswith("//") or dep.startswith(":"):
        return dep

    # An alias
    if dep.startswith("@"):
        alias = _go_expand_dep_alias(dep, aliases)
        if alias:
            return _go_third_party_module_to_label(alias)

        fail("unknown alias in Go dep: '%s'. Known aliases are: %s" % (dep, aliases.keys()))

    # A third_party vendored go dep
    if dep.startswith("/third_party/"):
        return "//third_party/go/vendor" + dep.removeprefix("/third_party")

    # A first party go dep
    if dep.startswith("/"):
        return "//src/go" + dep

    # Else we assume its a third_party module
    return _go_third_party_module_to_label(dep)

Quoting the commit that introduced this function:

@//<rest...>, //<rest...>, :<rest...> Anything that is a valid absolute label is passed through directly
@<alias>:<rest...> Callers may reference any known alias via starting their dep spec with an ‘@’. These are primarily for shortening otherwise long module specs, but can be used for any Go module string.
Aliases can be created via the make_alias function, and passed into make_go_defs via the aliases dict kwarg.
/third_party/<rest...> Reference a module in //third_party/go/vendor, for any third_party deps we have vendored ourselves.
/<rest...> Reference a first party module from //src/go
Everything else is interpreted as a Go module string and expanded as such
There is one key difference to normal Go module strings: You must separate the top level module (where the go.mod sits) from any subpackages by a ‘:’.

Calculating importpaths
#

Lastly, we automatically calculate each go_* rule’s importpath, via a provided modbase, anchor and path. These roughly equal the module stanza in your go.mod (e.g my.internal), the current package (via native.package_relative_label) and the name of the go_* rule that is being created.

def _go_relative_import_path(parent, strip, anchor, path):
    p = anchor.package.removeprefix(strip).removeprefix("/").removesuffix("/")

    # Handle cases where the Bazel package name and the importpath right most
    # segment are equal
    #
    # Most commonly in cases where the go_library's name equals the Bazel
    # package name
    #
    # Ex: //src/go/foo:foo
    if xtpath.basename(p) == path:
        return xtpath.join(parent, p)

    # If the importpath is a subpath under the current Bazel package, strip the
    # package name before concating so we don't duplicate it
    #
    # Ex: //src/go/foo:foo/config
    if "/" in path and path.startswith(xtpath.basename(p)):
        return xtpath.join(parent, p, path.removeprefix(xtpath.basename(p)).removeprefix("/")).rstrip("/")

    # Otherwise we can just do things normally
    #
    # Ex: //src/go/foo:http
    return xtpath.join(parent, p, path).rstrip("/")

Putting it all together
#

To tie it all together at the top level, we create a make_go_defs function which returns a struct with members that wrap each upstream rule_go rule:

def make_go_defs(modbase, strip, aliases):
    """
    Return a struct of rules_go rule wrappers, that utilize the given modbase,
    strip and aliases.

    ...snipped for brevity
    """.strip()
    return struct(
        binary = lambda **kwargs: make_go_binary_wrapper(
            modbase = modbase,
            strip   = strip,
            aliases = aliases,
            **kwargs
        ),
        library = lambda **kwargs: make_go_library_wrapper(
            modbase = modbase,
            strip   = strip,
            aliases = aliases,
            **kwargs
        ),
        test = lambda **kwargs: make_go_test_wrapper(
            modbase = modbase,
            strip   = strip,
            aliases = aliases,
            **kwargs
        ),
        # ... rest of the rules_go rules as reexports
    )

Which is then used like so:

load("//lib/bazel/go:defs.bzl", "make_go_defs", mkalias = "make_alias")

_aliases = {
    # ...snipped for brevity
}

# This is what the go.binary, go.library stuff from the TLDR is referring to
go = make_go_defs(
    modbase = "my.internal",
    strip   = "src/go",
    aliases = _aliases,
)

All in all, I think it makes it much nicer to use rules_go / Go in Bazel.

Closing
#

Anyone else come across a similar issue when using rules_go? I figure at least the issue of writing importpaths everywhere is something that anyone using Go + Bazel encountered quickly and probably came up with a fix for.

None of this publicly available, but if you’re interested in it, I’m not opposed to making it into a Bazel module for you.

Maybe my rules_pulumi rule set should be the subject of another post? ↩︎
https://github.com/bazel-contrib/bazel-gazelle/blob/v0.49.0/label/label.go#L236 ↩︎

Reply by Email

TLDR#

How it works#

Translate#

Dep strings#

Calculating importpaths#

Putting it all together#

Closing#