feat(bazel): wire up prebuilt protoc toolchain #24115

zhangskz · 2025-11-17T20:23:54Z

We probably still want this updated to ensure it doesn't end up too ancient. Perhaps this can get updated via GHA too triggered by a new release to update this when there are new minor releases (e.g. v33.0) in at least main and the corresponding release branch.

Sure, that's possible, I can propose some GHA automation after this lands.

zhangskz · 2025-11-17T20:28:45Z

This will need to be set to false for most of our CI tests at least.

Sure, I wouldn't expect the default for end-users to be motivated similarly to how protobuf team will test. In fact, this default could vary between this repo and the Bazel module we publish, using the BCR patches feature.

-Original file line number
+Diff line change
@@ Expand Up / @@ -37,6 +37,7 @@ mkdir -p ${PREFIX}/bazel/private @@
     cat >${INTEGRITY_FILE} <<EOF
     "Generated during release by release_prep.sh"
+    RELEASE_VERSION="${TAG}"
     RELEASED_BINARY_INTEGRITY = $(
     curl -s https://api.github.com/repos/protocolbuffers/protobuf/releases/tags/${TAG} \
       | jq -f <(echo "$filter_releases")
@@ Expand Down @@

-Original file line number
+Diff line change
@@ Expand Up / @@ -29,7 +29,12 @@ jobs: @@
             runner: [ ubuntu, windows, macos ]
             bazelversion: [ '7.6.1', '8.0.0' ]
             bzlmod: [ true, false ]
-            toolchain_resolution: [ "", "--incompatible_enable_proto_toolchain_resolution=true" ]
+            toolchain_resolution:
+              # Default flags, uses from-source protoc
+              - ""
+              # still uses from-source protoc unless
+              # --@com_google_protobuf//bazel/toolchains:prefer_prebuilt_protoc is set
+              - "--incompatible_enable_proto_toolchain_resolution=true"
         runs-on: ${{ matrix.runner }}-latest
         name: ${{ matrix.continuous-only && inputs.continuous-prefix || '' }} Examples ${{ matrix.runner }} ${{ matrix.bazelversion }}${{ matrix.bzlmod && ' (bzlmod)' || '' }} ${{ matrix.toolchain_resolution && ' (toolchain resolution)' || '' }}
         steps:
@@ Expand Down Expand Up / @@ -72,3 +77,14 @@ jobs: @@
               bash: >
                 cd examples;
                 bazel build //... @com_google_protobuf-examples-with-hyphen//... $BAZEL_FLAGS --enable_bzlmod=${{ matrix.bzlmod }} --enable_workspace=${{ !matrix.bzlmod }} ${{ matrix.toolchain_resolution }};
+          - name: Prebuilt test
+            if: ${{ matrix.bzlmod && (!matrix.continuous-only || inputs.continuous-run) }}
+            uses: protocolbuffers/protobuf-ci/bazel@v5
+            with:
+              credentials: ${{ secrets.GAR_SERVICE_ACCOUNT }}
+              bazel-cache: examples-prebuilt-${{ matrix.bazelversion }}-${{ matrix.toolchain_resolution }}
+              version: ${{ matrix.bazelversion }}
+              bash: >
+                cd examples/example_without_cc_toolchain;
+                bazel build //... $BAZEL_FLAGS --enable_bzlmod=true ${{ matrix.toolchain_resolution }};

-Original file line number
+Diff line change
@@ Expand Up / @@ -115,7 +115,27 @@ register_toolchains( @@
         dev_dependency = True,
     )
-    # Proto toolchains
+    # Define toolchains that use pre-built protoc binaries.
+    prebuilt_protoc = use_extension("//bazel/private:prebuilt_protoc_extension.bzl", "protoc")
+    use_repo(
+        prebuilt_protoc,
+        "prebuilt_protoc.linux_aarch_64",
+        "prebuilt_protoc.osx_aarch_64",
+        "prebuilt_protoc.linux_ppcle_64",
+        "prebuilt_protoc.linux_s390_64",
+        "prebuilt_protoc.linux_x86_32",
+        "prebuilt_protoc.linux_x86_64",
+        "prebuilt_protoc.osx_x86_64",
+        "prebuilt_protoc.win32",
+        "prebuilt_protoc.win64",
+    )
+    # However this registration only matters if the config_setting for prefer_prebuilt_protoc is true,
+    # using --@protobuf//bazel/toolchains:prefer_prebuilt_protoc
+    register_toolchains("//bazel/private/toolchains/prebuilt:all")
+    # From-source protobuf toolchains
+    # Fallback if nothing is already registered
     register_toolchains("//bazel/private/toolchains:all")
     SUPPORTED_PYTHON_VERSIONS = [
@@ Expand Down @@

-Original file line number
+Diff line change
@@ -0,0 +1,15 @@
+    "Module extensions for use under bzlmod"
+    load("@bazel_skylib//lib:modules.bzl", "modules")
+    load("//toolchain:platforms.bzl", "PROTOBUF_PLATFORMS")
+    load("//bazel/private:prebuilt_protoc_toolchain.bzl", "prebuilt_protoc_repo")
+    def create_all_toolchain_repos(name = "prebuilt_protoc"):
+        for platform in PROTOBUF_PLATFORMS.keys():
+            prebuilt_protoc_repo(
+                # We must replace hyphen with underscore to workaround rules_python py_proto_library constraint
+                name = ".".join([name, platform.replace("-", "_")]),
+                platform = platform,
+            )
+    protoc = modules.as_extension(create_all_toolchain_repos)

-Original file line number
+Diff line change
@@ -0,0 +1,57 @@
+    "Repository rule that downloads a pre-compiled protoc from our official release for a single platform."
+    load(":prebuilt_tool_integrity.bzl", "RELEASED_BINARY_INTEGRITY", "RELEASE_VERSION")
+    load("//toolchain:platforms.bzl", "PROTOBUF_PLATFORMS")
+    def release_version_to_artifact_name(release_version, platform):
+        # versions have a "v" prefix like "v28.0"
+        stripped_version = release_version.removeprefix("v")
+        # release candidate versions like "v29.0-rc3" have artifact names
+        # like "protoc-29.0-rc-3-osx-x86_64.zip"
+        artifact_version = stripped_version.replace("rc", "rc-")
+        return "{}-{}-{}.zip".format(
+            "protoc",
+            artifact_version,
+            platform,
+        )
+    def _prebuilt_protoc_repo_impl(rctx):
+        filename = release_version_to_artifact_name(
+            RELEASE_VERSION,
+            rctx.attr.platform,
+        )
+        rctx.download_and_extract(
+            url = "https://github.com/protocolbuffers/protobuf/releases/download/{}/{}".format(
+                RELEASE_VERSION,
+                filename,
+            ),
+            sha256 = RELEASED_BINARY_INTEGRITY[filename],
+        )
+        rctx.file("BUILD.bazel", """\
+    # Generated by @protobuf//bazel/private:prebuilt_protoc_toolchain.bzl
+    load("@com_google_protobuf//bazel/toolchains:proto_toolchain.bzl", "proto_toolchain")
+    package(default_visibility = ["//visibility:public"])
+    proto_toolchain(
+        name = "prebuilt_protoc_toolchain",
+        proto_compiler = "{protoc_label}",
+    )
+    """.format(
+            protoc_label = "bin/protoc.exe" if rctx.attr.platform.startswith("win") else "bin/protoc",
+        ))
+    prebuilt_protoc_repo = repository_rule(
+        doc = "Download a pre-built protoc and create a concrete toolchains for it",
+        implementation = _prebuilt_protoc_repo_impl,
+        attrs = {
+            "platform": attr.string(
+                doc = "A platform that protobuf ships a release for",
+                mandatory = True,
+                values = PROTOBUF_PLATFORMS.keys(),
+            ),
+        },
+    )

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(bazel): wire up prebuilt protoc toolchain #24115

Diff view

Diff view

There are no files selected for viewing

Uh oh!

zhangskz Nov 17, 2025

Uh oh!

alexeagle Nov 17, 2025

Uh oh!

Uh oh!

Uh oh!

zhangskz Nov 17, 2025

Uh oh!

alexeagle Nov 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

-Original file line number
+Diff line change
@@ Expand Up / @@ -3,18 +3,18 @@ @@
     This file contents are entirely replaced during release publishing, by .github/workflows/release_prep.sh
     so that the integrity of the prebuilt tools is included in the release artifact.
-    The checked in content is only here to allow load() statements in the sources to resolve.
+    The checked in content is only here to allow load() statements in the sources to resolve, and permit local testing.
     """
-    # Create a mapping for every tool name to the hash of /dev/null
-    NULLSHA = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
-    RELEASED_BINARY_INTEGRITY = {
-        "-".join([
-            "protoc",
-            os,
-            arch,
-        ]): NULLSHA
-        for [os, arch] in {
-            "linux": ["aarch_64", "x86_64"],
-        }
-    }
+    # An arbitrary version of protobuf that includes pre-built binaries.
+    # See /examples/example_without_cc_toolchain which uses this for testing.
+    # TODO(alexeagle): add some automation to update this version occasionally.
+    _TEST_VERSION = "v33.0"
+    _TEST_SHAS = dict()
+    # Add a couple platforms which are commonly used for testing.
+    _TEST_SHAS["protoc-33.0-linux-x86_64.zip"] = "d99c011b799e9e412064244f0be417e5d76c9b6ace13a2ac735330fa7d57ad8f"
+    _TEST_SHAS["protoc-33.0-osx-aarch_64.zip"] = "3cf55dd47118bd2efda9cd26b74f8bbbfcf5beb1bf606bc56ad4c001b543f6d3"
+    _TEST_SHAS["protoc-33.0-win64.zip"] = "3742cd49c8b6bd78b6760540367eb0ff62fa70a1032e15dafe131bfaf296986a"
+    RELEASE_VERSION = _TEST_VERSION
+    RELEASED_BINARY_INTEGRITY = _TEST_SHAS

-Original file line number
+Diff line change
@@ Expand Up / @@ -113,6 +113,7 @@ def _proto_library_impl(ctx): @@
                 default_runfiles = ctx.runfiles(),  # empty
                 data_runfiles = data_runfiles,
             ),
+            OutputGroupInfo(_validation = ctx.attr._authenticity_validation[OutputGroupInfo]._validation),
         ]
     def _process_srcs(ctx, srcs, import_prefix, strip_import_prefix):
@@ Expand Down Expand Up @@
     for use with MessageSet.
     """,
             ),
+            "_authenticity_validation": attr.label(
+                default = "//bazel/private/toolchains/prebuilt:authenticity_validation",
+                doc = "Validate that the binary registered on the toolchain is produced by protobuf team",
+            ),
             # buildifier: disable=attr-license (calling attr.license())
             "licenses": attr.license() if hasattr(attr, "license") else attr.string_list(),
             "_experimental_proto_descriptor_sets_include_source_info": attr.label(
@@ Expand Down @@

-Original file line number
+Diff line change
@@ -0,0 +1,32 @@
+    """Create lazy definitions to reference the pre-built protoc toolchains.
+    Ensures that Bazel only downloads required binaries for selected toolchains.
+    In particular, see comment below on the toolchain#toolchain attribute.
+    """
+    load(":protoc_authenticity.bzl", "protoc_authenticity")
+    load("//toolchain:platforms.bzl", "PROTOBUF_PLATFORMS")
+    [
+        toolchain(
+            name = "{}_toolchain".format(platform.replace("-", "_")),
+            exec_compatible_with = meta["compatible_with"],
+            # Toolchain resolution will only permit this toolchain if the config_setting for prefer_prebuilt_protoc is true,
+            target_settings = ["@com_google_protobuf//bazel/toolchains:prefer_prebuilt_protoc.flag_set"],
+            # Bazel does not follow this attribute during analysis, so the referenced repo
+            # will only be fetched if this toolchain is selected.
+            toolchain = "@prebuilt_protoc.{}//:prebuilt_protoc_toolchain".format(platform.replace("-", "_")),
+            toolchain_type = "@com_google_protobuf//bazel/private:proto_toolchain_type",
+        )
+        for platform, meta in PROTOBUF_PLATFORMS.items()
+    ]
+    # Support verification of user-registered toolchains
+    protoc_authenticity(
+        name = "authenticity_validation",
+        fail_on_mismatch = select({
+            "//bazel/toolchains:allow_nonstandard_protoc.flag_set": False,
+            "//conditions:default": True,
+        }),
+        visibility = ["//visibility:public"],
+    )

-Original file line number
+Diff line change
@@ -0,0 +1,68 @@
+    "Validate that the protoc binary is authentic and not spoofed by a malicious actor."
+    load("//bazel/common:proto_common.bzl", "proto_common")
+    load("//bazel/private:toolchain_helpers.bzl", "toolchains")
+    load("//bazel/private:prebuilt_tool_integrity.bzl", "RELEASE_VERSION")
+    def _protoc_authenticity_impl(ctx):
+        # When this flag is disabled, then users have no way to replace the protoc binary with their own toolchain registration.
+        # Therefore there's no validation to perform.
+        if not proto_common.INCOMPATIBLE_ENABLE_PROTO_TOOLCHAIN_RESOLUTION:
+            return [OutputGroupInfo(_validation = depset())]
+        toolchain = ctx.toolchains[toolchains.PROTO_TOOLCHAIN]
+        if not toolchain:
+            fail("Protocol compiler toolchain could not be resolved.")
+        proto_lang_toolchain_info = toolchain.proto
+        validation_output = ctx.actions.declare_file("validation_output.txt")
+        ctx.actions.run_shell(
+            outputs = [validation_output],
+            tools = [proto_lang_toolchain_info.proto_compiler],
+            command = """\
+            {protoc} --version > {validation_output}
+            grep -q -e "-dev$" {validation_output} && {{
+              echo 'WARNING: Detected a development version of protoc.
+              Development versions are not validated for authenticity.
+              To ensure a secure build, please use a released version of protoc.'
+              exit 0
+            }}
+            grep -q "^libprotoc {RELEASE_VERSION}" {validation_output} || {{
+              echo '{severity}: protoc version does not match protobuf Bazel module; we do not support this.
+              It is considered undefined behavior that is expected to break in the future even if it appears to work today.'
+              echo '{suppression_note}'
+              echo 'Expected: libprotoc {RELEASE_VERSION}'
+              echo -n 'Actual:   '
+              cat {validation_output}
+              exit {mismatch_exit_code}
+            }} >&2
+            """.format(
+                protoc = proto_lang_toolchain_info.proto_compiler.executable.path,
+                validation_output = validation_output.path,
+                RELEASE_VERSION = RELEASE_VERSION.removeprefix("v"),
+                suppression_note = (
+                    "To suppress this error, run Bazel with --@com_google_protobuf//bazel/toolchains:allow_nonstandard_protoc"
+                    if ctx.attr.fail_on_mismatch else ""
+                ),
+                mismatch_exit_code = 1 if ctx.attr.fail_on_mismatch else 0,
+                severity = "ERROR" if ctx.attr.fail_on_mismatch else "INFO",
+            ),
+        )
+        return [OutputGroupInfo(_validation = depset([validation_output]))]
+    protoc_authenticity = rule(
+        implementation = _protoc_authenticity_impl,
+        fragments = ["proto"],
+        attrs = {
+            "fail_on_mismatch": attr.bool(
+                default = True,
+                doc = "If true, the build will fail when the protoc binary does not match the expected version.",
+            ),
+        } | toolchains.if_legacy_toolchain({
+            "_proto_compiler": attr.label(
+                cfg = "exec",
+                executable = True,
+                allow_files = True,
+                default = "//src/google/protobuf/compiler:protoc_minimal",
+            ),
+        }),
+        toolchains = toolchains.use_toolchain(toolchains.PROTO_TOOLCHAIN),
+    )

-Original file line number
+Diff line change
@@ -1,13 +1,16 @@
     load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
+    load("@bazel_skylib//rules:common_settings.bzl", "bool_flag")
-    package(default_applicable_licenses = ["//:license"])
+    package(
+        default_applicable_licenses = ["//:license"],
+        default_visibility = ["//visibility:public"],
+    )
     bzl_library(
         name = "proto_toolchain_bzl",
         srcs = [
             "proto_toolchain.bzl",
         ],
-        visibility = ["//visibility:public"],
         deps = [
             "//bazel/private:proto_toolchain_rule_bzl",
             "//bazel/private:toolchain_helpers_bzl",
@@ Expand All / @@ -19,7 +22,6 @@ bzl_library( @@
         srcs = [
             "proto_lang_toolchain.bzl",
         ],
-        visibility = ["//visibility:public"],
         deps = [
             "//bazel/common:proto_common_bzl",
             "//bazel/private:proto_lang_toolchain_rule_bzl",
@@ Expand All / @@ -39,3 +41,27 @@ filegroup( @@
             "//bazel:__pkg__",
         ],
     )
+    # The public API users set
+    bool_flag(
+        name = "prefer_prebuilt_protoc",
+        # TODO(alexeagle): this should be True after the feature is vetted with some adoption
+        build_setting_default = False,
+    )
+    config_setting(
+        name = "prefer_prebuilt_protoc.flag_set",
+        flag_values = {":prefer_prebuilt_protoc": "true"},
+    )
+    # The public API users set to disable the validation action failing.
+    bool_flag(
+        name = "allow_nonstandard_protoc",
+        build_setting_default = False,
+    )
+    config_setting(
+        name = "allow_nonstandard_protoc.flag_set",
+        flag_values = {":allow_nonstandard_protoc": "true"},
+    )

-Original file line number
+Diff line change
@@ -1,2 +1,2 @@
     # Ignore the bazel symlinks
-    /bazel-*
+    bazel-*

-Original file line number
+Diff line change
@@ -0,0 +1,6 @@
+    # Simulate a non-functional CC toolchain
+    common --per_file_copt=external/.*protobuf.*@--THIS_CC_TOOLCHAIN_IS_BROKEN
+    common --host_per_file_copt=external/.*protobuf.*@--THIS_CC_TOOLCHAIN_IS_BROKEN
+    # But, users should be able to use pre-built protoc toolchains instead.
+    common --incompatible_enable_proto_toolchain_resolution
+    common --@com_google_protobuf//bazel/toolchains:prefer_prebuilt_protoc

-Original file line number
+Diff line change
@@ -0,0 +1,6 @@
+    load("@com_google_protobuf//bazel:proto_library.bzl", "proto_library")
+    proto_library(
+        name = "empty_proto",
+        srcs = ["empty.proto"],
+    )

-Original file line number
+Diff line change
@@ -0,0 +1,13 @@
+    """Bazel module dependencies"""
+    module(
+        name = "com_google_protobuf-example-without-cc-toolchain",
+        version = "0.0.0",
+        compatibility_level = 1,
+    )
+    bazel_dep(name = "protobuf", repo_name = "com_google_protobuf")
+    local_path_override(
+        module_name = "protobuf",
+        path = "../..",
+    )

Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,2 @@
		This example demonstrates what happens when a Bazel user doesn't have a proper CC toolchain installed.
		This case commonly happens in projects with no C++ code, so they don't have a hermetic method of building C++ code.

-Original file line number
+Diff line change
@@ -0,0 +1,5 @@
+    edition = "2023";
+    package examples.without.cc.toolchain;
+    message EmptyMessage {}

feat(bazel): wire up prebuilt protoc toolchain #24115

feat(bazel): wire up prebuilt protoc toolchain #24115

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

zhangskz Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

alexeagle Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zhangskz Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

alexeagle Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!