From cce013b43e4e8d6b9d98bb6d611b68d9c2fe8d22 Mon Sep 17 00:00:00 2001 From: Chandler Carruth Date: Tue, 5 Sep 2023 13:06:41 -0700 Subject: [PATCH] Use high concurrency on CI build & test. (#3192) The expensive local actions will be separately gated to not overwhelm the machine, and currently highly asynchronous actions are a dominant part of our builds due to downloading cached artifacts. Without a high concurrency, these are downloaded roughly 2-at-a-time currently. I've verified that on a small machine without a good cache this doesn't seem to generate huge amounts of work and local build and test actions are successfully gated on the local flags. There is already a Bazel issue tracking this limitation: https://github.com/bazelbuild/bazel/issues/6394 --- .github/workflows/tests.yaml | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/.github/workflows/tests.yaml b/.github/workflows/tests.yaml index 71d433bfc5a3..d428e6dce0d7 100644 --- a/.github/workflows/tests.yaml +++ b/.github/workflows/tests.yaml @@ -163,6 +163,24 @@ jobs: build --remote_cache=https://storage.googleapis.com/carbon-builds-github-v${CACHE_VERSION}-${{ env.os_for_cache }} build --google_credentials=$HOME/gcp-builds-service-account.json + # Set an artificially high jobs count. This flag controls the number + # of concurrency Bazel itself uses, which is essential for actions + # that are internally blocked on for example downloading results form + # the cache above. Without setting this high, Bazel will pick a small + # number based on the available host CPUs and the reality will be a + # long chain of largely serialized download events with little or no + # usage of the host machine. Fortunately, local actions are + # *separately* gated on `--local_*_resources` that will avoid a large + # jobs value overwhelming the host. There is a bug to make downloads + # behave completely asynchronously and remove the need for this filed + # back in 2018 but work seemed to not finish: + # https://github.com/bazelbuild/bazel/issues/6394 + # + # There is a new effort (yay!) but until then it seems worth using the + # workaround of a high jobs value. The biggest downside (increased + # heap usage) seems like it isn't currently a big loss for our builds. + build --jobs=200 + # General build options. build --verbose_failures test --test_output=errors