carbon-language/carbon-lang
1
0
Fork 0
mirror of https://github.com/carbon-language/carbon-lang.git synced 2026-09-24 21:10:12 +01:00
Code Issues Packages Projects Releases 10 Wiki Activity

Remove LLVM from the repo, and clean up history. (#1344)

Browse Source 
This proposal establishes a plan for moving away from the embedded copy
of LLVM and instead downloading it with Bazel.

The goal is that after this lands, we will do a history-rewrite to
cleanup the repository. There are instructions on how folks can move any
in-flight work over to the newly tidied repo.

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
This commit is contained in:
Chandler Carruth
2022-06-28 13:49:38 -07:00
committed by GitHub
co-authored by Richard Smith Jon Ross-Perkins josh11b
parent b626cb9c64
commit df345b5ec7
5 changed files with 416 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.pre-commit-config.yaml
+1
View File
@@ -168,6 +168,7 @@ repos:
.bazelversion|
compile_flags.txt|
third_party/.*|
bazel/llvm-patches/.*\.patch|
.*\.def|
.*\.svg|
.*/fuzzer_corpus/.*|
WORKSPACE
+10 -2
View File
@@ -84,10 +84,18 @@ http_archive(
# LLVM libraries
###############################################################################
new_local_repository(
# We pin to specific upstream commits and try to track top-of-tree reasonably
# closely rather than pinning to a specific release.
llvm_version = "e2f627e5e3855309f3a7421f6786b401efb6b7c7"
http_archive(
name = "llvm-raw",
build_file_content = "# empty",
path = "third_party/llvm-project",
patch_args = ["-p1"],
patches = ["@//:bazel/llvm-patches/0001-Patch-for-mallinfo2-when-using-Bazel-build-system.patch"],
sha256 = "228c37eecf8a8027ab32ac466b988712136191a0076d80750c646a3a9b1dc5d2",
strip_prefix = "llvm-project-%s" % llvm_version,
urls = ["https://github.com/llvm/llvm-project/archive/%s.tar.gz" % llvm_version],
)
load("@llvm-raw//utils/bazel:configure.bzl", "llvm_configure")
bazel/llvm-patches/0001-Patch-for-mallinfo2-when-using-Bazel-build-system.patch
+32
View File
@@ -0,0 +1,32 @@
From 80f5475adfe179739a45d42850f8c06a630bc3a0 Mon Sep 17 00:00:00 2001
From: Chandler Carruth <chandlerc@gmail.com>
Date: Fri, 17 Jun 2022 09:10:41 +0000
Subject: [PATCH] Patch for mallinfo2 when using Bazel build system.
This detects and defines the `HAVE_MALLINFO2` macro based on the glibc
version to allow easy use of the Bazel build on systems with modern
glibc installs.
---
llvm/lib/Support/Unix/Process.inc | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/llvm/lib/Support/Unix/Process.inc b/llvm/lib/Support/Unix/Process.inc
index d3d9fb7d7187..da3e721146f6 100644
--- a/llvm/lib/Support/Unix/Process.inc
+++ b/llvm/lib/Support/Unix/Process.inc
@@ -31,6 +31,13 @@
#if HAVE_SIGNAL_H
#include <signal.h>
#endif
+// When glibc is in use, detect mallinfo2 to address mallinfo deprecation
+// warnings.
+#if !defined(HAVE_MALLINFO2) && defined(__GLIBC_PREREQ)
+#if __GLIBC_PREREQ(2, 33)
+#define HAVE_MALLINFO2
+#endif // __GLIBC_PREREQ(2, 33)
+#endif // !defined(HAVE_MALLINFO2) && defined(__GLIBC_PREREQ)
#if defined(HAVE_MALLINFO) || defined(HAVE_MALLINFO2)
#include <malloc.h>
#endif
--
2.36.1
docs/project/contribution_tools.md
+3 -4
View File
@@ -127,10 +127,9 @@ brew install bazelisk
### Clang and LLVM
[Clang](https://clang.llvm.org/) and [LLVM](https://llvm.org/) are used to
compile and link Carbon as part of its build. Their source code are also
provided in a [third_party subtree](/third_party/llvm-project) for incorporation
into Carbon or Carbon tools as libraries. While the subtree tracks upstream
LLVM, the project expects the LLVM 12 release (or newer) to be installed with
compile and link Carbon as part of its build. Bazel will also download and build
against a specific upstream LLVM commit. While the Bazel uses upstream LLVM
sources, the project expects the LLVM 12 release (or newer) to be installed with
Clang and other tools in your `PATH` for use in building Carbon itself.
Our recommended way of installing is:
proposals/p1344.md
+370
View File
@@ -0,0 +1,370 @@
# Remove LLVM from the repository, and clean up history.
<!--
Part of the Carbon Language project, under the Apache License v2.0 with LLVM
Exceptions. See /LICENSE for license information.
SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-->
[Pull request](https://github.com/carbon-language/carbon-lang/pull/1344)
<!-- toc -->
## Table of contents
- [Problem](#problem)
- [Proposal](#proposal)
- [Details](#details)
- [Implementation](#implementation)
- [Patching LLVM](#patching-llvm)
- [Updating forks and clones](#updating-forks-and-clones)
- [Archive your existing fork and clones](#archive-your-existing-fork-and-clones)
- [Create a fresh fork and clone](#create-a-fresh-fork-and-clone)
- [Porting in-progress branches from your archived clone](#porting-in-progress-branches-from-your-archived-clone)
- [Updating pending PRs](#updating-pending-prs)
- [Review comments may be disrupted](#review-comments-may-be-disrupted)
- [Rationale](#rationale)
- [Alternatives considered](#alternatives-considered)
- [Do nothing](#do-nothing)
- [Don't rewrite the repository history.](#dont-rewrite-the-repository-history)
- [Go back to submodules](#go-back-to-submodules)
- [Rename the repository, and create a new one](#rename-the-repository-and-create-a-new-one)
- [Manually extract and archive some review comments](#manually-extract-and-archive-some-review-comments)
<!-- tocstop -->
## Problem
We have tried both Git submodules and subtrees to manage accessing the LLVM
source code as part of Carbon. We have had significant issues with both of these
over time.
Git submodules are well supported in general, but make the repository
significantly less user-friendly. There are a number of common operations, not
least the initial clone of the repository, that are made frustratingly more
complex in the presence of a submodule. In some cases (switching branches), it
can even cause hard to diagnose errors for users. It also doesn't directly help
with carrying local patches. Because of all of these, we switched to subtrees.
Git subtrees in some ways work much better once set up as they are simply a
normal directory for most users. They also make it especially easy to make and
carry local patches. However, setting up the subtree and updating the subtree
are bug prone and interact very poorly with GitHub's pull request workflow. The
result is that updating LLVM is currently both very difficult and error prone.
Another fundamental problem subtrees expose is that the LLVM repository is
_huge_. It includes large and complex projects that Carbon is unlikely to ever
use, and it doesn't make sense for us to start our repository off with all of
that history and space if we can avoid it. Including LLVM causes things like
`git status` or cloning the repository to be dramatically more expensive without
significant benefit.
## Proposal
We should switch to using Bazel to download a snapshot of LLVM during the build,
and do a destructive history re-write to the Carbon repository to completely
remove LLVM from it. All of the building against LLVM will continue to work
seamlessly. However, the repository will become tiny and fast to work with, and
the snapshot download is significantly cheaper than cloning or working with the
full-history of LLVM.
The only way to capture the benefit here is to do a destructive update to the
repository's history. This is unfortunate, but essential to do as soon as
possible and before we shift to be public. Despite the cost of making this
change, the cost of _not_ making it will grow without bound for the life of the
project.
**Important:** This _will_ require a manual update of some kind for every fork
and clone. The steps to update them are provided below.
## Details
### Implementation
<!-- google-doc-style-ignore -->
This will be accomplished using the
[`git-filter-repo` tool](https://github.com/newren/git-filter-repo). This makes
it very easy to prune any trees from the history, replaying the history left and
resulting in a clean and clear result.
<!-- google-doc-style-resume -->
We will remove three other trees while here that are no longer used:
- `src/jekyll/`
- `src/firebase/`
- `website/`
Once we're taking a history break, we should capture the value we can and have a
minimal history of the repository we are actually using. The largest of these is
`src/jekyll` (988kb even packed), but it seems cleanest to remove the three
together.
### Patching LLVM
We still need to support patching LLVM. Initially, we will start with a simple
approach of applying a collection of patch files from the Carbon repository to
the LLVM snapshot. This is especially convenient while Carbon's repositories are
not public. Eventually, we can switch to having a fork of LLVM that we snapshot
and developing any needed patches there. Having a more robust long-term solution
is expected to be important if larger scale development is needed on Clang when
building interoperability support.
<!-- google-doc-style-ignore -->
Developing patches is also very easy when using Bazel. You can pass
`--override_repository=llvm-project=<directory>`
([docs](https://bazel.build/reference/command-line-reference#flag--override_repository))
to have Bazel use a local checkout of LLVM (with any patches you are testing)
rather than downloading it. If doing significant development, this can be added
to your `user.bazelrc` file to consistently use your development area.
<!-- google-doc-style-resume -->
### Updating forks and clones
If you have a fork and any clones of the repository with work that you want to
save, we suggest the following process:
#### Archive your existing fork and clones
1. Go to the archived repository:
https://github.com/carbon-language/archived-carbon-lang
2. Fork this archived repository just like you normally would (into your
_personal_ space). It is important to fork the `carbon-language` hosted
archived repository so you pick up the ACLs. You should **not** create your
own personal repository.
3. Update any clones to use these archived remotes to avoid accidentally trying
to interact with the new repository. Example commands assuming the remote
`origin` points to your fork and and `upstream` points to the main
repository. You may also need to adjust from the SSH-style URLs to HTTPS
ones to match:
```
git remote set-url upstream git@github.com:carbon-language/archived-carbon-lang.git git@github.com:carbon-language/carbon-lang.git
git remote set-url origin git@github.com:$USER/archived-carbon-lang.git git@github.com:$USER/carbon-lang.git
```
4. For each clone, mirror everything into the new `origin` (your fork):
```
git push --mirror origin
```
If you have multiple clones with different but overlapping branches, this
may require some extra steps to avoid collisions when doing the mirror push.
At this point, your clones should all point to an archival fork and should be
able to function "normally" cleanly. The goal here is just archival and making
sure you don't lose anything.
#### Create a fresh fork and clone
Delete your existing fork of `carbon-lang` (_not_ `archived-carbon-lang`):
1. Go to your fork's settings page:
`https://github.com/$USER/carbon-lang/settings`. Be certain this is your
_fork_ and not the `carbon-language` organization.
2. Scroll to the bottom, and click `Delete this repository`.
Once deleted, create a fresh fork from the now-filtered main repository:
https://github.com/carbon-language/carbon-lang/fork
Clone this fork and you're ready to go with the new repository structure.
#### Porting in-progress branches from your archived clone
For any branch in your archive clones that you want to import, there are two
approaches.
The simplest way is to extract the branch as a patch file and apply it in the
new repository:
```
# If your old clone is in the `archived-carbon-lang` directory and your new
# clone is in `carbon-lang`, both of them on the `trunk` branch:
cd archived-carbon-lang
git switch $BRANCH
git diff upstream/trunk... > ../$BRANCH.patch
cd ../carbon-lang
git switch -c $BRANCH
patch -p1 < ../$BRANCH.patch
git commit
```
This will just flatten the branch into a single commit in the new clone.
If you want to preserve the commit history, you can do that using
`git format-patch`. Some points to remember here:
- Make sure you've rebased the branch into a clean series of commits on top of
the latest trunk in the archived repository.
- Use `git format-patch` to get a series of patches in a directory. See its
help for detailed usage.
- Use `git am` to apply the series of patches in your new clone under some
branch. Again, see its help for detailed usage.
#### Updating pending PRs
The pending PRs should end up closed automatically when you delete your fork
above. If not, the simplest approach is to close them and just create a new PR.
In-flight discussion comment threads will be awkward, but there isn't much else
to do. Rename the proposal document if needed.
We will explore whether we can re-open PRs with a fresh forced push from the
fresh fork, but aren't confident in this working. It also isn't necessary, and
we only have 12 PRs open at the moment.
### Review comments may be disrupted
When we edit the repository, some PR comments (especially pending PRs that are
updated to follow a freshly created fork) may lose their association with the
line of code they were made against, and it is possible we may be unable to find
some comments. This will at most apply to inline comments within the code of
PRs, but that is the common case for PRs. We expect most of these to still be
somewhat visible in the conversation view of the pull request, but they may be
hard to find due to no longer being attached to a line.
## Rationale
- [Community and culture](/docs/project/goals.md#community-and-culture)
- This will make it even easier and smoother for folks to clone, build,
and even contribute.
- It will make clones significantly cheaper, especially those not building
code but just working on documentation.
- [Language tools and ecosystem](/docs/project/goals.md#language-tools-and-ecosystem)
- Preserving the ability to build with LLVM and develop patches will be
important as we expand the set of language tools and ecosystem being
developed.
## Alternatives considered
### Do nothing
We could simply continue using the moderately broken Git subtree approach.
Advantages:
- No need to change our approach.
- No destructive operation on the repository requiring everyone to update.
Disadvantages:
- Every update of LLVM remains difficult, manual, and error prone.
- We're currently carrying an LLVM patch, and it's easy to lose (and has
been lost) in updates.
- Will continually hit bugs in Git subtree as it seems both brittle and not a
priority. We will have to struggle to understand and fix or work around
them.
- The repository remains massive, slow, and contains significant LLVM code
that we will never use.
- If we ever have users that wish to import the Carbon repository into some
other environment, they will have to pay the cost of LLVM or remove it
somehow. It may even cause them to have two copies of LLVM.
We think this problem is worth solving.
### Don't rewrite the repository history.
We could fix this without rewriting history. If we choose not to rewrite history
now, it should be noted that the cost of rewriting history only grows and so we
should expect to _never_ rewrite history.
Advantages:
- No need for manual steps to update forks, clones, in-flight patches, etc.
- Commit hashes remain stable.
- All code review comment information associated with commit hashes will be
retained.
Disadvantages:
- We pay the cost of having imported LLVM forever. Even if this cost is
incrementally small (some amount of repository space when cloning with
history), it is a cost we will never stop paying.
While the immediate costs are high, the unbounded time frame for which we will
pay for leaving LLVM in the history means that will eventually be the dominant
cost and we should just rewrite history, and the sooner the better.
### Go back to submodules
Rather than switching to use Bazel downloaded snapshots, we could go back to
using Git submodules. The original motivation to move away from submodules was
needing to carry a local patch. Submodules and the proposed direction have the
same options there, and our approach is discussed above.
Advantages:
- Somewhat more of a Git-native approach.
- Would avoid needing to invent another approach if we add a non-Bazel build
system.
Disadvantages:
- Much of the cloning cost and other Git command costs we see with subtrees
would still be present.
- It makes working with the Git repository even more tricky to get right.
- While less esoteric than subtrees, it still exposes a less polished surface
of Git that we will have to cope with.
- Most notably, this will re-introduce the stumbling block of users first
encountering Carbon not having a seamless experience.
We think we should try something simpler here, even if a bit less of a
native-Git solution. It is also especially important for us to optimize the
initial new contributor flow, and the Bazel approach is expected to be more
seamless.
### Rename the repository, and create a new one
Rather than editing the repository in place, this would move it aside and create
a new one with the edited history.
Advantages:
- Some disruptive aspects of the in-place history edit would be avoided,
specifically parts of PRs that are associated with commit hashes would
likely continue working due to the commit hashes being preserved.
Disadvantages:
- For most cases, this would be equally disruptive -- forks and clones would
have the same update needed as they reference the repository by name.
- We would either need to build a system to carefully re-create the exact
issue numbering and PR numbering, which may not even be possible, or we
would need to allow all issue numbers and PR numbers to churn.
- The PR numbers churning is especially disruptive as the commit log
references them to connect commits to the code review that led to the
commit. They are also the basis of the proposal numbers.
- There isn't likely a way to move the PRs back to the main repository, so
browsing the historical code reviews would become problematic.
This seems to largely trade off preservation of some of the comment history
within PRs for preserving the location, numbers, and links of all PRs and
issues. That doesn't seem like the right tradeoff.
### Manually extract and archive some review comments
We could attempt to extract and archive review comments in case they are lost or
made hard to find by the change.
Advantages:
- Defense in depth against any information loss here.
Disadvantages:
- Would be a decent amount of work to get this right.
- May not lose much information here. We think the comments will still be in
the conversation view, but maybe hard to find.
- Unclear whether any of these will actually have value, especially extracted
and out of context from the review where they were made.
- Serious concerns about arbitrarily scraping and moving comments other folks
authored outside of the system (GitHub) that they authored them within.
Overall, while there is some risk here, we don't think it is too high and the
cost of trying to mitigate this seems unreasonable given the relatively small
total amount of data at issue here.