There was a file that got into the testing filegroups but not the actual
built tarball. The result was that the installation failed to locate
itself correctly.
We also didn't have any testing to catch this. I'd like to add a bit
more end-to-end testing long-term, but for now just add a Python test
that validates the same set of files are in both.
Sadly, building and testing the compressed release is really slow and
not likely worthwhile outside of the actual release, but it would be
good to catch this stuff earlier. I tried switching from `bz2` to `gz`,
which made very little file size difference (so we should do it
anyways), and it is still too slow to do routinely. Instead, I've added
a Starlark macro that builds both the `pkg_tar` and the `py_test` to
validate it for both uncompressed `tar` and compressed `tar.gz`. This
lets us continuously test the `tar` version, and just test the `tar.gz`
in the nightly release run.
Updates the nightly release workflow to both test, run this specific
test, and use `gz`.
Last but not least, removes the `pkg_zip` as I don't think we have a use
case for this yet and some TODOs that should have been removed when the
version got added.
There is still a lot more we should try to do here, but this gets us
a decent start and should be enough for some of the developers working
on Carbon to use for simple cases, or tools like Compiler Explorer.
This first factors most of the complex setup for our CI workflow into
local actions that we can reuse in other workflows. It's actually
a fairly nice cleanup I think even without the specific use case of
a nightly release process. These use "composite" actions which is the
cleanest model for factoring out a subsequence of steps from a workflow.
I've tried to keep the inputs fairly minimal and easy to spot.
Next, it adds a cron-scheduled action that builds and creates a nightly
release. It also supports explicitly running the workflow, which can be
useful if the nightly release is broken in some way: a fix can be landed
and then a force run used.
Tested the `tests` workflow using our `action-test` branch:
https://github.com/carbon-language/carbon-lang/actions/runs/9545546278
Also tested the nightly release action by hacking in a `push` trigger
and using *draft* release. If you have admin rights, you can see an
example draft nightly build here:
https://github.com/carbon-language/carbon-lang/releases/tag/untagged-6a85cdce7b3a784a2e6b
Note that the GitHub releases are easily deleted, and even the tags
created in the repository can be cleaned up as needed so it should be
fairly harmless to fix this forward if needed.
With bazel 7.2.0, there are some dependency changes, resulting in:
```
WARNING: For repository 'bazel_skylib', the root module requires module version bazel_skylib@1.5.0, but got bazel_skylib@1.6.1 in the resolved dependency graph.
WARNING: For repository 'platforms', the root module requires module version platforms@0.0.8, but got platforms@0.0.9 in the resolved dependency graph.
```
This goes around doing some updates... protobuf's include structure has
changed a little too.
Not sure what changed (I think an upstream LLVM change, but maybe a
Linux distro change), but several folks have been running into problems
finding a standard C++ library when running the Carbon link step. It was
actually breaking our example build when LLD didn't find the right
libstdc++ install, but it finds one reliably on our build bots and
sometimes for some of the developers.
For now, just remove the C++ standard library from the link. We're not
doing that level of interop, and the plan is really to do that not with
the system standard C++ library but by building and bundling libc++ with
the installed toolchain.
Left a comment explaining what's going on here.
It turns out we can make these work with very minimal complexity because
the LF is still in the right place either way. This also lets us easily
support mixtures of LF and CR+LF line endings gracefully. We create the
line structures around the LF bytes and then have the byte-dispatch loop
notice a CR followed by an LF and skip to the LF behavior.
Rather than add the remaining complexity around supporting bare CR and
LF+CR sequences (both of which are quite rare now), this just adds
diagnostics when we encounter a CR byte that won't fall out of our CR+LF
handling. This is a better experience for users than the alternative. We
still have a TODO to handle the full complexity of vertical whitespace,
but I've updated it to reflect that the common case should be handled
already.
This isn't complete though: we need to add support in string literal
lexing, and we need to teach the diagnostic rendering to handle the
error messages above better. But those will be future PRs, this is
enough to unblock folks who happen to edit a Carbon source file with
notepad on Windows which seems important.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
I inadvertently didn't commit and push the actual fixes suggested and
made to that PR, and didn't notice this before putting it into the merge
queue. =[ Very sorry, these were supposed to be in that PR.
This adds a defined Carbon version to the Bazel build and codebase that
can be used both to implement features like version checks and to report
a meaningful version on the command line. This replaces a hard-coded
string and a TODO in the driver.
As part of this, it adds support for defining the version in Bazel, and
special build flags for overriding relevant parts such as the
pre-release marker used. The exact structure and meaning of our version
string, including the pre-release parts, is implemented here in line
with the draft proposal:
https://docs.google.com/document/d/11S5VAPe5Pm_BZPlajWrqDDVr9qc7-7tS2VshqO0wWkk/edit?resourcekey=0-2YFC9Uvl4puuDnWlr2MmYw
This also introduces a workspace status command to the repository to
extract the git commit SHA and other information when building, and the
logic to stamp that into binaries as part of the version string when
useful. The technique used leverages weak symbols with whole archive
linking to allow a link-time override of unstamped data with stamped
data in the leaf executable. This makes building with `--stamp` a
reasonable default, especially for development builds. The CI system is
explicitly opted out of this as there it has no benefit.
Last but not least, all of these are wired into the install rules so
that we build installable packages with the version number in a
conventional place in the directory and filename.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Require mapping from a `ConstantId` to an `InstId` to go through the
`ConstantValueStore`.
This is a preparatory step for an upcoming generics change where
symbolic `ConstantId`s are no longer just a thin wrapper around an
`InstId` but instead are indexes into a table with additional
information about the symbolic constant beyond its `InstId`.
Instead of redundantly storing both the `return_type_id` and
`return_storage_id`, where the declared return type is just the type of
the return storage, store only the `return_storage_id`.
Add a convenience property to get the declared return type of the
function.
In addition to avoiding storing redundant information, this is a
preparatory step for an upcoming change for generics support that will
make it more expensive and awkward to store `TypeId`s in places other
than the type of an instruction.
Model such calls as call operations, and compute the resulting type in
constant evaluation rather than in type-checking. This means we no
longer form non-constant `ClassType` or `InterfaceType` values, and that
we produce an `<error>` type in the case of bad argument lists rather
than a broken / meaningless `ClassType` / `InterfaceType` that doesn't
actually represent a type. This in turn suppresses some follow-on
diagnostics.
This adds two different growth APIs. This is instead of the more
conventional STL `reserve` method. One allows users that aren't trying
to grow in anticipation of an *exact* count of insertions, but generally
trying to size the table to the correct ballpark with a power-of-two
estimate.
The other API allows pre-growing to allow a specific number of
insertions to be performed without further growth. This API takes the
maximum load factor and other implementation details into account.
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
While it was convenient once to have an immediate check while inserting,
it is indeed far too quadratic. The test was taking 30-60 seconds for
me. =[ So most of the fix here is just to stop doing the check on every
insertion for all previous elements.
There were a few other somewhat slow steps, and I tried to pull those
back as well. I don't think we lose any utility here. Now everything
runs nice and quickly. =]
This uses a header-only extraction of the Boost unordered hashtable
project to allow a trivial Bazel build and for us to benchmark against
it effectively.
Previously we just looked at the raw count of probed keys. Now, we
compute the average and max of both the probe _distance_ measured in the
number of _groups_ probed, and the number of probe _compares_ measured
in the compares required _before_ finding the matching entry.
This lets us understand the relative impact of probe-distance vs. tag
collisions on a given set of benchmark keys. Some of this is motivated
by considering additional optimization techniques similar to those used
in Boost's table and the F14 table from Facebook/Meta.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
The hash table design is heavily based on Abseil's ["Swiss
Tables"][swiss-tables] design. It uses an array of bytes storing
metadata about each entry and an array of entries where each is a pair
of key and value. The metadata byte consists of 7-bits of hash of the
key (distinct from the bits used to index the table), and one bit
indicating the presence of a special entry -- either empty or deleted.
[swiss-tables]: https://abseil.io/about/design/swisstables
There are a large range of optimizations and other nuanced aspects of
this hash table design and implementation, a good point to understand
that context is `raw_hashtable.h` which has an overview of the design
and references to various other files for relevant details.
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
In some cases this is just a refactoring; in others, we no longer import
both an instruction in no block plus a constant instruction, resulting
in creating fewer instructions. The non-constant version of the
instruction always ended up unreferenced in the affected cases.
Following up on a comment in #4042.
Merges handle_let.cpp and handle_variable.cpp into a single file in
order to take better advantage of commonalities.
Both still have a bunch of kludges, and there's still divergent handling
in handle_binding_pattern that will yield different results. However,
this fixes some things about `let` like starting to make use of
global_init (imperfectly, due to how VarStorage works, I grant), and in
particular adding `let` names to a name scope, not just lexical lookup.
This is mostly mechanically duplicating work done for generic classes to
also support generic interfaces.
Also fix both generic interfaces and generic classes to support
importing class and interface types with arguments from another file.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com>
This is a modifier if it appears before an introducer and a control flow
keyword if it appears before `=>`. Use introducer highlighting if it's
neither.
VSCode's JSON parser no longer accepts comments in at least textmate
grammars, so remove the comments. It's unclear whether VSCode changed
here or whether this hasn't worked since the comments were added.
Clean up the textmate grammar: refactor, add some missing operators and
keywords, classify keywords as introducer / modifier as appropriate, fix
comment grammar to require space after `//`, and generalize recognition
of iN, fN, uN to recognize all such type literals.
Be more cautious in language server -- I was seeing frequent crashes for
`IdentifierName` parse nodes whose token was not actually an identifier,
presumably due to error recovery.
Don't use the pretty-printed type name, because that's intended for
diagnostics, not for a theoretically machine-readable format like SemIR.
Types are always constants, so omit the leading `constant.` on the type
instruction name.
Don't format the *type of* the function `F` as `"F"`. Instead use
`"<type of F>"`.
Also improve instruction naming for function and generic class name
values: name the value after the function or generic class, and add a
`.type` suffix to the type.
Adds access to the name lookup table in name scopes. This is so that we
can quickly check access during name lookup without resolving the entity
itself. Does this for names in general, but does not implement handling
for entity-scoped names, only namespace-scoped names (where they're
essentially just not exported).
Excludes `private` names from exports. Although names should be
accessible to `impl` files, that's not implemented here because we'll
probably want to do it by directly copying name lookup tables.
Playing with the macro suggestion on #4028, replace DeclKind with some
templating that asserts the in-use token is an introducer token. This
allows type-safe usage of Lex::TokenKind, reducing the benefit of a
separate enum while improving stdout (since now this will ostream as the
keyword name).
Check the parameters specified in a name qualifier against the
parameters of the entity that the qualifier refers to.
For interfaces, this required adding minimal support for parameterized
interface names.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This also allows us to default construct installs in an error state,
which is useful for cases like fuzzers where we want to default
construct a global, but then re-initilaize it That said, happy to
consider alternative designs here.
With private modifiers, we'll want to start checking modifiers later,
e.g. after a name conflict is detected (and potentially merged). I think
we've agreed to be more explicit about whether the modifier functions
are manipulating state, versus trying to keep the state on the stack a
little longer (moving Pop to the end of these functions).
Removing FileScope from the stack and renaming it to DeclIntroducerStack
to better reflect the usage and behavior. The FileScope mostly reflects
an approach that wasn't ultimately adopted.
Most of these are places where we failed to include a header file and
simply never got an error about this. The fix is to include the header
file.
Most other cases are functions that should have been marked `static` but
were not. Finding all of these was a main motivation for me enabling the
warning despite how much work it is.
One complicating factor was that we weren't including the `handle.h` for
all the state-based handler functions. While this isn't a tiny amount of
code, it is just declarations and doesn't add any extra dependencies. It
also lets us have the checking for which functions need to be `static`
and which don't. For the `parse` library I had to add the `handle.h`
header as well, I tried to match the design of it in `check`.
I have also had to work around a bug in the warning, but given the value
it seems to be providing, that seems reasonable. I've filed the bug
upstream: https://github.com/llvm/llvm-project/issues/94138
I also had to use some hacks to work around limitations of Bazel rules
that wrap `cc_library` rules and don't expose `copts`. I filed a bug for
`cc_proto_library` specifically:
~https://github.com/bazelbuild/bazel/issues/22610~https://github.com/bazelbuild/bazel/issues/4446
Following up on discussion from #3948, doing a general rename of
"enclosing scope" to "parent scope" (and "enclosing scopes" to "ancestor
scopes"). The intent is to improve understandability and collide less
with C++ terminology for "enclosing scope". Note this changes most uses
of "enclosing", but leaves behind a few like "enclosing function" and
"enclosing block".
Note this does create some "parent class" mentions for "adapt" and "var"
(the class they're within), which is maybe unfortunate, but we'd
probably say "base class" if we meant inheritance so perhaps that's
okay. Along the same lines, these are the only `parent_class` uses I see
now, and we do have a few `base_class`.
This has is a nice-to-have for me. Frequently I want to run a specific
test, and end up digging through output to be able to copy-paste the run
line. This uses TIP lines to inject the command into the file when using
AUTOUPDATE.
Note, one of the reasons I want this is because "bazel test
//toolchain/testing:file_test --test_output=all" has been regularly
exceeding bazel's output limit for me (workaround is either opening the
output file or specifying an obscure output limit flag), making it a
little harder for me to get the commands. However, frequently I'm adding
a file and want to iterate on it, so that's really the use case I have
in mind here.
Despite this name not being in the runfiles manifest, this works in
Bazel because of how the module and workspace are configured. I should
have realized that as it was used elsewhere as well. This should
consolidate all of our runfiles paths to use the exact same patterns
now. These were the only remaining divergences.
Just replacing AUTOUPDATE with NOAUTOUPDATE, and removing the autoupdate
script. Tests will still run, but autoupdate may need to be fixed if
significant changes are made.
I noticed this while trying to autoupdate for #4007 (because we verify
that tests have been autoupdated). Autoupdate was likely broken by
#3449.
Trying to run with no changes gives:
```
CHECK failure at testing/file_test/file_test_base.cpp:801: !absl::GetFlag(FLAGS_test_targets_file).empty(): Missing --test_targets_file.
```
This is because the `file_test` rule creates a file with test inputs
that it runs with, which the prebuilt binary doesn't provide.
A local kludge to create a `file_test` target not using prebuilt_binary
showed another error:
```
: CommandLine Error: Option ': CommandLine Error: Option 'parser_debug' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
trace_phase' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
```
I'm thinking here that the explorer code hasn't been in enough use, and
we're seeing some rot as a consequence. Rather than trying to maintain
it, I'm suggesting to go with NOAUTOUPDATE.
Instead of adding a `Token<Kind>` field to typed parse nodes that
contains a `TokenIndex`, make the parse node directly hold a
`TokenIndex` of a more specific kind.
Jon correctly pointed out that this wasn't being used any more.
Originally, the code needed a different tree for the install prefix in
order to test different code paths of detection, but since those are now
explicit in the code it is already using a single tree. But I never
updated the actual BUILD file to stop building and depending on the
extra prefix.
Just spotted these while looking at warnings that seem to fire on our
code are probably are things we'd fix if we saw them. None of these seem
important FWIW.
Also removes a redundant flag that is part of `-Wall`.
I have a follow-up for the high-value warning I spotted that motivated
me to look at all of this. But it's noisy so kept it as a separate PR.
I misunderstood how this flag worked, which explains why it didn't work.
Oops. Sorry about that. So far we've been lucky to not bump into this,
but we've had to work around it with updating libraries, etc.
Moreover, there are quite a few warnings that seem useful for us to
enable, but doing so would cause lots of friction without this working.
The added warnings will be separate PRs along with their cleanups.
This removes the `data_dir` from the driver favoring the installation
abstraction for the both locating the prelude and linking utilities.
With this, an installed toolchain should also be able to compile and
link Carbon successfully, and the build of the examples should exercise
this path almost exactly. (The only difference is using the driver
`cc_binary` directly rather than relying on the symlink from inside the
install tree.)
This takes the installation layout and replicates it using `rules_pkg`
to build either a tarball or a zip file of the toolchain. Correctly
manages file permissions and symlinks, etc.
There are some big remaining things here:
- Figure out how we want to test this. We can add shell tests maybe?
A bit awkward. Nicer would be to make the `//examples` tree build
using this rather than the more native-bazel install data, however
building these is quite slow and it seems bad to pay that cost
constantly so dedicated testing is probably better. For now, I've
tested these manually.
- Need to add versions to the toolchain and then thread them through
here so they install properly as a versioned release.
But my primary goal for now is just to be able to validate that the
install tree is working outside of Bazel and this does enough for that.
The above will be longer-term things.
The install directory contains the BUILD logic for creating an
installable tree of data files and executables for the toolchain, and
a library to facilitate toolchain code accessing the paths to their data
within this installation.
Then adds an installation of LLD in a synthetic LLVM installation, and
teaches the Clang runner to configure this and use it for linking
instead of the system linker.
Currently, the install paths only really manage access to the LLVM
binaries installed and used by the Clang runner for linking, but
eventually other data files like the prelude and runtime libraries will
be fleshed out as well. There are TODOs for moving more things over here
such as the prelude.
One interesting aspect of this is where to put helpers like parts of
LLVM in our install. This PR suggests nesting those files under
`lib/carbon`. While using a `lib` subdirectory isn't a perfect fit for
the FHS (Filesystem Hierarchy Standard), having a single location where
private data is collected is significantly superior to spreading them
across the system. This also matches similar patterns used by Clang
itself and several other language toolchains and standard libraries.
The install directory also provides a natural place for us to build out
packaging rules to create installable packages in various formats, but
that remains future work.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Chandler Carruth
Jon Ross-Perkins
Richard Smith
dependabot[bot]
