For now, a builtin function is defined by specifying a string literal
initializer in a function declaration:
```carbon
fn MyBuiltin(a: i32) -> i32 = "builtin.name";
```
End-to-end support is included for a sample `"int.add"` builtin
performing integer addition, covering constant evaluation and code
generation.
The implementation here needs substantial refactoring before we'll be
ready to start adding more builtins. That refactoring work will be
coming next. This change is aiming to checkpoint some incremental
progress.
I'd kept these in to separate the bazel module update from the BUILD
file changes, then forgot about it. I think all of these can be cleanly
removed now. I think it's something we should clean up for consistency
with the bazel central repository names; I think it's best to reduce
that divergence.
llvm_zlib and llvm_zstd remain because of how llvm depends on the
particular names.
Similar to #3705, we actually have a mix of `Make` and `Create` in
factory functions too, so this PR is normalizing on `Make`. It's
intended to be consistent with the naming choice for Carbon factory
functions.
Note, MakeSyntheticBlock is the only one I feel a little weird about
because llvm's own APIs use Create, and this is essentially wrapping
LLVM calls. But the flipside is it also feels like a vague line to draw,
when we also differ from LLVM coding style in other ways.
The `command_library` provides functionality for end users to query
documentation and help information about commands through the command
line interface itself. At present, this is implemented as:
* A flag, `--help`, on every command
* A meta subcommand, `help`, on every command that has other subcommands
To match with common functionality in other command line interfaces,
this PR amends the `help` meta subcommand to accept an optional
positional string argument. This argument specifies which subcommand to
print help information for. When not specified, the present behavior is
maintained, printing the help information for its parent command.
A potentially unexpected consequence of this implementation is that you
can query for help on meta subcommands as well: `help help` is a valid
input which prints the help information about the `help` meta subcommand
(e.g., same for `version`). I don't see any harm in this behavior so I
chose not to explicitly prevent it, but I'm open to preventing it if
deemed unwanted. (I'm also happy to workshop the strings in this PR.)
Closes#3694.
#### Before
```
$ carbon help compile
```
```
ERROR: Found unexpected positional argument or subcommand: 'compile'
```
#### After
```
$ carbon help compile
```
```
Compile Carbon source code.
This subcommand runs the Carbon compiler over input source code, checking it for errors and producing the requested output.
Error messages are written to the standard error stream.
Different phases of the compiler can be selected to run, and intermediate state can be written to standard output as these phases progress.
Subcommand 'compile' usage:
carbon [-v] compile [OPTIONS] <FILE>...
...
```
This removes the use of `enable_if` and tries to adopt concepts instead
of type traits when available.
The `ostream.h` change is a bit subtle as it adds a restriction not
previously in place -- that the stream is *contvertible* to
`std::ostream` as well as having it as a base class. This seems to match
the intent of the code.
The `hashing.h` code adds an implementation detail concept, and so I've
also clarified that the dispatch namespace is an internal one that isn't
part of the public API.
This required adding a few headers that were found transitively before,
but not too many. This is sadly a fairly manual process of opening every
file in my IDE, but I think I got everything in `//common` and
`//toolchain`.
There are a few cases where technically we don't need `foo.h` to be
included into `foo.cpp`, but I've forced those to stay with a pragma.
I've tried to catch the places where we can cut deps in Bazel as well,
but not sure I got all of those.
I had been noticing these in other PRs and it seemed better to isolate
the change.
Testing this hash function with representative hash table
implementations showed significant differences in quality between
different multiplicative hashing constants. The constants used and
documented were OK but had clear limitations when merely using a single
64-bit multiplication. However, a search uncovered (partly by luck)
a constant that has empirically been shown to be both significantly
better than other constants and generally not have problematic
weaknesses. There are still plenty of collisions for string keys and
heavily loaded hash tables of course, but no examples of severe outlier
collision rates as observed with all other constants we have tried.
There is a slightly longer comment explaining some of this context and
the other constants we have tried in the PR as well.
To this day, we still don't fully understand why the constant used here
behaves so much better than other constants we have tried, including all
of those we've found in other hashing algorithms.
This patches bazel_clang_tidy handling of headers. I found an equivalent
change at https://github.com/erenon/bazel_clang_tidy/pull/13, but that
was [already
rejected](https://github.com/erenon/bazel_clang_tidy/pull/13#issuecomment-1047007424).
Per the criticism, this will result in redundant processing of headers.
The project instead uses `HeaderFilterRegex: ".*"`, but that results in
two problems:
1. When running with `-k`, errors are repeated when a header is included
more than once, which is common.
2. clang-tidy including errors from headers that are included from other
modules (e.g., abseil-cpp); filtering correctly is difficult.
Given the trade-offs and options (including forking), I thought patching
was preferable so long as it remains narrow.
Add a mechanism to define instruction categories, to support inspecting
the common representation of similar kinds of instruction. Use that
mechanism to make formatting of branch instructions slightly more
type-safe.
The idea here is to use the existing `Inst` mechanism for converting to
and from structs, extended to operate on a struct representing multiple
different kinds of instruction. In this case, the concrete kind of
instruction is stored in the struct in a `kind` field, rather than being
implied by the type.
Factored out of #3555 where this mechanism is used to provide a common
interface for runtime and symbolic name bindings.
This sets things up to use `bazel` to run `clang-tidy` using
https://github.com/erenon/bazel_clang_tidy.
I'm fixing issues outside of explorer, and disabling clang-tidy for
targets in explorer that have legacy issues. I was going to disable
clang-tidy for targets in explorer such as interpreter anyways, because
they're slow to parse, and just extended that to the currently failing
targets.
These are intended to allow the structure of a parse tree node to be
described more precisely in code, to support these use cases:
- Automated checking that the parse tree conforms to the expected
structure. (Added to `Tree::Verify`.)
- Easier reading and understanding of the structure of the parse tree by
toolchain developers. (See `parse/typed_nodes.h`.)
- Easier navigation of the parse tree, for example for tooling uses and
for use when forming diagnostics.
On this last point, an object representing the file may be inspecting
using `Tree::ExtractFile`, as in:
```
auto file = tree->ExtractFile();
for (AnyDeclId decl_id : file.decls) {
// `decl_id` is convertible to a `NodeId`.
if (std::optional<FunctionDecl> fn_decl =
tree->ExtractAs<FunctionDecl>(decl_id)) {
// fn_decl->params is a `TuplePatternId` (which extends `NodeId`)
// that is guaranteed to reference a `TuplePattern`.
std::optional<TuplePattern> params = tree->Extract(fn_decl->params);
// `params` has a value unless there was an error in that node.
} else if (auto class_def = tree->ExtractAs<ClassDefinition>(decl_id)) {
// ...
}
}
```
The `Extract...` functions collect the child nodes into the typed parse
node's fields (internally using a `Tree::SiblingIterator`) for easy
access. However, this is not as fast as directly observing the tree
structure using the postorder strategy being used by the check stage.
These functions rely on using struct reflection on the typed parse node
definitions from `parse/typed_nodes.h` to get the expected structure of
child nodes and then populate them.
Note that validating these in `Tree::Verify` adds significant cost to
it, and is currently included in the parsing stage. Without this change,
a 10 mloc test case of lex & parse takes 4.129 s ± 0.041 s. With this
change, it takes 5.768 s ± 0.036 s.
This builds upon and completes #3393.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Choosing to make the constructor explicit in the test, rather than
NOLINT, because it seems to better reflect how our code is usually
written (and may be more likely to trip an issue).
Per their documentation, the `llvm::Initialize*` functions are only
supposed to be called by the main program, not by a library like
toolchain/codegen. Fixes a hang due to a data race in multithreaded
autoupdate.
Add a utility class `Carbon::InitLLVM` to do the common LLVM
initialization shared by all Carbon tools, optionally including
initializing the LLVM targets. Because the LLVM targets add a lot of
binary size, only initialize them for binaries that opt in by depending
on a new target `//common:all_llvm_targets`.
Also fix `//explorer:file_test` and `//explorer:file_test.trace` to
share a binary rather than linking an identical binary twice.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Per [#toolchain
discussion](https://discord.com/channels/655572317891461132/655578254970716160/1176632520834560211)
We'd at one point been trying to put `[[nodiscard]]` everywhere, but
then we stopped because it had felt verbose without finding many issues
(plus, people plain forgot to add it). Some history in #888.
Since newer code gets added without it, we now have code like:
```
auto GetLineInfo(Line line) -> LineInfo&;
[[nodiscard]] auto GetLineInfo(Line line) const -> const LineInfo&;
auto AddLine(LineInfo info) -> Line;
auto GetTokenInfo(Token token) -> TokenInfo&;
[[nodiscard]] auto GetTokenInfo(Token token) const -> const TokenInfo&;
auto AddToken(TokenInfo info) -> Token;
[[nodiscard]] auto GetTokenPrintWidths(Token token) const -> PrintWidths;
```
Here, the lack of `[[nodiscard]]` doesn't mean anything: for example,
`GetLineInfo` should not have its result discarded if it's called. But
the mix could be confusing for readers.
As a resolution, remove the attribute. `[[nodiscard]]` should be treated
like other attributes going forward, which essentially means "avoid in
general, add a comment to explain why the attribute is needed" rather
than use-as-default.
Provides an adapter for the llvm::yaml API because it otherwise needs a
bunch of const/non-const definitions, and the traits are difficult to
diagnose issues with. The current approach is pretty simple to use, even
if it's not super efficient (which, yaml output is more of a debugging
thing so I'm not really expecting it to be an issue).
Changes the format of yaml output to provide more index information,
just as reminders when seeing something like `node+0`. Note this would
create more churn in deltas if we were reliant on the output yaml in
tests, but we aren't so it should be okay.
Running `bazel test //...` reported:
```
Test execution time outside of range for MODERATE tests.
Consider setting timeout="short" or size="small".
```
This change adds size="small" to avoid such warnings being reported.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Replace `SemIR::Node::GetAsFoo` and `SemIR::Node::Foo::Make` with
`SemIR::Foo` class that represents a particular kind of node, with named
fields.
Rename `SemIR::IntegerLiteral` and `SemIR::RealLiteral` to
`IntegerValue` / `RealValue` to better reflect their purpose and avoid a
name collision with the corresponding `SemIR` node kinds.
Remove `NodeKind::Invalid` and the `SemIR::Node` default constructor
entirely, as they were not used for anything.
This file group exists to allow a `genquery` rule and a Python test to
verify our non-test dependency graph. We don't actually need to build
the binaries in the file group as part of that. The `genquery` rule
seems to do the right thing -- building it directly doesn't cause the
binaries in the group to be built. But without a manual tag, the group
itself is part of `:all` and thus part of `//...` and part of the rules
that will be built even with PR #3106. A consequence is that any change
to the toolchain causes several other binaries to be built as well
because this file group is in the impacted set. Making it manual should
avoid all of this, and without breaking the actual use from `genquery`.
For example, before this change, in a fully cached build after a `bazel
clean`:
```
> bazel test //bazel/check_deps:all
INFO: Invocation ID: 2d83ebee-4c00-425d-be33-23f42b079614
INFO: Analyzed 3 targets (103 packages loaded, 7137 targets configured).
INFO: Found 2 targets and 1 test target...
INFO: Elapsed time: 4.081s, Critical Path: 2.61s
INFO: 3111 processes: 2796 disk cache hit, 315 internal.
INFO: Build completed successfully, 3111 total actions
```
After this change:
```
> bazel test //bazel/check_deps:al
INFO: Invocation ID: c94089e8-a420-4d3c-9902-134e6b55b297
INFO: Analyzed 2 targets (92 packages loaded, 568 targets configured).
INFO: Found 1 target and 1 test target...
INFO: Elapsed time: 0.700s, Critical Path: 0.01s
INFO: 7 processes: 2 disk cache hit, 5 internal.
INFO: Build completed successfully, 7 total actions
```
While here, re-generate the file group, and fix several issues it
uncovers: mark test utilities as `testonly` and update our LLVM package
allowlist to include `clangd`'s package.
Rationale: this convention avoids forcing closely-related code to be far
apart in the namespace hierarchy, and vice versa. By the same token, it
makes the namespace hierarchy more consistent with the directory
hierarchy.
With the toolchain splitting namespaces, ostream.h's `operator<<`
templates aren't reliably found with name lookup, likely due to the loss
of associated namespaces (zygoloid commented on this at
https://github.com/carbon-language/carbon-lang/pull/3161#discussion_r1307941999).
This is especially a barrier to moving the lex files into `Carbon::Lex`;
versus other parts of the toolchain, they contain more printable types
which are used cross-namespace, including `Carbon::Testing`. As a
consequence, I'm looking at migrating ostream.h to a more reliable
approach that doesn't rely as much on everything being in the `Carbon`
namespace.
#2569 added PrintAsID to //common/ostream.h, but given it's
explorer-specific behavior, I don't think it's the right home for it.
Noticed this while pondering better ostream interfaces.
Continuing on #3070.
I moved ParseTree::Node to just Parse::Node, versus Parse::Tree::Node.
Other name changes are just removing "Parse" or "Parser" prefixes.
In EnumBase, I'm directly defining operator<< because the ostream.h
approach just isn't working, not for either of Parse::State nor
Parse::NodeKind. Errors look like:
```toolchain/parser/parser_context.cpp:449:34: error: invalid operands to binary expression ('llvm::raw_ostream' and 'const Carbon::Parse::State')
output << "\t" << i << ".\t" << entry.state;
~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~
```
The expected template in `Carbon::` is not in the error list; I only see
the:
```
./common/ostream.h:112:6: note: candidate template ignored: requirement 'std::is_base_of_v<std::ostream, llvm::raw_ostream>' was not satisfied [with S = llvm::raw_ostream, T = Carbon::Parse::State]
auto operator<<(S& standard_out, const T& value) -> S& {
^
```
I'm still prodding at this, but not seeing an obvious fix.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
The different approach to Names avoids the issues with trying to define
a static member (or also member function) of the templated instance of
Carbon::Internal::EnumBase from a non-enclosing namespace such as
Carbon::SemIR.
Note, I'm trying to do this from the cpp file. An alternative might be
to do `inline constexpr llvm::StringLiteral Names[]` in the .h file, but
I think concerns had been raised about that needing deduplication.
Splits IR files into SemIR, and logic files into Check. These will be
split into separate directories as part of a later move; the namespaces
are being done first in order to vet the switch, and hopefully make
conflicts a little easier to manage due to the substantial renames.
A lot of this is just automated removal of Semantics prefixes from
names, adding namespace references where needed. A few special-cases
are:
- SemanticsIR -> SemIR::File
- A few things were discussed, like Unit, CompileUnit, or CompiledUnit.
Unit was too vague for chandlerc, and I thought CompileUnit might lead
to incorrect inferences (CompilationUnit would be more precise, but
typically written as SemIR::CompilationUnit which is pretty long). File
seemed to be a short name that we could agree on.
- SemanticsIRFormatter -> SemIR::Formatter
- FormatSemanticsIR -> SemIR::FormatFile
- SemanticsFileTest -> CheckFileTest
- It remains in the Testing namespace, where just "FileTest" might be
too broad a name.
- SemanticsDeclarationNameStack::Context ->
Check::DeclarationNameStack::NameContext
- This avoids a Check::Context name shadowing.
Changes check_internal.h to include ostream.h to improve finding of
Print/operator<< (otherwise it didn't compile).
This is part of #3070
The main motivation for this is to get python loads in using the
`native-py` lint fix. However, enabling that made me wonder, maybe we
should fix in general?
`native-cc` is delayed, but not wholly cancelled (and `native-py`
picking up might indicate `native-cc` won't be too far behind). There's
also some automated fixes for `.append` and dict sorting -- this felt
okay to me, maybe not something to eagerly add but probably not worth
stopping buildifier from fixing (I've noticed the warnings in the past
and had been ignoring them).
Running everything does mean that load orders are sorted automatically
now, which I think is a positive. Most generally, I think these fixes
aren't _harmful_, and having them done automatically seems beneficial:
my biggest concern about `native-py` and `native-cc` was actually that
regressions wouldn't be caught, but this addresses that issue
automatically.
This library is designed around supporting the kinds of use cases we
expect in the toolchain and other Carbon tools. It supports subcommands,
options, and prints help.
There is still a decent chunk of work to be done to finish polishing
this, but it should give us a solid starting point.
For details about the library and a brief roadmap, see the main comment
in `command_line.h` which provides a comprehensive overview of the
library and a roadmap of the remaining work.
Porting the driver to this went fairly well, but did require some
changes. That port is separated into a follow-up PR #2979.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Lucile Rose Nihlen <luci.the.rose@gmail.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Putting fuzzer files under //testing to emphasize the testonly aspect
(consolidates bazel and common subdirectories). The attributes on
explorer_fuzzer are also a little skewed from what's desirable; it's
been working okay, but this should still be a refinement.
I think there's more we can do here, but this seemed like a good
checkpoint to make sure the path I'm going down is roughly what you
expected. There's one actual edit in if expression structure to match
the increased enforcement.
Replacing direct raw_string_ostream uses. I figure the wrapper should be used more consistently.
There are still remaining raw_string_ostream uses that weren't compatible -- I'm continuing to look at those, but felt it was cleaner to have this on its own.
This is just a cleanup. Since we now have a testing directory, I think this is a better home for testonly libraries than //common. (I was thinking about this when I was considering adding more test_raw_ostream deps)
This is a convenient test helper for anything that can use injected
streams. Extract this to where it can be used by other tests and add
some basic tests, mostly documenting how it works.
protobuf deps have become complexly structured, but AFAICT this is still the preferred dep. Also fixes some the fuzzer include checking, but that's a trivial misconfig.
I'm partly doing this because the current setup would be difficult to share with the toolchain. e.g., ProtoToCarbon isn't explorer-specific, but the only way to run it via CLI is the explorer's fuzzverter. I want a separate tool.
This change:
- Adds a //common/fuzzing:proto_to_carbon tool.
- The rest of fuzzverter is now just //explorer/fuzzing:ast_to_proto.
- The change simplifies overall handling and removes a LLVM CLI dependency.
- Stops allowing unknown fields in the proto.
- This has mostly led to forgetting to remove fuzzer inputs that were for removed features.
- Moves more non-explorer-specific bits to //common/fuzzing.
- Cleans up remaining pieces in //explorer/fuzzing
- Merges the //explorer/fuzzing proto tests, which deduplicates AstToString copies.
- These tests also had duplicate dependencies, etc -- and all complete in ~6s.
- Updates and fixes regen_corpus which was previously broken by other changes.
- Updates the README to reflect changes.
- Removes obsolete proto-fuzzer build configuration (AFAICT this is no longer needed).
Removes `__continuation`, `__await`, and `__run`.
In part here, the discussion was that while the feature had been useful for validating the early explorer design, it's no longer needed for that role as the explorer is now quite robust. Continuations have been experimental and, at this point, don't have an owner pushing to a proposal.
The triggering factor is that, as we push to address fuzzer issues, I ran into a crash bug in this code; basically, `fn Main() -> i32 { __await; return 0; }`. When I mentioned this, the reaction seemed to trend towards removal of the feature.
This keeps all of our C++ code written in C++ source files, and avoids the increasing size of the RTTI generation script we'd started to see in #2699.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This PR is making two main changes to the Explorer and Toolchain:
- Replace the `is` keyword in `where SomeType is SomeInterface` with `impls`, so it is `where SomeType impls SomeInterface`
- Rewrite uses of the "impls" to something else to avoid, frequently "`impl` declarations" or "implementations", to avoid confusion with the `impls` keyword.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
The strategy that we use for now to support template instantiation is to check the impl declaration as if it were a generic, but to defer all checking of the impl definition until we see a use in which all template parameters have arguments. At that point, we clone the impl definition and type-check the whole thing, with constant values set on the template parameters corresponding to the given arguments.
No caching of template instantiations is performed yet; each time we form a reference to a template instantiation, we instantiate it afresh. We also don't implement the name lookup rule from #949 yet; lookups during template instantiation look only in the actual type and not in the constraint.
Depends on #2699
This is intended to be used for template instantiation, but for now takes no stance as to what it's cloning.
This is tested by parsing and cloning all of our test files, and checking that the result of converting each AST to proto is the same as the result of cloning and then converting each AST to proto.
This is in preparation for template instantiation being triggered during substitution, and being able to fail.
Fix rule-of-three violation (missing assignment operator) in `Error` that got in the way of using it to hold an error temporarily in a failed transformation.
Richard Smith
Jon Ross-Perkins
cui fliter
Calvin
Chandler Carruth
Jonathan B. CoeandChandler Carruth
CanftIn
kshokhin