This extracts the C-string `argv`-like building routine to a more
broadly reusable location. It also sinks the verbose logging logic out
of it and into the relevant runners. In turn, it simplifies the verbose
logging logic significantly.
The biggest functional change is removing the implicit synthesis of a
tool's `-v` verbose flag from the presence of a `vlog` stream. I thought
this would be helpful, but in practice of debugging these layers it has
been more of a hindrance than a help -- I pretty often only want verbose
logging on one side or the other, and we have ways of explicitly passing
a `-v` flag to the underlying tools already. I think my instinct to do
this was just wrong, so rip it out and simplify.
This does add an unused feature -- prepending a prefix of arguments
while building the C-string variant. This isn't used in this PR but will
be used in subsequent PRs and it seemed more disruptive to undo that
logic and then re-do it in a later PR. Let me know if it's too confusing
here.
Assisted-by: Gemini Code Assist
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Note, not trying to address every last error, just some obvious/easy ones.
```
/usr/local/google/home/jperkins/dev/carbon-lang/common/string_helpers.cpp:200:13: warning: prefer transparent functors 'less_equal<>' [modernize-use-transparent-functors]
auto le = std::less_equal<const char*>();
^
/usr/local/google/home/jperkins/dev/carbon-lang/toolchain/semantics/nodes/function.h:22:45: warning: pass by value and use std::move [modernize-pass-by-value]
Function(ParseTree::Node node, NodeId id, llvm::SmallVector<NodeRef> body)
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/ast/declaration.cpp:230:14: warning: static member accessed through instance [readability-static-accessed-through-instance]
return cast<SelfDeclaration>(declaration).name();
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/interpreter/type_checker.cpp:1104:33: warning: std::move of the variable 'impl' of the trivially-copyable type 'ConstraintType::ImplConstraint' has no effect [performance-move-const-arg]
impl_constraints_.push_back(std::move(impl));
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/interpreter/type_checker.cpp:1145:36: warning: std::move of the variable 'rewrite' of the trivially-copyable type 'ConstraintType::RewriteConstraint' has no effect [performance-move-const-arg]
rewrite_constraints_.push_back(std::move(rewrite));
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/interpreter/type_checker.cpp:1156:32: warning: std::move of the variable 'context' of the trivially-copyable type 'ConstraintType::LookupContext' has no effect [performance-move-const-arg]
lookup_contexts_.push_back(std::move(context));
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/fuzzing/fuzzverter.cpp:74:42: warning: argument name 'trace' in comment does not match parameter name 'parser_debug' [bugprone-argument-comment]
/*trace=*/false);
^
./explorer/syntax/parse.h:19:17: note: 'parser_debug' declared here
bool parser_debug) -> ErrorOr<Carbon::AST>;
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/fuzzing/proto_to_carbon_test.cpp:34:55: warning: argument name 'trace' in comment does not match parameter name 'parser_debug' [bugprone-argument-comment]
const ErrorOr<AST> ast = Carbon::Parse(&arena, f, /*trace=*/false);
^
./explorer/syntax/parse.h:19:17: note: 'parser_debug' declared here
bool parser_debug) -> ErrorOr<Carbon::AST>;
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/fuzzing/proto_to_carbon_test.cpp:42:41: warning: argument name 'trace' in comment does not match parameter name 'parser_debug' [bugprone-argument-comment]
&arena, f, source_from_proto, /*trace=*/false);
^
./explorer/syntax/parse.h:25:59: note: 'parser_debug' declared here
std::string_view file_contents, bool parser_debug)
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/syntax/parse_test.cpp:27:60: warning: argument name 'trace' in comment does not match parameter name 'parser_debug' [bugprone-argument-comment]
ParseFromString(&arena, "file.carbon", FileContents, /*trace=*/false);
^
./explorer/syntax/parse.h:25:59: note: 'parser_debug' declared here
std::string_view file_contents, bool parser_debug)
^
/usr/local/google/home/jperkins/dev/carbon-lang/migrate_cpp/cpp_refactoring/var_decl.cpp:57:58: warning: string concatenation results in allocation of unnecessary temporary strings; consider using 'operator+=' or 'string::append()' instead [performance-inefficient-string-concatenation]
segments.push_back({type_loc_class, qual_str + " " + range_str});
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/fuzzing/ast_to_proto_test.cpp:105:55: warning: argument name 'trace' in comment does not match parameter name 'parser_debug' [bugprone-argument-comment]
const ErrorOr<AST> ast = Carbon::Parse(&arena, f, /*trace=*/false);
^
./explorer/syntax/parse.h:19:17: note: 'parser_debug' declared here
bool parser_debug) -> ErrorOr<Carbon::AST>;
^
```
* test cases for raw string literals
* raw string literal implementation
* match as block string if starting with triple ", and better error message for simple string
except for *#"""#*
* fix broken test case
block string literal cannot be one line
* test cases for raw string literals
* raw string literal implementation
* match as block string if starting with triple ", and better error message for simple string
except for *#"""#*
* fix broken test case
block string literal cannot be one line
* removed unused initial value
* rename flag to indicate multi-line string and remove comment
* use * to get value from std::optional
* clean-ups
* removed skip_scan flag and directly return in case of a single line string starting with #+\'\'\'
* Updated error message: simple string -> single-line string.
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
* Updated test cases according to changes in error message
* Removed counting_hashtag flag.
* Implemented ScanHelper class to handle scanning
* Fixed explanation of ReadHashTags.
* Addressed PR comment.
* Clarify that scan_helper holds the source text.
* Addressed PR comments.
* Updated error messages in test cases.
* Added const keyword to return type of GetCurrentStr().
* addressed PR comments.
1. Moved ScanHelper class to lex_scan_helper.h and lex_scan_helper.cpp.
2. Moved ReadHashTags and Process* functions to lex_scan_helper.cpp. Moved YY_USER_ACTION, SIMPLE_TOKEN and ARG_TOKEN to lex_helper.h. Added a wrapper function YyinputWrapper to call static function yyinput in lexer.lpp.
3. Renamed ScanHelper with StringLexHelper.
4. Modified BUILD accordingly.
5. Renamed data members and functions.
* Addressed PR comments.
1. Adjusted order to keep ret usage close.
2. Used resize to construct the string to avoid creation of temp string.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Removed the multi_line flag and skip_read field to improve readability.
* Copied default parameter value to definition of UnescapeStringLiteral.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Copied default parameter value to definition of ParseBlockStringLiteral.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Prefix CARBON_ to SIMPLE_TOKEN and ARG_TOKEN macros.
* Rollback redefinition of arguments.
* Updated comment on the flex macro.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Updated wording.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Moved the EOF error out of the loop.
* Removed duplicated declaration.
* Changed type of `hashtag_num` and `leading_quotes` to int.
* Minor fix: string copy.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Added comment on YyinputWrapper.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Garmmar in comment.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Added check of eof before readling next char.
* Minor updates based on PR comments.
* Minor changes to address PR comments.
* Used a clearer way to calculate `hashtag_num` and `leading_quotes`. Switched back to indicate muti-line string with a flag.
* Directly copy StringRef for compilation error message.
* Make str_with_quote const as we don't change it.
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
* Added TODO for unsupported cases.
* Fixed a typo.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
I'm doing this to avoid macro name conflicts, following https://google.github.io/styleguide/cppguide.html#Preprocessor_Macros: "If you do export a macro from a header, it must have a globally unique name. To achieve this, it must be named with a prefix consisting of your project's namespace name (but upper case)."
Commands run:
```
sed -i 's/\(DCHECK\|CHECK\|FATAL\|MAKE_UNIQUE_NAME\|MAKE_UNIQUE_NAME_IMPL\|RAW_EXITING_STREAM\|RETURN_IF_ERROR\|RETURN_IF_ERROR_IMPL\|ASSIGN_OR_RETURN\|ASSIGN_OR_RETURN_IMPL\|DIAGNOSTIC_KIND\|RETURN_IF_STACK_LIMITED\)(/CARBON_\1(/g' $(git ls-files *.cpp *.h *.lpp *.ypp *.def ':!third_party')
sed -i 's/#undef DIAGNOSTIC_KIND/#undef CARBON_DIAGNOSTIC_KIND/' toolchain/diagnostics/diagnostic_registry.def
```
Note this isn't *quite* everything, but it's intended to be a large pass at everything:
```
╚╡git grep '#define ' *.cpp *.h *.lpp *.ypp *.def ':!third_party' | grep -v '#define CARBON' | grep -v _H_
explorer/syntax/lexer.lpp: #define YY_USER_ACTION \
explorer/syntax/lexer.lpp: #define SIMPLE_TOKEN(name) \
explorer/syntax/lexer.lpp: #define ARG_TOKEN(name, arg) \
explorer/syntax/parse_and_lex_context.h:#define YY_DECL \
migrate_cpp/cpp_refactoring/var_decl.cpp:#define ABSTRACT_TYPE(Class, Base)
migrate_cpp/cpp_refactoring/var_decl.cpp:#define TYPE(Class, Base) \
```
We may in particular want to do a pass to clean up #ifdef guards and make them be CARBON_ rooted.
* Initial implementation of block string literals following lexical_conventions/string_literals.md.
Enabled yyinput() in flex to implement parsing.
Added ParseBlockStringLiteral() helper to handler further transformations such as indenting.
Modified formar_grammar to support single-quoted strings to prevent a failure on lexer.lpp.
* Fixed _find_string_end quote parameter type int -> str.
* Update executable_semantics/syntax/BUILD
Co-authored-by: Geoff Romer <gromer@google.com>
* Addressed code review comments - split table-drived test into individual tests, renamed constants to match style guide.
* Addressed code review comments -- using EXPECT_THAT_EXPECTED() in tests, lexer comments and code cleanup.
Co-authored-by: Geoff Romer <gromer@google.com>
This is based on discussion on #732: that we should probably parse the invalid whitespace, then reject it as part of string validation, rather than having different parses. I worry the question of "how is this parsed" may lead to subtly unexpected results if we aren't consistent, so I'm switching the logic from the lexer to the unescape library (and also adjusting the list of rejected whitespace).
It was in my mind to add String in order to support libraries in `package`. `print` is added in order to have a String go to stdout. I've tried to do `print` in a way that won't be too hard to add other printable types, but it's probably also somewhat optional here -- that is, if desired, I could remove it. But it was a lot easier to doublecheck `\n` behavior with it, and I suspect it'll be helpful in other tests if it supports more value types.
On the side, this also fixes dereferencing in Pattern/Expression Print() calls, which I was noticing printing pointers instead of values. This may be another argument for moving away from passing pointers, since this seems to be a difficult-to-catch error.
Co-authored-by: Geoff Romer <gromer@google.com>
Chandler Carruth
Thomas Köppe
Jon Ross-Perkins
Priyananda Shenoy
Vineeth B V
Jon Meow
pk19604014andGeoff Romer