From c4fecf720f2776dbd2ea88da038177ff8c92316b Mon Sep 17 00:00:00 2001 From: Richard Smith Date: Fri, 6 May 2022 14:26:55 -0700 Subject: [PATCH] Bitwise operators (#1191) Add bitwise and bit-shift operators `&`, `|`, `^`, `<<`, `>>`. Replace C++ `~` with unary prefix `^`. Co-authored-by: Chandler Carruth --- docs/design/expressions/README.md | 42 +- docs/design/expressions/arithmetic.md | 22 +- docs/design/expressions/bitwise.md | 283 ++++++++++++ .../expressions/implicit_conversions.md | 16 + proposals/p1191.md | 401 ++++++++++++++++++ 5 files changed, 753 insertions(+), 11 deletions(-) create mode 100644 docs/design/expressions/bitwise.md create mode 100644 proposals/p1191.md diff --git a/docs/design/expressions/README.md b/docs/design/expressions/README.md index de60f8cc9da8..3603f2dfc493 100644 --- a/docs/design/expressions/README.md +++ b/docs/design/expressions/README.md @@ -65,6 +65,12 @@ graph BT click memberAccess "https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/expressions/member_access.md" negation["-x"] + click negation "https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/expressions/arithmetic.md" + + complement["^x"] + click complement "https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/expressions/bitwise.md" + + unary((" ")) as["x as T"] click as "https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/expressions/implicit_conversions.md" @@ -80,6 +86,17 @@ graph BT modulo["x % y"] click modulo "https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/expressions/arithmetic.md" + bitwise_and>"x & y"] + bitwise_or>"x | y"] + bitwise_xor>"x ^ y"] + click bitwise_and "https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/expressions/bitwise.md" + click bitwise_or "https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/expressions/bitwise.md" + click bitwise_xor "https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/expressions/bitwise.md" + + shift["x << y
+ x >> y"] + click shift "https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/expressions/bitwise.md" + comparison["x == y
x != y
x < y
@@ -91,12 +108,16 @@ graph BT not["not x"] click not "https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/expressions/logical_operators.md" + logicalOperand((" ")) + and>"x and y"] click and "https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/expressions/logical_operators.md" or>"x or y"] click or "https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/expressions/logical_operators.md" + logicalExpression((" ")) + if>"if x then y else z"] click if "https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/expressions/if.md" @@ -104,18 +125,23 @@ graph BT memberAccess --> parens & braces & unqualifiedName negation --> memberAccess + complement --> memberAccess + unary --> negation & complement %% Use a longer arrow here to put `not` next to `and` and `or`. not -----> memberAccess - multiplication & modulo & as --> negation + multiplication & modulo & as & bitwise_and & bitwise_or & bitwise_xor & shift --> unary addition --> multiplication - comparison --> modulo & addition & as - and & or --> comparison & not - if & expressionEnd --> and & or + comparison --> modulo & addition & as & bitwise_and & bitwise_or & bitwise_xor & shift + logicalOperand --> comparison & not + and & or --> logicalOperand + logicalExpression --> and & or + if & expressionEnd --> logicalExpression ``` The diagram's attributes are: -- Each node represents a precedence group. +- Each non-empty node represents a precedence group. Empty circles are used to + simplify the graph, and do not represent a precedence group. - When an expression is composed from different precedence groups, the interpretation is determined by the precedence edges: @@ -221,11 +247,17 @@ Most expressions are modeled as operators: | Category | Operator | Syntax | Function | | ---------- | ------------------------------- | --------- | --------------------------------------------------------------------- | | Arithmetic | [`-`](arithmetic.md) (unary) | `-x` | The negation of `x`. | +| Bitwise | [`^`](bitwise.md) (unary) | `^x` | The bitwise complement of `x`. | | Arithmetic | [`+`](arithmetic.md) | `x + y` | The sum of `x` and `y`. | | Arithmetic | [`-`](arithmetic.md) (binary) | `x - y` | The difference of `x` and `y`. | | Arithmetic | [`*`](arithmetic.md) | `x * y` | The product of `x` and `y`. | | Arithmetic | [`/`](arithmetic.md) | `x / y` | `x` divided by `y`, or the quotient thereof. | | Arithmetic | [`%`](arithmetic.md) | `x % y` | `x` modulo `y`. | +| Bitwise | [`&`](bitwise.md) | `x & y` | The bitwise AND of `x` and `y`. | +| Bitwise | [`\|`](bitwise.md) | `x \| y` | The bitwise OR of `x` and `y`. | +| Bitwise | [`^`](bitwise.md) (binary) | `x ^ y` | The bitwise XOR of `x` and `y`. | +| Bitwise | [`<<`](bitwise.md) | `x << y` | `x` bit-shifted left `y` places. | +| Bitwise | [`>>`](bitwise.md) | `x >> y` | `x` bit-shifted right `y` places. | | Conversion | [`as`](as_expressions.md) | `x as T` | Converts the value `x` to the type `T`. | | Comparison | [`==`](comparison_operators.md) | `x == y` | Equality: `true` if `x` is equal to `y`. | | Comparison | [`!=`](comparison_operators.md) | `x != y` | Inequality: `true` if `x` is not equal to `y`. | diff --git a/docs/design/expressions/arithmetic.md b/docs/design/expressions/arithmetic.md index 54d91eb2f3d1..6b3d2fcb395e 100644 --- a/docs/design/expressions/arithmetic.md +++ b/docs/design/expressions/arithmetic.md @@ -55,11 +55,17 @@ standard library. ## Precedence and associativity ```mermaid -graph TD - negation["-x"] --> multiplicative & modulo - multiplicative>"x * y
x / y"] --> additive - additive>"x + y
x - y"] +%%{init: {'themeVariables': {'fontFamily': 'monospace'}}}%% +graph BT + negation["-x"] + multiplication>"x * y
+ x / y"] + addition>"x + y
+ x - y"] modulo["x % y"] + + multiplication & modulo --> negation + addition --> multiplication ``` [Instructions for reading this diagram.](README.md#precedence) @@ -153,8 +159,12 @@ programming errors: will be aborted, or the arithmetic will evaluate to a mathematically incorrect result, such as a two's complement result or zero. The program might not in all cases be aborted immediately -- for example, multiple - overflow checks might be combined into one, and if the result of an - arithmetic operation is never observed, the abort may not happen at all. + overflow checks might be combined into one -- but no control flow or memory + access that depends on the value will be performed. + +**TODO:** Unify the description of these programming errors with those of +bit-shift domain errors, document the behavior in a common place and link to it +from here. **TODO:** In a hardened build, should we prefer to trap on overflow, give a two's complement result, or produce zero? Using zero may defeat some classes of diff --git a/docs/design/expressions/bitwise.md b/docs/design/expressions/bitwise.md new file mode 100644 index 000000000000..87b0d775d373 --- /dev/null +++ b/docs/design/expressions/bitwise.md @@ -0,0 +1,283 @@ +# Bitwise and shift operators + + + + + +## Table of contents + +- [Overview](#overview) +- [Precedence and associativity](#precedence-and-associativity) +- [Integer types](#integer-types) +- [Integer constants](#integer-constants) +- [Extensibility](#extensibility) +- [Alternatives considered](#alternatives-considered) +- [References](#references) + + + +## Overview + +Carbon provides a conventional set of operators for operating on bits: + +``` +var a: u8 = 5; +var b: u8 = 3; +var c: i8 = -5; + +// 250 +var complement: u8 = ^a; +// 1 +var bitwise_and: u8 = a & b; +// 7 +var bitwise_or: u8 = a | b; +// 6 +var bitwise_xor: u8 = a ^ b; +// 40 +var left_shift: u8 = a << b; +// 2 +var logical_right_shift: u8 = a >> 1; +// -3 +var arithmetic_right_shift: i8 = c >> 1; +``` + +These operators have [predefined meanings](#integer-types) for Carbon's integer +types. User-defined types can define the meaning of these operations by +[implementing an interface](#extensibility) provided as part of the Carbon +standard library. + +## Precedence and associativity + +```mermaid +%%{init: {'themeVariables': {'fontFamily': 'monospace'}}}%% +graph BT + complement["^x"] + bitwise_and>"x & y"] + bitwise_or>"x | y"] + bitwise_xor>"x ^ y"] + shift["x << y
+ x >> y"] + bitwise_and & bitwise_or & bitwise_xor & shift --> complement +``` + +[Instructions for reading this diagram.](README.md#precedence) + +Parentheses are required when mixing different bitwise and bit-shift operators. +Binary `&`, `|`, and `^` are left-associative. The bit-shift operators `<<` and +`>>` are non-associative. + +``` +// ✅ Same as (1 | 2) | 4, evaluates to 7. +var a: i32 = 1 | 2 | 4; + +// ❌ Error, parentheses are required to distinguish between +// (3 | 5) & 6, which evaluates to 6, and +// 3 | (5 & 6), which evaluates to 7. +var b: i32 = 3 | 5 & 6; + +// ❌ Error, parentheses are required to distinguish between +// (1 << 2) << 3, which evaluates to 4 << 3 == 32, and +// 1 << (2 << 3), which evaluates to 1 << 16 == 65536. +var c: i32 = 1 << 2 << 3; + +// ❌ Error, can't repeat the `^` operator. Use `^(^4)` or simply `4`. +var d: i32 = ^^4; +``` + +## Integer types + +Bitwise and bit-shift operators are supported for Carbon's built-in integer +types, and, unless that behavior is [overridden](#extensibility), for types that +can be implicitly converted to integer types, as follows: + +For binary bitwise operators, if one operand has an integer type and the other +operand can be implicitly converted to that type, then it is. If both operands +are of integer type, this results in the following conversions: + +- If the types are `uN` and `uM`, or they are `iN` and `iM`, the operands are + converted to the larger type. +- If one type is `iN` and the other type is `uM`, and `M` < `N`, the `uM` + operand is converted to `iN`. + +A built-in binary bitwise `&`, `|`, or `^` operation is performed if, after the +above conversion step, the operands have the same integer type. The result type +is that type, and the result value is produced by applying the relevant +operation -- AND, OR, or XOR -- to each pair of corresponding bits in the input, +including the sign bit for a signed integer type. + +A built-in complement operation is performed if the operand can be implicitly +converted to an integer type. The result type is that type, and the result value +is produced by flipping all bits in the input, including the sign bit for a +signed integer type. `^a` is equivalent to `a ^ x`, where `x` is the +all-one-bits value of the same type as `a`. + +A built-in bit-shift operation is performed if both operands are, or can be +implicitly converted to, integer types. The result type is the converted type of +the first operand. The result value is produced by shifting the first operand +left for `<<` or right for `>>` a number of positions equal to the second +operand. Vacant positions are filled with `0` bits, except for a right shift +where the first operand is of a signed type and has a negative value, in which +case they are filled with `1` bits. + +For the purposes of bit-shifts, bits are ordered by significance, with the most +significant bit being the leftmost bit and the least significant bit being the +rightmost bit. As a consequence, in the absence of overflow a left shift is +equivalent to a multiplication by a power of 2 and a right shift is equivalent +to a division by a power of two, rounding downwards. + +The second operand of a bit-shift is required to be between zero (inclusive) and +the bit-width of the first operand (exclusive); it is a programming error if the +second operand is not within that range. + +- In a development build, they will be caught immediately when they happen at + runtime. +- In a performance build, the optimizer may assume that this programming error + does not occur. +- In a hardened build, the result will have well the defined behavior of + either aborting the program or performing a shift of an unspecified number + of bits, which if wider than the first operand will result in `0` or `-1`. + In the case where the program is aborted, the program might not in all cases + be aborted immediately -- for example, multiple checks might be combined + into one -- but no control flow or memory access that depends on the value + will be performed. + +**TODO:** Unify the description of these programming errors with those of +arithmetic overflow, document the behavior in a common place and link to it from +here. + +## Integer constants + +These operations can also be applied to a pair of integer constants, or to an +integer constant and a value of integer type, as follows: + +- If any binary bitwise or bit-shift operator is applied to two integer + constants, or the unary `^` operator is applied to an integer constant, the + result is an integer constant. Integer constants are treated as having + infinitely many high-order bits, where all but finitely many of those bits + are sign bits. For example, `-1` comprises infinitely many `1` bits. Note + that there is no difference between an arithmetic and a logical right shift + on an integer constant, because every bit always has a higher-order bit to + shift from. + - It is easy to produce extremely large numbers by left-shifting an + integer constant. For example, the binary representation of + `1 << (1 << 1000)` is thought to be substantially larger than the total + entropy in the observable universe. In practice, Carbon implementations + will set a much lower limit on the largest integer constant that they + support. +- If a binary bitwise `&`, `|`, or `^` operation is applied to an integer + constant and a value of an integer type to which the constant can be + implicitly converted, the operand that is an integer constant is implicitly + converted to the integer type and the computation is performed as described + [above](#integer-types). +- If the second operand of a bit-shift operator is an integer constant and the + first opreand is not, and the second operand is between 0 (inclusive) and + the bit-width of the first operand (exclusive), the integer constant is + converted to an integer type that can hold its value and the computation is + performed as described above. + +Other operations involving integer constants are invalid. For example, a bitwise +`&` between a `u8` and an integer constant `500` is invalid because `500` +doesn't fit into `u8`, and `1 << n` is invalid if `n` is an integer variable +because we don't know what type to use to compute the result. + +Note that the unary `^` operator applied to a non-negative integer constant +results in a negative integer constant, and the binary `^` operator gives a +negative result if exactly one of the input operands was negative. For example, +`^0 == -1` evaluates to `true`. + +## Extensibility + +Bitwise and shift operators can be provided for user-defined types by +implementing the following family of interfaces: + +``` +// Unary `^`. +interface BitComplement { + let Result:! Type = Self; + fn Op[me: Self]() -> Result; +} +``` + +``` +// Binary `&`. +interface BitAndWith(U:! Type) { + let Result:! Type = Self; + fn Op[me: Self](other: U) -> Result; +} +constraint BitAnd { + extends BitAndWith(Self) where .Result = Self; +} +``` + +``` +// Binary `|`. +interface BitOrWith(U:! Type) { + let Result:! Type = Self; + fn Op[me: Self](other: U) -> Result; +} +constraint BitOr { + extends BitOrWith(Self) where .Result = Self; +} +``` + +``` +// Binary `^`. +interface BitXorWith(U:! Type) { + let Result:! Type = Self; + fn Op[me: Self](other: U) -> Result; +} +constraint BitXor { + extends BitXorWith(Self) where .Result = Self; +} +``` + +``` +// Binary `<<`. +interface LeftShiftWith(U:! Type) { + let Result:! Type = Self; + fn Op[me: Self](other: U) -> Result; +} +constraint LeftShift { + extends LeftShiftWith(Self) where .Result = Self; +} +``` + +``` +// Binary `>>`. +interface RightShiftWith(U:! Type) { + let Result:! Type = Self; + fn Op[me: Self](other: U) -> Result; +} +constraint RightShift { + extends RightShiftWith(Self) where .Result = Self; +} +``` + +Given `x: T` and `y: U`: + +- The expression `^x` is rewritten to `x.(BitComplement.Op)()`. +- The expression `x & y` is rewritten to `x.(BitAndWith(U).Op)(y)`. +- The expression `x | y` is rewritten to `x.(BitOrWith(U).Op)(y)`. +- The expression `x ^ y` is rewritten to `x.(BitXorWith(U).Op)(y)`. +- The expression `x << y` is rewritten to `x.(LeftShiftWith(U).Op)(y)`. +- The expression `x >> y` is rewritten to `x.(RightShiftWith(U).Op)(y)`. + +Implementations of these interfaces are provided for built-in types as necessary +to give the semantics described above. + +## Alternatives considered + +- [Use different symbols for bitwise operators](/proposals/p1191.md#use-different-symbols-for-bitwise-operators) +- [Provide different operators for arithmetic and logical shifts](/proposals/p1191.md#provide-different-operators-for-arithmetic-and-logical-shifts) +- [Provide rotate operators](/proposals/p1191.md#provide-rotate-operators) +- [Guarantee the behavior of large shifts](/proposals/p1191.md#guarantee-behavior-of-large-shifts) +- [Support shifting a constant by a variable](/proposals/p1191.md#support-shifting-a-constant-by-a-variable) + +## References + +- Proposal + [#1191: bitwise and shift operators](https://github.com/carbon-language/carbon-lang/pull/1191). diff --git a/docs/design/expressions/implicit_conversions.md b/docs/design/expressions/implicit_conversions.md index 00397a213360..dc72cff454d3 100644 --- a/docs/design/expressions/implicit_conversions.md +++ b/docs/design/expressions/implicit_conversions.md @@ -140,6 +140,21 @@ except: have singleton types; see issue [#508](https://github.com/carbon-language/carbon-lang/issues/508). +In addition to the above rules, a negative integer constant `k` can be +implicitly converted to the type `uN` if the value `k` + 2N can be +exactly represented, and converts to that value. Note that this conversion +violates the "semantics-preserving" test. For example, `(-1 as u8) as i32` +produces the value `255` whereas `-1 as i32` produces the value `-1`. However, +this conversion is important in order to allow bitwise operations with masks, so +we allow it: + +``` +// We allow ^0 == -1 to convert to `u32` to represent an all-ones value. +var a: u32 = ^0; +// ^4 == -5 is negative, but we want to allow it to convert to u32 here. +var b: u32 = a & ^4; +``` + ### Same type The following conversion is available for every type `T`: @@ -214,6 +229,7 @@ types. - [Provide no implicit conversions](/proposals/p0820.md#no-conversions) - [Provide no extensibility](/proposals/p0820.md#no-extensibility) - [Apply implicit conversions transitively](/proposals/p0820.md#transitivity) +- [Do not allow negative constants to convert to unsigned types](/proposals/p1191.md#converting-complements-to-unsigned-types) ## References diff --git a/proposals/p1191.md b/proposals/p1191.md new file mode 100644 index 000000000000..62a23f5e051f --- /dev/null +++ b/proposals/p1191.md @@ -0,0 +1,401 @@ +# Bitwise and shift operators + + + +[Pull request](https://github.com/carbon-language/carbon-lang/pull/1191) + + + +## Table of contents + +- [Problem](#problem) +- [Background](#background) + - [Overflow in shift operators](#overflow-in-shift-operators) +- [Proposal](#proposal) +- [Details](#details) +- [Rationale](#rationale) +- [Alternatives considered](#alternatives-considered) + - [Use different symbols for bitwise operators](#use-different-symbols-for-bitwise-operators) + - [Use a multi-character spelling](#use-a-multi-character-spelling) + - [Don't provide an xor operator](#dont-provide-an-xor-operator) + - [Use `~`, or some other symbol, for complement](#use--or-some-other-symbol-for-complement) + - [Provide different operators for arithmetic and logical shifts](#provide-different-operators-for-arithmetic-and-logical-shifts) + - [Provide rotate operators](#provide-rotate-operators) + - [Guarantee behavior of large shifts](#guarantee-behavior-of-large-shifts) + - [Support shifting a constant by a variable](#support-shifting-a-constant-by-a-variable) + - [Converting complements to unsigned types](#converting-complements-to-unsigned-types) + + + +## Problem + +Carbon needs operations for working with bit representations of values. + +## Background + +C++ provides four bitwise operations for Boolean algebra: complement (`~`), and +(`&`), or (`|`), and xor (`^`). These are all useful in bit-maipulation code +(although `^` is used substantially less than the others). In addition, C++ +provides two bit-shift operators `<<` and `>>` that can perform three different +operations: left shift, arithmetic right shift, and logical right shift. The +meaning of `>>` is determined by the signedness of the first operand. + +C and Swift use the same set of operators as C++. Rust uses most of the same +operators, but uses `!` instead of `~` for complement, unifying it with the +logical not operator, which is spelled `not` in Carbon and as `!` in Rust and +C++. Go uses most of the same operators as C++, but uses unary prefix `^` +instead of `~` for complement, mirroring binary `^` for xor. + +In addition to the operators provided by C and C++, bit-rotate operators are +present in some languages, and a short notation for them may be convenient. +Finally, there are other non-degenerate binary Boolean operations with no common +operator symbol: + +- The "implies" operator (equivalent to `~a | b`). +- The "implied by" operator (equivalent to `a | ~b`). +- The complement of each of the other operators (NAND, NOR, XNOR, "does not + imply", "is not implied by"). + +### Overflow in shift operators + +The behavior of shift operators in C++ had a turbulent past. The behavior of +shift operators has always been undefined if the right operand is out of range +-- not between zero inclusive and the bit-width of the left operator exclusive +-- but other restrictions have varied: + +- Unsigned left shift has never had any restrictions on the first operand. +- For signed left shift: + - In C++98, the result was fully unspecified. + - In C++11, the result was specified only if the first operand was + non-negative and the result fit into the destination type -- that is, if + no 1 bit is shifted into the sign bit. + - In C++14, the result was specified only if the first operand was + non-negative and the result fit into the unsigned type corresponding to + the destination type -- that is, if no 1 bit is shifted out of the sign + bit. + - In C++20 onwards, there are no restrictions beyond a range restriction + on the right operand, and the result is otherwise always specified, even + if the left operand is negative. +- Unsigned right shift has never had any restrictions on the first operand. +- For signed right shift: + - In C++17 and earlier, if the left operand is negative, the result is + implementation-defined. + - In C++20 onwards, the result is always specified, even if the left + operand is negative. + +There is a clear trend towards defining more cases, following two's complement +rules. + +## Proposal + +Use the same operator set as C++, but replace `~` with unary prefix `^`. + +Define the behavior for all cases of `<<` and `>>` except where the right +operand is either negative or is greater than or equal to the bit-width of the +left operand. + +## Details + +See changes to the design, and in particular +[the new section on bitwise and shift operators](/docs/design/expressions/bitwise.md). + +## Rationale + +- [Performance-critical software](/docs/project/goals.md#performance-critical-software) + - Bit operations are important low-level primitives. Providing operators + for them is important in order to allow low-level high-performance code + to be written elegantly in Carbon. + - By not defining semantics for `<<` and `>>` when the right-hand operand + is out of range, we can directly use hardware instructions for these + operations whose behavior in these cases vary between architectures. +- [Code that is easy to read, understand, and write](/docs/project/goals.md#code-that-is-easy-to-read-understand-and-write) + - Using operator notation rather than function call notation for bitwise + operators improves readability in code making heavy use of these + operations. +- [Practical safety and testing mechanisms](/docs/project/goals.md#practical-safety-and-testing-mechanisms) + - Carbon follows C++ in treating `<<` and `>>` as programming errors when + the right hand operand is out of range, but Carbon guarantees that such + errors will not directly result in unbounded misbehavior in hardened + builds. +- [Modern OS platforms, hardware architectures, and environments](/docs/project/goals.md#modern-os-platforms-hardware-architectures-and-environments) + - All hardware architectures we care to support are natively two's + complement architectures, and that assumption allows us to define the + semantics of shift operators in the way that makes the most sense for + such architectures. + - Our bitwise operations make no assumptions about the endianness of the + hardware architecture, although the shift operators make the most sense + on a little-endian or big-endian architecture, which are the only + endiannesses we expect to see in modern hardware platforms. +- [Interoperability with and migration from existing C++ code](/docs/project/goals.md#interoperability-with-and-migration-from-existing-c-code) + - The same set of operators is provided as in C++, making it easy for + programmers and programs to migrate, with the exception that the `~` + operator is mechanically replaced with `^`. This change is expected to + be easy for programmers to accommodate, especially given that Rust's + choice to replace `~` with `!` does not seem to be a source of sustained + complaints. + - The extensiblity support reflects the full scope of operator overloading + in C++, permitting separate overloading of each of the bitwise operators + with custom return types. This should allow smooth interoperability with + C++ overloaded operators. + +## Alternatives considered + +### Use different symbols for bitwise operators + +The operator syntax for bitwise operators was decided in +[#545](https://github.com/carbon-language/carbon-lang/issues/545). Some of the +specific alternatives considered are discussed below. + +#### Use a multi-character spelling + +We considered using various multi-character spellings for the bitwise and, or, +xor, and complement operators: + +- `&:`, `|:`, `^:`, `~:` +- `.&.`, `.|.`, `.^.`, `.~.` +- `.*.`, `.+.`, `.!=.`, `.!.` +- `/\`, `\/`, `(+)`, `-|` +- `bitand`, `bitor`, `bitxor`, `compl` + +The advantage of switching to such a set of operators is that this would free up +the single-character `&`, `|`, `^`, and `~` tokens for other uses that may occur +more frequently in Carbon programs. We have some candidate uses for these +operators already: + +- `&` is used for combining interfaces and as a unary address-of operator. +- `|` may be useful for sum types, for alternatives in patterns, or as another + kind of bracket as in + [Ruby's lambda notation](https://ruby-doc.org/docs/ruby-doc-bundle/Manual/man-1.4/syntax.html#iter). +- `~` may be useful as a destructive move notation. +- `^` may be useful as a postfix pointer dereference operator. + +Other motivations for switching to a different set of spellings include: + +- There are some concerns that `<` and `<<` are visually similar, analogous to + `&` and `&&`. +- Carbon has moved away from `&&` and other punctuation based _logical_ + operators and towards keywords like `and`. Bitwise operators could similarly + switch to keywords like `bitand`. + +However, moving substantially away from the C++ operator set comes with a set of +concerns: + +- There are strong established expectations and intuitions about these + operators and their spellings among C++ practitioners. +- In some of the code that uses these operators, they are used a lot, and a + more cumbersome operator may consequently cause an outsized detriment on + readability. +- These operations are used particularly in the area of low-level, + high-performance code, which is an area for which we want Carbon to be + especially appealing. Using short operators for these operations + demonstrates our commitment to providing good support for such code. +- Even if we didn't use these operators as bit operators, we would still want + to exercise caution when using them for some other purpose to avoid + surprising C++ developers. +- While some visual similarity exists such as between `<` and `<<`, the + contexts in which these operators are used are sufficiently different to + avoid any serious concerns. +- The primary motivation of using `and` instead of `&&` doesn't apply for + bitwise operators: the _logical_ operator represents _control flow_. + Separating logical and bitwise "and" operations more visibly seems + especially important because of this control flow semantic difference. + Without any control flow and with the keywords being significantly longer + for bitwise operations, the above considerations were the dominant ones that + led us to stick with familiar `&` and `|` spellings. + +#### Don't provide an xor operator + +We considered omitting the `^` operator, providing this functionality in some +other way, such as a named function or an `xor` keyword, while keeping the `&` +and `|` symbols for bitwise operations. We could take a similar approach for the +complement operation, such as by using a `compl` keyword. The primary motivation +was to avoid spending two precious operator characters on relatively uncommon +operations. However, we did not want to apply the same change to `&` and `|`, +and it seemed important to maintain consistency between the three binary bitwise +operators from C++. + +Using `^` for both operations provides some of the benefits here, allowing us to +reclaim `~`, without introducing the inconsistency that would result from using +keywords. + +#### Use `~`, or some other symbol, for complement + +We could follow C++ and use `~` as the complement operator. However, using `~` +for this purpose spends a precious operator character on a relatively uncommon +operation, and `~` is often visually confusible with `-`, creating the potential +for readability problems. Also, in C++, `~` is overloaded to also refer to +destruction, and we may want to use the same notation for destruction or +destructive move semantics in Carbon. + +We found `^` to be a satisfying alternative with a good rationale and mnemonic: +`^` is a bit-flipping operator -- `a ^ b` flips the bits in `a` that are +specified in `b` -- and complement is an operator that flips _all_ the bits. +`^a` is equivalent to `a ^ n`, where `n` is the all-one-bits value in the type +of `a`. + +We also considered using `!` for complement, like Rust does. Unlike in Rust, +this would not be a generalization of `!` on `bool`, because we use `not` for +`bool` negation, and repurposing `!` in this way compared to C++ seemed +confusing. + +### Provide different operators for arithmetic and logical shifts + +We could provide different operators for arithmetic right shift and logical +right shift. This might allow programmers to better express their intent. +However, it seems unnecessary, as using the type of the left operand is a +strategy that doesn't appear to have caused significant problems in practice in +the languages that have followed it. + +Basing the kind of shift on the signedness of the left operand also follows from +viewing a negative number as having an infinite number of leading 1 bits, which +is the underlying mathematical model behind the two's complement representation. + +### Provide rotate operators + +We could provide bitwise rotation operators. However, there doesn't seem to be a +sufficient need to justify adding another operator symbol for this purpose. + +### Guarantee behavior of large shifts + +Logically, the behavior of shifts is meaningful for all values of the second +operand: + +- A shift by an amount greater than or equal to the bit-width of the first + operand will shift out all of the original bits, producing a result where + all value bits are the same. +- A shift in one direction by a negative amount is treated as a shift in the + opposite direction by the negation of that amount. + +Put another way, we can view the bits of the first operand as an N-bit window +into an infinite sequence of bits, with infinitely many leading sign bits (all +zeroes for an unsigned value) and infinitely many trailing zero bits after a +notional binary point, and a shift moves that window around. Or equivalently, a +shift is always a multiplication by 2N followed by rounding and +wrapping. + +We could provide the correct result for all shifts, regardless of the magnitude +of the second operand. This is the approach taken by Python, except that Python +rejects negative shift counts. The primary reason we do not do this is lack of +hardware support. For example, x86 does not have an instruction to perform this +operation. Rather, x86 provides shift instructions that mask off all bits of the +second operand except for the bottom 5 or 6, meaning that a left shift of a +64-bit operand by 64 will return the operand unchanged rather than producing +zero. + +We could instead provide x86-like behavior, guaranteeing to consider only the +lowest `N` bits of the second operand when the first operand is an `iN` or `uN`. +This is the approach taken by Java for its 32-bit `int` type and 64-bit `long` +type, where the second operand is taken modulo 32 or 64, respectively, and in +JavaScript, where operands of bitwise and bit-shift operators are treated as +32-bit integers and the second operand of a shift is taken modulo 32. This +approach would provide an operation that can be implemented by a single +instruction on x86 platforms when `N` is 32 or 64, and for all smaller types and +for all other platforms the operation can be implemented with two instructions: +a mask and a shift. For larger types, single-instruction support may not be +available, but nonetheless the performance will be close to optimal, requiring +at most one additional mask. There is still some performance cost in some cases, +but the primary reason we do not do this is the same reason we choose to not +define signed integer overflow: this masked result is unlikely to be the value +that the developer actually wanted. + +Instead of the above options, Carbon treats a second operand that is not in the +interval [0, N) as a programming error, just like signed integer overflow: + +- Debugging builds can detect and report this error without the risk of false + positives. +- Performance builds can optimize on the basis that this situation will not + occur, and can in particular use the dedicated x86 instructions that ignore + the high order bits of the second operand. +- Optimized builds guarantee that either the programming error results in + program termination or that _some_ value is produced, and moreover that said + value is the result of applying _some_ mathematical shift to the input. For + example, it's valid for an `i32` shift to be implemented by an x86 64-bit + shift that will produce 0 if the second operand is in [32, 63) but that will + treat a second operand of, say, 64 or -64 the same as 0. + +### Support shifting a constant by a variable + +We considered various ways to support + +``` +var a: i32 = ...; +var b: i32 = 1 << a; +var c: i32 = 1234 >> a; +``` + +with no explicit type specified for the first operand of a bit-shift operator. +We considered the following options: + +- Use the type of the second operand as the result type. This would be + surprising, because the type of the second operand doesn't otherwise + influence the result type of a built-in bit-shift operator. +- Use some suitable integer type that can fit the first operand. However, this + is unlikely to do the right thing for a left-shift, and will frequently pick + either a type that's too large, resulting in the program being rejected due + to narrowing, or a type that's too small, resulting in a program that has + undefined behavior due to the second operand being too large. We could apply + this approach only for right shifts, but it was deemed too inconsistent to + use different rules for left and right shifts. +- We could find a way to defer picking the type in which the operation is + performed until later. For example, we could treat `1 << a` as a value of a + new type that carries its left-hand operand as a type parameter and its + right-hand operand as runtime state, and allow that type to be converted in + the same way as its integer constant. However, this would introduce + substantial complexity: reasonable and expected uses such as + ``` + var mask: u32 = (1 << a) - 1; + ``` + would require a second new type for a shifted value plus an offset, and + general support would require a facility analogous to + [expression templates](https://en.wikipedia.org/wiki/Expression_templates). + Further, this facility would allow implicit conversions that notionally + overflow, such as would happen in the above example when `a` is greater + than 32. + +In the absence of a good approach, we disallow such conversions for now. The +above example can be written as: + +``` +var a: i32 = ...; +var b: i32 = (1 as i32) << a; +var c: i32 = (1234 as i32) >> a; +``` + +### Converting complements to unsigned types + +We view an integer constant has having infinitely many high-order sign bits +followed by some number of lower-order value bits. As a consequence, the +complement of a positive integer constant is negative. As a result, some +important forms of initialization use a negative integer constant initializer +for an unsigned type: + +``` +// Initializer here is the integer value -8. +var mask: u32 = ^7; +``` + +We considered some options for handling this: + +- We could allow negative integer constants to convert to unsigned types if + doing so only discards sign bits. This violates the "semantics-preserving" + rule for implicit conversions. +- We could change our model of integer constants to distinguish between + "masks" -- numbers with infinitely many 1 bits preceding the value bits that + are nonetheless not considered to be negative. This was considered to + introduce too much complexity. +- We could allow conversions to unsigned types from signed types and negative + constants in general, or at least in cases where the signed operand is no + wider than the unsigned type, and perform wrapping. The latter option seems + plausible, but we don't have sufficient motivation for it, and were worried + about a risk of bugs from allowing an implicit conversion at runtime that + converts a negative value to an unsigned type. +- We could reject such initializations, with an explicit conversion required + to convert such values to unsigned types. This seems to present unacceptable + ergonomics for code performing bit-manipulation. + +On balance, our preferred option was to permit implicit conversions from +negative literals to unsigned types so long as we only discard sign bits.