* 🐛 fix BSON conformance issue Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🐛 fix BSON conformance issue Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🐛 reject ill-formed UTF-8 in CBOR/MessagePack/BSON text strings at decode time (#5531) from_cbor()/from_msgpack()/from_bson() copied the raw bytes of a decoded text string into the resulting json value without any UTF-8 validation, even though RFC 8949 §3.1 (CBOR) and the MessagePack/BSON specifications all require text strings to be valid UTF-8. Malformed input only failed later, if the value was dump()'d, with a type_error.316 - so the allow_exceptions=false pattern used specifically to get a discarded sentinel instead of an exception did not discard this category of malformed input, unlike every other kind of malformed binary input this library rejects at decode time (see #5529). Fix this at the single choke point shared by BSON/CBOR/MessagePack/UBJSON string reads, binary_reader::get_string(): validate the bytes with the UTF-8 DFA right after they are read, and report failures the same way as every other binary_reader error (parse_error.113), so allow_exceptions and strict discarding behave consistently. get_binary()/binary blob reads are untouched and still accept arbitrary bytes, since only text strings are required to be UTF-8. There were two independent implementations of a UTF-8 validator: the lexer's streaming scanner, and the serializer's Hoehrmann DFA used by dump_escaped_impl(). Rather than write a third, the serializer's decode() function, its utf8d table and the UTF8_ACCEPT/UTF8_REJECT constants are extracted into detail/string_utils.hpp (a low-level header already included before both detail/input/ and detail/output/), alongside a new is_valid_utf8() helper built on the same decode() step. serializer.hpp's dump_escaped_impl() now calls the shared decode(), so there is exactly one UTF-8 validator in the codebase; dump()'s exact type_error.316 messages and byte-index reporting are unchanged (see the added regression-guard test in unit-serialization.cpp). Claude-Session: https://claude.ai/code/session_01N4RQ1Ahan5YAGbnAQGjZTY Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * ⚡ validate only newly read bytes of binary-format strings get_string() validated the whole result after each call, but get_bytes() appends to it and CBOR indefinite-length strings collect all chunks in the same result, so every chunk re-validated everything read before it. An input of many small chunks took quadratic time (80000 one-byte chunks, 160 KB of input, took about 7 seconds). Only the newly read bytes are validated now, which also matches RFC 8949's requirement that every chunk is valid UTF-8 on its own. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
12 KiB
CBOR
The Concise Binary Object Representation (CBOR) is a data format whose design goals include the possibility of extremely small code sizes, fairly small message size, and extensibility without the need for version negotiation.
!!! abstract "References"
- [CBOR Website](http://cbor.io) - the main source on CBOR
- [CBOR Playground](http://cbor.me) - an interactive webpage to translate between JSON and CBOR
- [RFC 8949](https://www.rfc-editor.org/rfc/rfc8949.html) - the CBOR specification
Serialization
The library uses the following mapping from JSON values types to CBOR types according to the CBOR specification (RFC 8949):
| JSON value type | value/range | CBOR type | first byte |
|---|---|---|---|
| null | null |
Null | 0xF6 |
| boolean | true |
True | 0xF5 |
| boolean | false |
False | 0xF4 |
| number_integer | -9223372036854775808..-2147483649 | Negative integer (8 bytes follow) | 0x3B |
| number_integer | -2147483648..-32769 | Negative integer (4 bytes follow) | 0x3A |
| number_integer | -32768..-129 | Negative integer (2 bytes follow) | 0x39 |
| number_integer | -128..-25 | Negative integer (1 byte follow) | 0x38 |
| number_integer | -24..-1 | Negative integer | 0x20..0x37 |
| number_integer | 0..23 | Integer | 0x00..0x17 |
| number_integer | 24..255 | Unsigned integer (1 byte follow) | 0x18 |
| number_integer | 256..65535 | Unsigned integer (2 bytes follow) | 0x19 |
| number_integer | 65536..4294967295 | Unsigned integer (4 bytes follow) | 0x1A |
| number_integer | 4294967296..18446744073709551615 | Unsigned integer (8 bytes follow) | 0x1B |
| number_unsigned | 0..23 | Integer | 0x00..0x17 |
| number_unsigned | 24..255 | Unsigned integer (1 byte follow) | 0x18 |
| number_unsigned | 256..65535 | Unsigned integer (2 bytes follow) | 0x19 |
| number_unsigned | 65536..4294967295 | Unsigned integer (4 bytes follow) | 0x1A |
| number_unsigned | 4294967296..18446744073709551615 | Unsigned integer (8 bytes follow) | 0x1B |
| number_float | any value representable by a float | Single-Precision Float | 0xFA |
| number_float | any value NOT representable by a float | Double-Precision Float | 0xFB |
| string | length: 0..23 | UTF-8 string | 0x60..0x77 |
| string | length: 24..255 | UTF-8 string (1 byte follow) | 0x78 |
| string | length: 256..65535 | UTF-8 string (2 bytes follow) | 0x79 |
| string | length: 65536..4294967295 | UTF-8 string (4 bytes follow) | 0x7A |
| string | length: 4294967296..18446744073709551615 | UTF-8 string (8 bytes follow) | 0x7B |
| array | size: 0..23 | array | 0x80..0x97 |
| array | size: 24..255 | array (1 byte follow) | 0x98 |
| array | size: 256..65535 | array (2 bytes follow) | 0x99 |
| array | size: 65536..4294967295 | array (4 bytes follow) | 0x9A |
| array | size: 4294967296..18446744073709551615 | array (8 bytes follow) | 0x9B |
| object | size: 0..23 | map | 0xA0..0xB7 |
| object | size: 24..255 | map (1 byte follow) | 0xB8 |
| object | size: 256..65535 | map (2 bytes follow) | 0xB9 |
| object | size: 65536..4294967295 | map (4 bytes follow) | 0xBA |
| object | size: 4294967296..18446744073709551615 | map (8 bytes follow) | 0xBB |
| binary | size: 0..23 | byte string | 0x40..0x57 |
| binary | size: 24..255 | byte string (1 byte follow) | 0x58 |
| binary | size: 256..65535 | byte string (2 bytes follow) | 0x59 |
| binary | size: 65536..4294967295 | byte string (4 bytes follow) | 0x5A |
| binary | size: 4294967296..18446744073709551615 | byte string (8 bytes follow) | 0x5B |
Binary values with subtype are mapped to tagged values (0xD8..0xDB) depending on the subtype, followed by a byte string, see "binary" cells in the table above.
!!! success "Complete mapping"
The mapping is **complete** in the sense that any JSON value type can be converted to a CBOR value.
!!! info "NaN/infinity handling"
`NaN`, `Infinity`, and `-Infinity` are serialized as a CBOR half-precision float (type 0xF9, 3 bytes total):
`NaN` as `0xF9 0x7E 0x00`, `Infinity` as `0xF9 0x7C 0x00`, and `-Infinity` as `0xF9 0xFC 0x00`. This behavior
differs from the normal JSON serialization which serializes NaN or Infinity to `null`.
!!! note
Prior to version 3.13.0, NaN and Infinity were instead serialized as a CBOR double-precision float (type 0xFB,
9 bytes total), because the check used to select a smaller encoding compared magnitudes with NaN, which is
always `false` and caused the intended half-precision path to be skipped.
!!! info "Unused CBOR types"
The following CBOR types are not used in the conversion:
- UTF-8 strings terminated by "break" (0x7F)
- arrays terminated by "break" (0x9F)
- maps terminated by "break" (0xBF)
- byte strings terminated by "break" (0x5F)
- date/time (0xC0..0xC1)
- bignum (0xC2..0xC3)
- decimal fraction (0xC4)
- bigfloat (0xC5)
- expected conversions (0xD5..0xD7)
- simple values (0xE0..0xF3, 0xF8)
- undefined (0xF7)
- half-precision floats (0xF9)
- break (0xFF)
!!! info "Tagged items"
Binary subtypes will be serialized as tagged items. See [binary values](../binary_values.md#cbor) for an example.
??? example
```cpp
--8<-- "examples/to_cbor.cpp"
```
Output:
```c
--8<-- "examples/to_cbor.output"
```
Deserialization
The library maps CBOR types to JSON value types as follows:
| CBOR type | JSON value type | first byte |
|---|---|---|
| Integer | number_unsigned | 0x00..0x17 |
| Unsigned integer | number_unsigned | 0x18 |
| Unsigned integer | number_unsigned | 0x19 |
| Unsigned integer | number_unsigned | 0x1A |
| Unsigned integer | number_unsigned | 0x1B |
| Negative integer | number_integer | 0x20..0x37 |
| Negative integer | number_integer | 0x38 |
| Negative integer | number_integer | 0x39 |
| Negative integer | number_integer | 0x3A |
| Negative integer | number_integer | 0x3B |
| Byte string | binary | 0x40..0x57 |
| Byte string | binary | 0x58 |
| Byte string | binary | 0x59 |
| Byte string | binary | 0x5A |
| Byte string | binary | 0x5B |
| UTF-8 string | string | 0x60..0x77 |
| UTF-8 string | string | 0x78 |
| UTF-8 string | string | 0x79 |
| UTF-8 string | string | 0x7A |
| UTF-8 string | string | 0x7B |
| UTF-8 string | string | 0x7F |
| array | array | 0x80..0x97 |
| array | array | 0x98 |
| array | array | 0x99 |
| array | array | 0x9A |
| array | array | 0x9B |
| array | array | 0x9F |
| map | object | 0xA0..0xB7 |
| map | object | 0xB8 |
| map | object | 0xB9 |
| map | object | 0xBA |
| map | object | 0xBB |
| map | object | 0xBF |
| False | false |
0xF4 |
| True | true |
0xF5 |
| Null | null |
0xF6 |
| Half-Precision Float | number_float | 0xF9 |
| Single-Precision Float | number_float | 0xFA |
| Double-Precision Float | number_float | 0xFB |
!!! warning "Incomplete mapping"
The mapping is **incomplete** in the sense that not all CBOR types can be converted to a JSON value. The following CBOR types are not supported and will yield parse errors:
- simple values (0xE0..0xF3, 0xF8)
- undefined (0xF7)
Tagged items (0xC0..0xDB) are not interpreted either; see the note on tagged items below.
!!! warning "Negative integer overflow"
CBOR negative integers (major type 1) are decoded as `-1 - n`. If the encoded magnitude `n` is too large for the
result to fit into `number_integer_t` (`std::int64_t` by default), parsing fails with a
[`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) exception rather than overflowing
silently.
!!! warning "Object keys"
CBOR allows map keys of any type, whereas JSON only allows strings as keys in object values. Therefore, CBOR maps with keys other than UTF-8 strings are rejected.
!!! warning "UTF-8 validation of text strings"
[RFC 8949, Section 3.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-3.1) requires CBOR text strings
(major type 3) to be valid UTF-8. This library validates the bytes of every text string (object keys included) at
decode time and rejects ill-formed UTF-8 with a
[`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) exception (or, with
`allow_exceptions` set to `false`, a discarded value), rather than only failing later when the resulting value is
dumped. Byte strings (major type 2) are unaffected and are never validated, since they are not required to hold
text.
!!! warning "Tagged items"
Tagged items (0xC0..0xDB) will throw a parse error by default. They can be ignored by passing `cbor_tag_handler_t::ignore` to function `from_cbor`, in which case the tag is skipped and the enclosed data item is parsed on its own. They can be stored by passing `cbor_tag_handler_t::store` to function `from_cbor`. Note that no tag is ever interpreted: for instance, a text string tagged with tag 0 (date/time) stays a string.
??? example
```cpp
--8<-- "examples/from_cbor.cpp"
```
Output:
```json
--8<-- "examples/from_cbor.output"
```