mirror of https://github.com/nlohmann/json.git synced 2026-09-28 14:20:12 +01:00

Files

T

Claude 5e5a087cb3 Validate UTF-8 in CBOR/MessagePack/BSON/UBJSON/BJData text strings on develop

PR #5531 fixed the UTF-8-validation gap described in #5529, but it was
merged onto the still-unmerged bson-sizes branch rather than develop, so
develop was left with the original bug for all affected formats. A
follow-up comment on #5529 reproduced this on develop and additionally
found that UBJSON (and, by the same code path, BJData) has the identical
gap, undocumented.

Port the same fix directly onto develop: extract the UTF-8 DFA decoder
out of serializer<>::decode() into a shared detail::decode()/is_valid_utf8()
in string_utils.hpp, and call it from binary_reader::get_string() - the
single choke point shared by all five binary readers - so malformed text
strings are rejected at decode time (parse_error.113) instead of only
failing later on dump() (type_error.316). Byte/binary payloads are
unaffected. Add matching decode-time tests for CBOR, MessagePack, BSON,
UBJSON, and BJData, and document the new behavior on all five binary
format pages (the two UBJSON/BJData pages didn't get this note in #5531).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017sdieJCn6BHxzRMaXP49sP

2026-09-16 05:20:00 +00:00

6.2 KiB

Raw Blame History

BSON

BSON, short for Binary JSON, is a binary-encoded serialization of JSON-like documents. Like JSON, BSON supports the embedding of documents and arrays within other documents and arrays. BSON also contains extensions that allow representation of data types that are not part of the JSON spec. For example, BSON has a Date type and a BinData type.

!!! abstract "References"

- [BSON Website](http://bsonspec.org) - the main source on BSON
- [BSON Specification](http://bsonspec.org/spec.html) - the specification

Serialization

The library uses the following mapping from JSON values types to BSON types:

JSON value type	value/range	BSON type	marker
null	`null`	null	0x0A
boolean	`true`, `false`	boolean	0x08
number_integer	-9223372036854775808..-2147483649	int64	0x12
number_integer	-2147483648..2147483647	int32	0x10
number_integer	2147483648..9223372036854775807	int64	0x12
number_unsigned	0..2147483647	int32	0x10
number_unsigned	2147483648..9223372036854775807	int64	0x12
number_unsigned	9223372036854775808..18446744073709551615	uint64	0x11
number_float	any value	double	0x01
string	any value	string	0x02
array	any value	document	0x04
object	any value	document	0x03
binary	any value	binary	0x05

!!! warning "Incomplete mapping"

The mapping is **incomplete**, since only JSON-objects (and things contained therein) can be serialized to BSON.
Also, keys may not contain U+0000, since they are serialized a zero-terminated c-strings.

!!! warning "BSON type 0x11 interoperability"

The BSON specification defines type `0x11` as a Timestamp. This library uses marker `0x11` when serializing
`number_unsigned` values in the range `9223372036854775808..18446744073709551615`. Other BSON implementations may
therefore interpret these values as Timestamps instead of unsigned integers.

!!! info "Binary values without a subtype"

BSON requires every binary value to have a subtype. If a binary value has no subtype, this library serializes it
with the generic subtype `0x00`. After deserialization, `has_subtype()` returns `true` and `subtype()` returns `0`.
As a result, serializing and deserializing a JSON object containing such a value produces a different JSON object,
even though the binary data is unchanged.

??? example

```cpp
--8<-- "examples/to_bson.cpp"
```

Output:

```c
--8<-- "examples/to_bson.output"
```

Deserialization

The library maps BSON record types to JSON value types as follows:

BSON type	BSON marker byte	JSON value type
double	0x01	number_float
string	0x02	string
document	0x03	object
array	0x04	array
binary	0x05	binary
undefined	0x06	unsupported
ObjectId	0x07	unsupported
boolean	0x08	boolean
UTC Date-Time	0x09	unsupported
null	0x0A	null
Regular Expr.	0x0B	unsupported
DB Pointer	0x0C	unsupported
JavaScript Code	0x0D	unsupported
Symbol	0x0E	unsupported
JavaScript Code w/ scope	0x0F	unsupported
int32	0x10	number_integer
uint64(Timestamp)	0x11	number_unsigned
int64	0x12	number_integer
128-bit decimal float	0x13	unsupported
Max Key	0x7F	unsupported
Min Key	0xFF	unsupported

!!! warning "Incomplete mapping"

The mapping is **incomplete**. The unsupported mappings are indicated in the table above.

!!! note "Handling of BSON type 0x11"

This library deserializes BSON type `0x11` (Timestamp) as a `number_unsigned` value. The 64-bit value is preserved,
but the Timestamp type information is not.

!!! warning "Lenient BSON input handling"

The BSON reader is lenient in a few areas where the BSON specification is more restrictive:

- array element keys are not checked against the required decimal sequence (`0`, `1`, `2`, ...),
- any non-zero byte is accepted as `true` for the boolean type, and
- the payload for binary subtype `0x02` is returned as-is, including its inner length prefix.

If BSON input must be validated for strict specification compliance, validate it separately before passing it to
`from_bson()`.

!!! warning "UTF-8 validation of string values"

The BSON specification requires `string` values (type `0x02`) to be valid UTF-8. This library validates the
bytes of every such string at decode time and rejects ill-formed UTF-8 with a
[`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) exception (or, with `allow_exceptions`
set to `false`, a discarded value), rather than only failing later when the resulting value is dumped. Element
(key) names and `binary` values (type `0x05`) are unaffected and are never validated, since they are read
byte-by-byte as a C string, or are not required to hold text, respectively.

??? example

```cpp
--8<-- "examples/from_bson.cpp"
```

Output:

```json
--8<-- "examples/from_bson.output"
```

6.2 KiB Raw Blame History

BSON

Serialization

Deserialization

6.2 KiB

Raw Blame History