Compare commits

...
Author SHA1 Message Date
Niels Lohmann c9d30bf924 Correct the UTF-8 claims for binary input in the security guide
untrusted_input.md said that a string that is not valid UTF-8 in a
binary format is a parse_error.113. By default the readers keep the
bytes unchanged (error_handler_t::keep); only error_handler_t::strict
gives parse_error.113, and BON8 always validates and reports
parse_error.112. Checked with from_cbor, from_msgpack, from_ubjson,
from_bjdata and from_bson (strings and keys), default and strict, and
with from_bon8: default accepts the bytes, strict gives 113, BON8 gives
112 (as in tests/src/unit-bon8.cpp).

Changes:
- the Strings section says so, and its lead-in is limited to JSON text;
- the sentence that named JSON_STRICT_NUL_HANDLING as a source of
  unvalidated strings (unrelated to the binary readers) now names the
  default error_handler_t::keep;
- the checklist tells the reader to pass error_handler_t::strict to the
  from_* function for untrusted binary input;
- json_strict_binary_utf8.md listed from_cbor as "also affected" although
  the same page says the binary readers behave independent of the macro,
  which is what the code does (verified with the macro defined to 1:
  from_cbor, from_msgpack and from_ubjson still accept 0xFF).

Documentation only.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-11 09:18:56 +01:00
2 changed files with 12 additions and 6 deletions

No files matched your search

@@ -93,7 +93,7 @@ The default value is `0` (disabled, the behavior of version 3.12.0 and earlier i
## See also
- [Binary formats](../../features/binary_formats/index.md) - overview of the supported binary formats
- [`from_cbor`](../basic_json/from_cbor.md) - also affected when parsing CBOR strings
- [`from_cbor`](../basic_json/from_cbor.md) - not affected by the macro; pass `error_handler` to validate CBOR strings
- [`to_cbor`](../basic_json/to_cbor.md) - create a CBOR serialization of a JSON value
- [`to_ubjson`](../basic_json/to_ubjson.md) - create a UBJSON serialization of a JSON value
- [`to_bjdata`](../basic_json/to_bjdata.md) - create a BJData serialization of a JSON value
@@ -79,15 +79,19 @@ For the binary formats, an announced size is never trusted outright:
## Strings
Invalid UTF-8 is rejected while parsing, not just while serializing:
Invalid UTF-8 is rejected while parsing JSON text, not just while serializing:
- In JSON text, an ill-formed UTF-8 byte in a string is a
[`parse_error.101`](../../home/exceptions.md#jsonexceptionparse_error101) ("invalid string: ill-formed UTF-8 byte").
- In UTF-16 or UTF-32 input, an ill-formed sequence is a
[`parse_error.101`](../../home/exceptions.md#jsonexceptionparse_error101) as well: a lone surrogate, a high surrogate
that is not followed by its low surrogate, and a code point above `U+10FFFF` are all rejected.
- In a binary format, a string that is not valid UTF-8 is a
[`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113).
In a binary format, a string that is not valid UTF-8 is accepted unchanged by default (none of the formats except BON8
requires a decoder to reject it). Pass [`error_handler_t::strict`](../../api/basic_json/error_handler_t.md) to the
`from_*` function to get a [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) instead. BON8
always validates ([`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) for an ill-formed
sequence).
A `#!cpp '\0'` (NUL) byte *inside* a quoted JSON string is always rejected (it must be escaped as `\u0000`). A NUL byte
*outside* of a string is different: by default it is silently treated as the end of the input, so trailing bytes after
@@ -98,8 +102,8 @@ that happens to embed a NUL is a way to make part of it disappear without a pars
unexpected byte instead.
Parsing is not the only place invalid UTF-8 matters: a string that reached a `#!cpp json` value some other way (for
example, constructed by application code, or, before JSON_STRICT_NUL_HANDLING existed, read from a binary format that
does not validate strings) still has to round-trip back to JSON text. By default,
example, constructed by application code, or read from a binary format with the default `error_handler_t::keep`)
still has to round-trip back to JSON text. By default,
[`dump()`](../../api/basic_json/dump.md) throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316)
if the string is not valid UTF-8; passing
[`error_handler_t::replace`](../../api/basic_json/error_handler_t.md) or `error_handler_t::ignore` avoids the exception
@@ -152,6 +156,8 @@ indices (or the pointer as a whole) before using them, or use [`at`](../../api/b
format lengths.
- Decide up front how you want a string with invalid UTF-8 — from any source, not only parsing — to be serialized
(`strict`, `replace`, or `ignore`), rather than discovering it from an uncaught `type_error.316`.
- For untrusted input in a binary format, pass `error_handler_t::strict` to the `from_*` function if the strings must be
valid UTF-8; by default they are not checked (BON8 always checks).
- If a stray NUL byte silently truncating trailing input is a problem for your input format, define
`JSON_STRICT_NUL_HANDLING`.
- Decide whether duplicate object keys should be an error for your application, and add a callback if so.