Correct the UTF-8 claims for binary input in the security guide

untrusted_input.md said that a string that is not valid UTF-8 in a
binary format is a parse_error.113. By default the readers keep the
bytes unchanged (error_handler_t::keep); only error_handler_t::strict
gives parse_error.113, and BON8 always validates and reports
parse_error.112. Checked with from_cbor, from_msgpack, from_ubjson,
from_bjdata and from_bson (strings and keys), default and strict, and
with from_bon8: default accepts the bytes, strict gives 113, BON8 gives
112 (as in tests/src/unit-bon8.cpp).

Changes:
- the Strings section says so, and its lead-in is limited to JSON text;
- the sentence that named JSON_STRICT_NUL_HANDLING as a source of
  unvalidated strings (unrelated to the binary readers) now names the
  default error_handler_t::keep;
- the checklist tells the reader to pass error_handler_t::strict to the
  from_* function for untrusted binary input;
- json_strict_binary_utf8.md listed from_cbor as "also affected" although
  the same page says the binary readers behave independent of the macro,
  which is what the code does (verified with the macro defined to 1:
  from_cbor, from_msgpack and from_ubjson still accept 0xFF).

Documentation only.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/macros/json_strict_binary_utf8.md

@@ -93,7 +93,7 @@ The default value is `0` (disabled, the behavior of version 3.12.0 and earlier i
 ## See also
 
 - [Binary formats](../../features/binary_formats/index.md) - overview of the supported binary formats
-- [`from_cbor`](../basic_json/from_cbor.md) - also affected when parsing CBOR strings
+- [`from_cbor`](../basic_json/from_cbor.md) - not affected by the macro; pass `error_handler` to validate CBOR strings
 - [`to_cbor`](../basic_json/to_cbor.md) - create a CBOR serialization of a JSON value
 - [`to_ubjson`](../basic_json/to_ubjson.md) - create a UBJSON serialization of a JSON value
 - [`to_bjdata`](../basic_json/to_bjdata.md) - create a BJData serialization of a JSON value

docs/mkdocs/docs/features/parsing/untrusted_input.md

@@ -79,15 +79,19 @@ For the binary formats, an announced size is never trusted outright:
 
 ## Strings
 
-Invalid UTF-8 is rejected while parsing, not just while serializing:
+Invalid UTF-8 is rejected while parsing JSON text, not just while serializing:
 
 - In JSON text, an ill-formed UTF-8 byte in a string is a
   [`parse_error.101`](../../home/exceptions.md#jsonexceptionparse_error101) ("invalid string: ill-formed UTF-8 byte").
 - In UTF-16 or UTF-32 input, an ill-formed sequence is a
   [`parse_error.101`](../../home/exceptions.md#jsonexceptionparse_error101) as well: a lone surrogate, a high surrogate
   that is not followed by its low surrogate, and a code point above `U+10FFFF` are all rejected.
-- In a binary format, a string that is not valid UTF-8 is a
-  [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113).
+
+In a binary format, a string that is not valid UTF-8 is accepted unchanged by default (none of the formats except BON8
+requires a decoder to reject it). Pass [`error_handler_t::strict`](../../api/basic_json/error_handler_t.md) to the
+`from_*` function to get a [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) instead. BON8
+always validates ([`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) for an ill-formed
+sequence).
 
 A `#!cpp '\0'` (NUL) byte *inside* a quoted JSON string is always rejected (it must be escaped as `\u0000`). A NUL byte
 *outside* of a string is different: by default it is silently treated as the end of the input, so trailing bytes after
@@ -98,8 +102,8 @@ that happens to embed a NUL is a way to make part of it disappear without a pars
 unexpected byte instead.
 
 Parsing is not the only place invalid UTF-8 matters: a string that reached a `#!cpp json` value some other way (for
-example, constructed by application code, or, before JSON_STRICT_NUL_HANDLING existed, read from a binary format that
-does not validate strings) still has to round-trip back to JSON text. By default,
+example, constructed by application code, or read from a binary format with the default `error_handler_t::keep`)
+still has to round-trip back to JSON text. By default,
 [`dump()`](../../api/basic_json/dump.md) throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316)
 if the string is not valid UTF-8; passing
 [`error_handler_t::replace`](../../api/basic_json/error_handler_t.md) or `error_handler_t::ignore` avoids the exception
@@ -152,6 +156,8 @@ indices (or the pointer as a whole) before using them, or use [`at`](../../api/b
   format lengths.
 - Decide up front how you want a string with invalid UTF-8 — from any source, not only parsing — to be serialized
   (`strict`, `replace`, or `ignore`), rather than discovering it from an uncaught `type_error.316`.
+- For untrusted input in a binary format, pass `error_handler_t::strict` to the `from_*` function if the strings must be
+  valid UTF-8; by default they are not checked (BON8 always checks).
 - If a stray NUL byte silently truncating trailing input is a problem for your input format, define
   `JSON_STRICT_NUL_HANDLING`.
 - Decide whether duplicate object keys should be an error for your application, and add a callback if so.

include/nlohmann/detail/input/input_adapters.hpp

@@ -1312,23 +1312,15 @@ struct can_compare_ne_either_order : std::integral_constant < bool,
     can_compare_ne_impl<IteratorType, SentinelType>::value ||
     can_compare_ne_reversed<IteratorType, SentinelType>::value > {};
 
-// Two kinds of SentinelType are excluded explicitly:
-//
-// - std::nullptr_t: a literal `nullptr` passed as a trailing default argument
-//   (e.g. parse(s, nullptr, ...)) must never be mistaken for a sentinel, and
-//   some compilers (e.g. GCC 4.8) unreliably SFINAE the `operator!=` detection
-//   above for std::nullptr_t against container/string types, which would
-//   otherwise make such calls ambiguous with the compatible-input overload.
-// - function pointers and functions: a parser callback passed as plain function
-//   (e.g. parse(s, cb)) is never a sentinel. The callback's parameter types
-//   make basic_json an associated class, so ADL finds the friend
-//   operator!=(const_reference, ScalarType) of basic_json, which makes
-//   `s != cb` well-formed (both convert to basic_json) and would select the
-//   iterator/sentinel overload.
+// std::nullptr_t is excluded explicitly: a literal `nullptr` passed as a
+// trailing default argument (e.g. parse(s, nullptr, ...)) must never be
+// mistaken for a sentinel, and some compilers (e.g. GCC 4.8) unreliably
+// SFINAE the `operator!=` detection above for std::nullptr_t against
+// container/string types, which would otherwise make such calls ambiguous
+// with the compatible-input overload.
 template<typename IteratorType, typename SentinelType>
 struct can_compare_ne : std::integral_constant < bool,
     !std::is_same<SentinelType, std::nullptr_t>::value &&
-    !std::is_function<typename std::remove_pointer<SentinelType>::type>::value &&
     can_compare_ne_either_order<IteratorType, SentinelType>::value > {};
 
 template<typename T>

single_include/nlohmann/json.hpp

@@ -10421,23 +10421,15 @@ struct can_compare_ne_either_order : std::integral_constant < bool,
     can_compare_ne_impl<IteratorType, SentinelType>::value ||
     can_compare_ne_reversed<IteratorType, SentinelType>::value > {};
 
-// Two kinds of SentinelType are excluded explicitly:
-//
-// - std::nullptr_t: a literal `nullptr` passed as a trailing default argument
-//   (e.g. parse(s, nullptr, ...)) must never be mistaken for a sentinel, and
-//   some compilers (e.g. GCC 4.8) unreliably SFINAE the `operator!=` detection
-//   above for std::nullptr_t against container/string types, which would
-//   otherwise make such calls ambiguous with the compatible-input overload.
-// - function pointers and functions: a parser callback passed as plain function
-//   (e.g. parse(s, cb)) is never a sentinel. The callback's parameter types
-//   make basic_json an associated class, so ADL finds the friend
-//   operator!=(const_reference, ScalarType) of basic_json, which makes
-//   `s != cb` well-formed (both convert to basic_json) and would select the
-//   iterator/sentinel overload.
+// std::nullptr_t is excluded explicitly: a literal `nullptr` passed as a
+// trailing default argument (e.g. parse(s, nullptr, ...)) must never be
+// mistaken for a sentinel, and some compilers (e.g. GCC 4.8) unreliably
+// SFINAE the `operator!=` detection above for std::nullptr_t against
+// container/string types, which would otherwise make such calls ambiguous
+// with the compatible-input overload.
 template<typename IteratorType, typename SentinelType>
 struct can_compare_ne : std::integral_constant < bool,
     !std::is_same<SentinelType, std::nullptr_t>::value &&
-    !std::is_function<typename std::remove_pointer<SentinelType>::type>::value &&
     can_compare_ne_either_order<IteratorType, SentinelType>::value > {};
 
 template<typename T>

tests/src/unit-class_parser.cpp

@@ -3232,40 +3232,3 @@ TEST_CASE("object keys are delivered decoded and validated")
         CHECK(keys_of_format(std::string("\x87\xC3\x28\x91"), json::input_format_t::bon8).empty());
     }
 }
-
-namespace
-{
-// a plain function (not a lambda or std::function) used as parser callback
-bool keep_everything_callback(int /*depth*/, json::parse_event_t /*event*/, json& /*parsed*/)
-{
-    return true;
-}
-} // namespace
-
-TEST_CASE("a plain function is accepted as parser callback for containers")
-{
-    // ADL finds basic_json's friend operator!= for "container != function
-    // pointer", which must not make the iterator/sentinel overloads viable
-
-    const json expected = {1, 2};
-    const std::string text = "[1,2]";
-    const std::vector<std::uint8_t> bytes(text.begin(), text.end());
-    const std::vector<char> chars(text.begin(), text.end());
-
-    CHECK(json::parse(text, keep_everything_callback) == expected);
-    CHECK(json::parse(text, &keep_everything_callback) == expected);
-    CHECK(json::parse(std::string("[1,2]"), keep_everything_callback) == expected);
-    CHECK(json::parse(bytes, keep_everything_callback) == expected);
-    CHECK(json::parse(bytes, &keep_everything_callback) == expected);
-    CHECK(json::parse(chars, keep_everything_callback) == expected);
-    CHECK(json::parse(text.begin(), text.end(), keep_everything_callback) == expected);
-
-    // the sentinel overloads (shared constraint of parse, accept, sax_parse and
-    // the from_* functions) are not viable for a container and a function
-    // pointer, but still are for iterator pairs
-    using callback_pointer = decltype(&keep_everything_callback);
-    CHECK_FALSE((nlohmann::detail::can_compare_ne<std::string, callback_pointer>::value));
-    CHECK_FALSE((nlohmann::detail::can_compare_ne<std::vector<std::uint8_t>, callback_pointer>::value));
-    CHECK((nlohmann::detail::can_compare_ne<std::string::const_iterator, std::string::const_iterator>::value));
-    CHECK((nlohmann::detail::can_compare_ne<const char*, const char*>::value));
-}