Fix parse(container, function) for plain functions as callback

Passing a plain function or function pointer as parser callback, as in json::parse(std::string("[1,2]"), cb), stopped compiling in C++11/14/17. The iterator+sentinel overloads added for C++20 ranges are constrained by can_compare_ne, i.e. whether `first != last` is well-formed. The callback's parameter types make basic_json an associated class, so ADL finds the friend operator!=(const_reference, ScalarType): the container converts to basic_json and the function pointer is a scalar. The sentinel overload was thereby selected and failed hard in its body.

can_compare_ne now excludes function pointers and functions as sentinel type, since a callback is never a sentinel. This shared constraint guards parse, accept, sax_parse and all from_* overloads. Checking IteratorType via std::iterator_traits was avoided on purpose: it would instantiate iterator_traits for every two-argument call, which is a hard error on standard libraries with a non-SFINAE-friendly iterator_traits.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/macros/json_strict_binary_utf8.md

@@ -93,7 +93,7 @@ The default value is `0` (disabled, the behavior of version 3.12.0 and earlier i
 ## See also
 
 - [Binary formats](../../features/binary_formats/index.md) - overview of the supported binary formats
-- [`from_cbor`](../basic_json/from_cbor.md) - not affected by the macro; pass `error_handler` to validate CBOR strings
+- [`from_cbor`](../basic_json/from_cbor.md) - also affected when parsing CBOR strings
 - [`to_cbor`](../basic_json/to_cbor.md) - create a CBOR serialization of a JSON value
 - [`to_ubjson`](../basic_json/to_ubjson.md) - create a UBJSON serialization of a JSON value
 - [`to_bjdata`](../basic_json/to_bjdata.md) - create a BJData serialization of a JSON value

docs/mkdocs/docs/features/parsing/untrusted_input.md

@@ -79,19 +79,15 @@ For the binary formats, an announced size is never trusted outright:
 
 ## Strings
 
-Invalid UTF-8 is rejected while parsing JSON text, not just while serializing:
+Invalid UTF-8 is rejected while parsing, not just while serializing:
 
 - In JSON text, an ill-formed UTF-8 byte in a string is a
   [`parse_error.101`](../../home/exceptions.md#jsonexceptionparse_error101) ("invalid string: ill-formed UTF-8 byte").
 - In UTF-16 or UTF-32 input, an ill-formed sequence is a
   [`parse_error.101`](../../home/exceptions.md#jsonexceptionparse_error101) as well: a lone surrogate, a high surrogate
   that is not followed by its low surrogate, and a code point above `U+10FFFF` are all rejected.
-
-In a binary format, a string that is not valid UTF-8 is accepted unchanged by default (none of the formats except BON8
-requires a decoder to reject it). Pass [`error_handler_t::strict`](../../api/basic_json/error_handler_t.md) to the
-`from_*` function to get a [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) instead. BON8
-always validates ([`parse_error.112`](../../home/exceptions.md#jsonexceptionparse_error112) for an ill-formed
-sequence).
+- In a binary format, a string that is not valid UTF-8 is a
+  [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113).
 
 A `#!cpp '\0'` (NUL) byte *inside* a quoted JSON string is always rejected (it must be escaped as `\u0000`). A NUL byte
 *outside* of a string is different: by default it is silently treated as the end of the input, so trailing bytes after
@@ -102,8 +98,8 @@ that happens to embed a NUL is a way to make part of it disappear without a pars
 unexpected byte instead.
 
 Parsing is not the only place invalid UTF-8 matters: a string that reached a `#!cpp json` value some other way (for
-example, constructed by application code, or read from a binary format with the default `error_handler_t::keep`)
-still has to round-trip back to JSON text. By default,
+example, constructed by application code, or, before JSON_STRICT_NUL_HANDLING existed, read from a binary format that
+does not validate strings) still has to round-trip back to JSON text. By default,
 [`dump()`](../../api/basic_json/dump.md) throws [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316)
 if the string is not valid UTF-8; passing
 [`error_handler_t::replace`](../../api/basic_json/error_handler_t.md) or `error_handler_t::ignore` avoids the exception
@@ -156,8 +152,6 @@ indices (or the pointer as a whole) before using them, or use [`at`](../../api/b
   format lengths.
 - Decide up front how you want a string with invalid UTF-8 — from any source, not only parsing — to be serialized
   (`strict`, `replace`, or `ignore`), rather than discovering it from an uncaught `type_error.316`.
-- For untrusted input in a binary format, pass `error_handler_t::strict` to the `from_*` function if the strings must be
-  valid UTF-8; by default they are not checked (BON8 always checks).
 - If a stray NUL byte silently truncating trailing input is a problem for your input format, define
   `JSON_STRICT_NUL_HANDLING`.
 - Decide whether duplicate object keys should be an error for your application, and add a callback if so.

include/nlohmann/detail/input/input_adapters.hpp

@@ -1312,15 +1312,23 @@ struct can_compare_ne_either_order : std::integral_constant < bool,
     can_compare_ne_impl<IteratorType, SentinelType>::value ||
     can_compare_ne_reversed<IteratorType, SentinelType>::value > {};
 
-// std::nullptr_t is excluded explicitly: a literal `nullptr` passed as a
-// trailing default argument (e.g. parse(s, nullptr, ...)) must never be
-// mistaken for a sentinel, and some compilers (e.g. GCC 4.8) unreliably
-// SFINAE the `operator!=` detection above for std::nullptr_t against
-// container/string types, which would otherwise make such calls ambiguous
-// with the compatible-input overload.
+// Two kinds of SentinelType are excluded explicitly:
+//
+// - std::nullptr_t: a literal `nullptr` passed as a trailing default argument
+//   (e.g. parse(s, nullptr, ...)) must never be mistaken for a sentinel, and
+//   some compilers (e.g. GCC 4.8) unreliably SFINAE the `operator!=` detection
+//   above for std::nullptr_t against container/string types, which would
+//   otherwise make such calls ambiguous with the compatible-input overload.
+// - function pointers and functions: a parser callback passed as plain function
+//   (e.g. parse(s, cb)) is never a sentinel. The callback's parameter types
+//   make basic_json an associated class, so ADL finds the friend
+//   operator!=(const_reference, ScalarType) of basic_json, which makes
+//   `s != cb` well-formed (both convert to basic_json) and would select the
+//   iterator/sentinel overload.
 template<typename IteratorType, typename SentinelType>
 struct can_compare_ne : std::integral_constant < bool,
     !std::is_same<SentinelType, std::nullptr_t>::value &&
+    !std::is_function<typename std::remove_pointer<SentinelType>::type>::value &&
     can_compare_ne_either_order<IteratorType, SentinelType>::value > {};
 
 template<typename T>

single_include/nlohmann/json.hpp

@@ -10421,15 +10421,23 @@ struct can_compare_ne_either_order : std::integral_constant < bool,
     can_compare_ne_impl<IteratorType, SentinelType>::value ||
     can_compare_ne_reversed<IteratorType, SentinelType>::value > {};
 
-// std::nullptr_t is excluded explicitly: a literal `nullptr` passed as a
-// trailing default argument (e.g. parse(s, nullptr, ...)) must never be
-// mistaken for a sentinel, and some compilers (e.g. GCC 4.8) unreliably
-// SFINAE the `operator!=` detection above for std::nullptr_t against
-// container/string types, which would otherwise make such calls ambiguous
-// with the compatible-input overload.
+// Two kinds of SentinelType are excluded explicitly:
+//
+// - std::nullptr_t: a literal `nullptr` passed as a trailing default argument
+//   (e.g. parse(s, nullptr, ...)) must never be mistaken for a sentinel, and
+//   some compilers (e.g. GCC 4.8) unreliably SFINAE the `operator!=` detection
+//   above for std::nullptr_t against container/string types, which would
+//   otherwise make such calls ambiguous with the compatible-input overload.
+// - function pointers and functions: a parser callback passed as plain function
+//   (e.g. parse(s, cb)) is never a sentinel. The callback's parameter types
+//   make basic_json an associated class, so ADL finds the friend
+//   operator!=(const_reference, ScalarType) of basic_json, which makes
+//   `s != cb` well-formed (both convert to basic_json) and would select the
+//   iterator/sentinel overload.
 template<typename IteratorType, typename SentinelType>
 struct can_compare_ne : std::integral_constant < bool,
     !std::is_same<SentinelType, std::nullptr_t>::value &&
+    !std::is_function<typename std::remove_pointer<SentinelType>::type>::value &&
     can_compare_ne_either_order<IteratorType, SentinelType>::value > {};
 
 template<typename T>

tests/src/unit-class_parser.cpp

@@ -3232,3 +3232,40 @@ TEST_CASE("object keys are delivered decoded and validated")
         CHECK(keys_of_format(std::string("\x87\xC3\x28\x91"), json::input_format_t::bon8).empty());
     }
 }
+
+namespace
+{
+// a plain function (not a lambda or std::function) used as parser callback
+bool keep_everything_callback(int /*depth*/, json::parse_event_t /*event*/, json& /*parsed*/)
+{
+    return true;
+}
+} // namespace
+
+TEST_CASE("a plain function is accepted as parser callback for containers")
+{
+    // ADL finds basic_json's friend operator!= for "container != function
+    // pointer", which must not make the iterator/sentinel overloads viable
+
+    const json expected = {1, 2};
+    const std::string text = "[1,2]";
+    const std::vector<std::uint8_t> bytes(text.begin(), text.end());
+    const std::vector<char> chars(text.begin(), text.end());
+
+    CHECK(json::parse(text, keep_everything_callback) == expected);
+    CHECK(json::parse(text, &keep_everything_callback) == expected);
+    CHECK(json::parse(std::string("[1,2]"), keep_everything_callback) == expected);
+    CHECK(json::parse(bytes, keep_everything_callback) == expected);
+    CHECK(json::parse(bytes, &keep_everything_callback) == expected);
+    CHECK(json::parse(chars, keep_everything_callback) == expected);
+    CHECK(json::parse(text.begin(), text.end(), keep_everything_callback) == expected);
+
+    // the sentinel overloads (shared constraint of parse, accept, sax_parse and
+    // the from_* functions) are not viable for a container and a function
+    // pointer, but still are for iterator pairs
+    using callback_pointer = decltype(&keep_everything_callback);
+    CHECK_FALSE((nlohmann::detail::can_compare_ne<std::string, callback_pointer>::value));
+    CHECK_FALSE((nlohmann::detail::can_compare_ne<std::vector<std::uint8_t>, callback_pointer>::value));
+    CHECK((nlohmann::detail::can_compare_ne<std::string::const_iterator, std::string::const_iterator>::value));
+    CHECK((nlohmann::detail::can_compare_ne<const char*, const char*>::value));
+}