Merge branch 'develop' into claude/issue-5387-duplicate-check-bd7853

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/accept.md

@@ -90,6 +90,10 @@ Linear in the length of the input. The parser is a predictive LL(1) parser.
 
 A UTF-8 byte order mark is silently ignored.
 
+By default, a `'\0'` (NUL) byte anywhere in the input is treated as end of input, rather than as an ordinary (and,
+outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-bytes-in-the-input) for details and the
+[`JSON_STRICT_NUL_HANDLING`](../macros/json_strict_nul_handling.md) macro to opt into rejecting it instead.
+
 ## Examples
 
 ??? example
@@ -111,6 +115,8 @@ A UTF-8 byte order mark is silently ignored.
 - [parse](parse.md) - deserialize from a compatible input
 - [sax_parse](sax_parse.md) - parse input using the SAX interface
 - [operator>>](../operator_gtgt.md) - deserialize from stream
+- [`JSON_STRICT_NUL_HANDLING`](../macros/json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input
+  instead of treating it as end of input
 
 ## Version history
 
@@ -120,6 +126,8 @@ A UTF-8 byte order mark is silently ignored.
 - Added `ignore_trailing_commas` in version 3.13.0.
 - Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
 - Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
+- `JSON_STRICT_NUL_HANDLING` added in version 3.13.0 to optionally reject a NUL byte in the input instead of treating
+  it as end of input; planned to become the default in version 4.0.0.
 
 !!! warning "Deprecation"
 

docs/mkdocs/docs/api/basic_json/parse.md

@@ -103,6 +103,10 @@ A UTF-8 byte order mark is silently ignored.
 Invalid Unicode escapes and unpaired surrogates in the input are reported as
 [`parse_error.101`](../../home/exceptions.md#jsonexceptionparse_error101) with a detailed message.
 
+By default, a `'\0'` (NUL) byte anywhere in the input is treated as end of input, rather than as an ordinary (and,
+outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-bytes-in-the-input) for details and the
+[`JSON_STRICT_NUL_HANDLING`](../macros/json_strict_nul_handling.md) macro to opt into rejecting it instead.
+
 ## Examples
 
 ??? example "Parsing from a character array"
@@ -236,6 +240,8 @@ Invalid Unicode escapes and unpaired surrogates in the input are reported as
 - [accept](accept.md) - check if the input is valid JSON
 - [sax_parse](sax_parse.md) - parse input using the SAX interface
 - [operator>>](../operator_gtgt.md) - deserialize from stream
+- [`JSON_STRICT_NUL_HANDLING`](../macros/json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input
+  instead of treating it as end of input
 
 ## Version history
 
@@ -246,6 +252,8 @@ Invalid Unicode escapes and unpaired surrogates in the input are reported as
 - Added `ignore_trailing_commas` in version 3.13.0.
 - Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
 - Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
+- `JSON_STRICT_NUL_HANDLING` added in version 3.13.0 to optionally reject a NUL byte in the input instead of treating
+  it as end of input; planned to become the default in version 4.0.0.
 
 !!! warning "Deprecation"
 

docs/mkdocs/docs/api/macros/index.md

@@ -14,6 +14,11 @@ header. See also the [macro overview page](../../features/macros.md).
 - [**JSON_DIAGNOSTIC_POSITIONS**](json_diagnostic_positions.md) - access positions of elements
 - [**JSON_NOEXCEPTION**](json_noexception.md) - switch off exceptions
 
+## Parsing
+
+- [**JSON_STRICT_NUL_HANDLING**](json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input instead of
+  treating it as end of input
+
 ## Language support
 
 - [**JSON_HAS_CPP_11**<br>**JSON_HAS_CPP_14**<br>**JSON_HAS_CPP_17**<br>**JSON_HAS_CPP_20**](json_has_cpp_11.md) - set supported C++ standard

docs/mkdocs/docs/api/macros/json_strict_nul_handling.md

@@ -0,0 +1,126 @@
+# JSON_STRICT_NUL_HANDLING
+
+```cpp
+#define JSON_STRICT_NUL_HANDLING /* value */
+```
+
+When defined to `1`, a `'\0'` (NUL) byte in JSON text input is rejected with `parse_error.101`, like any other
+unexpected byte, instead of being silently treated as end of input.
+
+The macro only affects the JSON text parser ([`parse`](../basic_json/parse.md), [`accept`](../basic_json/accept.md),
+[`sax_parse`](../basic_json/sax_parse.md), and [`operator>>`](../operator_gtgt.md)). There are three cases where a NUL
+byte is still not rejected:
+
+- The binary formats ([`from_bjdata`](../basic_json/from_bjdata.md), [`from_bson`](../basic_json/from_bson.md),
+  [`from_cbor`](../basic_json/from_cbor.md), [`from_msgpack`](../basic_json/from_msgpack.md),
+  [`from_ubjson`](../basic_json/from_ubjson.md)) are never affected: there, `0x00` is ordinary data.
+- A bare `const char*` pointer has no length of its own, so its length is still determined with `strlen()`. The first
+  NUL byte therefore still marks the end of the input, and nothing after it is read.
+- One trailing `'\0'` at the end of a `char` array (e.g., a string literal) is trimmed; see the warning below.
+
+## Default definition
+
+The default value is `0` (disabled — existing behavior is preserved).
+
+```cpp
+#define JSON_STRICT_NUL_HANDLING 0
+```
+
+## Notes
+
+!!! note "Background"
+
+    By default, a `'\0'` byte anywhere in the input is treated the same as the real end of the input, rather than as
+    an ordinary (and, outside of a string, invalid) byte. Everything from that byte onward is silently ignored,
+    without a parse error - including further, otherwise well-formed JSON:
+
+    ```cpp
+    json::parse(std::string("123") + '\0');          // == 123, no error
+    json::parse(std::string("123") + '\0' + "true"); // == 123, the "true" is silently ignored too
+    ```
+
+    This falls out of the same convention used when no explicit input length is given at all: parsing from a
+    `const char*` already stops at the first NUL byte via `strlen()`, since a bare pointer has no length of its own.
+    The library applies that same NUL-terminated-C-string convention uniformly, rather than only when a length is
+    genuinely unavailable - so a `std::string`, iterator range, or container whose content happens to include a NUL
+    byte is affected the same way a raw `const char*` would be (see the
+    [FAQ entry](../../home/faq.md#nul-bytes-in-the-input) for a fuller explanation).
+
+    This was not fixed unconditionally, because doing so is backwards-incompatible for any caller who happens to
+    depend on the current behavior - even unknowingly, for instance because their input already contains trailing
+    padding they never noticed was being discarded (see [#5530](https://github.com/nlohmann/json/issues/5530)).
+    This macro instead offers an opt-in path to the corrected behavior ahead of version 4.0.0, where it is planned to
+    become the default.
+
+!!! warning "Opt-in only"
+
+    This macro must be defined **before** including `<nlohmann/json.hpp>`. Defining it after the include has no
+    effect.
+
+    Enabling it also changes how a `char` array (including a string literal, e.g. `json::parse("123")`) is read: such
+    an array normally carries a trailing `'\0'` contributed by the compiler, not by the source text. With this macro
+    enabled, that one trailing byte is trimmed if present so that parsing a string literal keeps working; every other
+    byte in the array - including any `'\0'` that is not the very last element - is read as real data and rejected
+    like any other unexpected byte. Arrays of any other element type (`unsigned char`, `std::uint8_t`, ...), as used
+    for CBOR or MessagePack, are never affected by this trimming; their full extent - including a genuine trailing
+    `0x00` - is always preserved, in both states of this macro.
+
+!!! tip "Workaround without the macro"
+
+    To reject a NUL byte without enabling this macro, trim your input yourself before calling `parse()`:
+
+    ```cpp
+    s.resize(s.find('\0')); // drop everything from the first NUL onward, if any
+    json::parse(s);
+    ```
+
+## Examples
+
+??? example "Default behavior (macro not defined)"
+
+    Without the macro, a NUL byte silently ends parsing at that point:
+
+    ```cpp
+    #include <nlohmann/json.hpp>
+
+    using json = nlohmann::json;
+
+    int main()
+    {
+        json j = json::parse(std::string("123") + '\0' + "true");
+        // j is 123 -- the '\0' and everything after it is silently ignored
+    }
+    ```
+
+??? example "Opt-in strict handling (macro defined to 1)"
+
+    With the macro, a NUL byte is rejected like any other unexpected byte:
+
+    ```cpp
+    #define JSON_STRICT_NUL_HANDLING 1
+    #include <nlohmann/json.hpp>
+
+    using json = nlohmann::json;
+
+    int main()
+    {
+        json j = json::parse(std::string("123") + '\0' + "true");
+        // throws parse_error.101 -- the NUL byte is now invalid input,
+        // exactly like any other unexpected trailing byte
+
+        json ok = json::parse("123");
+        // ok is 123 -- parsing from a string literal still works
+    }
+    ```
+
+## See also
+
+- [FAQ: NUL bytes in the input](../../home/faq.md#nul-bytes-in-the-input)
+- [**parse**](../basic_json/parse.md) - deserialize from a compatible input
+- [**accept**](../basic_json/accept.md) - check if the input is valid JSON
+- [**operator>>**](../operator_gtgt.md) - deserialize from stream
+
+## Version history
+
+- Added in version 3.13.0.
+- Planned to become the default (with the macro removed) in version 4.0.0.

docs/mkdocs/docs/api/operator_gtgt.md

@@ -72,6 +72,13 @@ input >> j2;  // parses the next value
 Note that reading concatenated values does **not** work for [JSON Lines](../features/parsing/json_lines.md)
 (newline-delimited JSON) input -- see that page for why and for the recommended alternative.
 
+By default, a `'\0'` (NUL) byte encountered while reading a value is treated as end of input, rather than as an
+ordinary (and, outside of a string, invalid) byte; see the [FAQ entry](../home/faq.md#nul-bytes-in-the-input) for
+details and the [`JSON_STRICT_NUL_HANDLING`](macros/json_strict_nul_handling.md) macro to opt into rejecting it
+instead. Because `operator>>` only parses a single value and does not require the rest of the stream to be consumed,
+a NUL byte *after* a complete value has no effect on `operator>>` either way; it only matters while a value is still
+being read.
+
 !!! warning "Deprecation"
 
     This function replaces function `#!cpp std::istream& operator<<(basic_json& j, std::istream& i)` which has
@@ -98,7 +105,11 @@ Note that reading concatenated values does **not** work for [JSON Lines](../feat
 
 - [accept](basic_json/accept.md) - check if the input is valid JSON
 - [parse](basic_json/parse.md) - deserialize from a compatible input
+- [`JSON_STRICT_NUL_HANDLING`](macros/json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input
+  instead of treating it as end of input
 
 ## Version history
 
 - Added in version 1.0.0.
+- `JSON_STRICT_NUL_HANDLING` added in version 3.13.0 to optionally reject a NUL byte in the input instead of treating
+  it as end of input; planned to become the default in version 4.0.0.

docs/mkdocs/docs/features/macros.md

@@ -112,6 +112,19 @@ using the library with compilers that do not fully support C++11 and may only wo
 
 See [full documentation of `JSON_SKIP_UNSUPPORTED_COMPILER_CHECK`](../api/macros/json_skip_unsupported_compiler_check.md).
 
+## `JSON_STRICT_NUL_HANDLING`
+
+When defined to `1`, a `'\0'` (NUL) byte anywhere in the input is rejected with `parse_error.101`, like any other
+unexpected byte, instead of being silently treated as end of input (see the
+[FAQ entry](../home/faq.md#nul-bytes-in-the-input) for background). The default value is `0`, which preserves the
+existing behavior; this is planned to become the default in version 4.0.0.
+
+The strict handling can also be enabled with the CMake option
+[`JSON_StrictNulHandling`](../integration/cmake.md#json_strictnulhandling) (`OFF` by default) which sets
+`JSON_STRICT_NUL_HANDLING` accordingly.
+
+See [full documentation of `JSON_STRICT_NUL_HANDLING`](../api/macros/json_strict_nul_handling.md).
+
 ## `JSON_THROW_USER(exception)`
 
 This macro overrides `#!cpp throw` calls inside the library. The argument is the exception to be thrown.

docs/mkdocs/docs/home/faq.md

@@ -90,6 +90,54 @@ The library supports **Unicode input** as follows:
 In most cases, the parser is right to complain, because the input is not UTF-8 encoded. This is especially true for Microsoft Windows, where Latin-1 or ISO 8859-1 is often the standard encoding.
 
 
+### NUL bytes in the input
+
+!!! question "Questions"
+
+    - Why does `json::parse()` silently ignore part of my input?
+    - Why does a `std::string`/buffer with extra data after the JSON text parse without error, while a similar-looking string with extra text does not?
+
+A `'\0'` (NUL) byte anywhere in the input is treated the same as the real end of the input, rather than as an ordinary (and, outside of a string, invalid) byte. Everything from that byte onward is silently ignored, without a parse error — including further, otherwise well-formed JSON:
+
+```cpp
+json::parse(std::string("123") + '\0');          // == 123, no error
+json::parse(std::string("123") + '\0' + "true"); // == 123, the "true" is silently ignored too
+```
+
+This is different from any other unexpected trailing byte, which *does* raise [`parse_error.101`](../home/exceptions.md#jsonexceptionparse_error101):
+
+```cpp
+json::parse("123x"); // throws parse_error.101: unexpected additional data
+```
+
+This falls out of the same convention used when no explicit input length is given at all: `json::parse(const char*)` already stops at the first NUL byte via `strlen()`, since a bare pointer has no length of its own. The library applies that same NUL-terminated-C-string convention uniformly, rather than only when a length is genuinely unavailable — so a `std::string`, iterator range, or container whose content happens to include a NUL byte is affected the same way a raw `const char*` would be.
+
+If your input may contain a trailing or embedded NUL that is **not** meant to signal the end of the JSON text — for instance, a fixed-size, zero-padded buffer — trim it yourself before calling `parse()`, since the library will otherwise silently stop there instead of raising an error:
+
+```cpp
+s.resize(s.find('\0')); // drop everything from the first NUL onward, if any
+json::parse(s);
+```
+
+**Opt-in strict handling (since version 3.13.0)**
+
+Manually trimming every input is easy to forget. If you define [`JSON_STRICT_NUL_HANDLING`](../api/macros/json_strict_nul_handling.md) to `1` before including the library, a `'\0'` byte is instead rejected like any other unexpected byte and raises `parse_error.101`, instead of being treated as end of input:
+
+```cpp
+#define JSON_STRICT_NUL_HANDLING 1
+#include <nlohmann/json.hpp>
+
+json::parse(std::string("123") + '\0'); // throws parse_error.101 instead of silently returning 123
+```
+
+This macro defaults to `0` (disabled, preserving the behavior described above) to avoid breaking existing code that may depend on it, even unknowingly; it is planned to become the default in version 4.0.0. See [its documentation](../api/macros/json_strict_nul_handling.md) for details, including how it also affects `char` arrays such as string literals.
+
+Note that this is unrelated to an *unescaped* NUL byte occurring **inside** a quoted JSON string, which is a different, already-invalid case and is correctly rejected either way:
+
+```cpp
+json::parse(std::string("\"") + '\0' + "\""); // throws parse_error.101: control character U+0000 (NUL) must be escaped to \u0000
+```
+
 ### Wide string handling
 
 !!! question

docs/mkdocs/docs/integration/cmake.md

@@ -198,6 +198,11 @@ Use the non-amalgamated version of the library. This option is `ON` by default.
 
 Treat the library headers like system headers (i.e., adding `SYSTEM` to the [`target_include_directories`](https://cmake.org/cmake/help/latest/command/target_include_directories.html) call) to check for this library by tools like Clang-Tidy. This option is `OFF` by default.
 
+### `JSON_StrictNulHandling`
+
+Reject a `'\0'` (NUL) byte in the input instead of treating it as end of input, by defining the macro
+[`JSON_STRICT_NUL_HANDLING`](../api/macros/json_strict_nul_handling.md). This option is `OFF` by default.
+
 ### `JSON_Valgrind`
 
 Execute the test suite with [Valgrind](https://valgrind.org). This option is `OFF` by default. Depends on `JSON_BuildTests`.

docs/mkdocs/mkdocs.yml

@@ -295,6 +295,7 @@ nav:
           - 'JSON_NO_THREAD_LOCAL': api/macros/json_no_thread_local.md
           - 'JSON_SKIP_LIBRARY_VERSION_CHECK': api/macros/json_skip_library_version_check.md
           - 'JSON_SKIP_UNSUPPORTED_COMPILER_CHECK': api/macros/json_skip_unsupported_compiler_check.md
+          - 'JSON_STRICT_NUL_HANDLING': api/macros/json_strict_nul_handling.md
           - 'JSON_USE_GLOBAL_UDLS': api/macros/json_use_global_udls.md
           - 'JSON_USE_IMPLICIT_CONVERSIONS': api/macros/json_use_implicit_conversions.md
           - 'JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON': api/macros/json_use_legacy_discarded_value_comparison.md

docs/mkdocs/requirements.txt

@@ -1,7 +1,7 @@
 wheel==0.48.0
 
 mkdocs==1.6.1                                       # documentation framework
-mkdocs-git-revision-date-localized-plugin==1.5.4    # plugin "git-revision-date-localized"
+mkdocs-git-revision-date-localized-plugin==1.6.0    # plugin "git-revision-date-localized"
 mkdocs-material==9.7.7                             # theme for mkdocs
 mkdocs-material-extensions==1.3.1                   # extensions
 mkdocs-minify-plugin==0.8.0                         # plugin "minify"