Put the stream position fix behind JSON_PRECISE_STREAM_POSITION

Leaving the character that terminates a number in the stream is observable:
reading "1,2,3" with repeated operator>> works today only because the comma
after each number is swallowed, and std::getline after a number skips the
line break. Both break with the fix, so make it opt-in for 3.x, as suggested
by @gregmarr in the review.

- JSON_PRECISE_STREAM_POSITION (default 0) selects the peek-based
  input_stream_adapter. Without it, the adapter is the consuming one from
  develop and has no supports_lookahead, so lexer::release_lookahead() and
  the parser's calls to it compile to nothing.
- The macro changes input_stream_adapter's layout and member functions, so
  it gets the ABI tag _psp, after _bics. The ABI config tests, the natvis
  generator, and nlohmann_json.natvis (regenerated) know the tag.
- The tests for the fix move to unit-precise-stream-position.cpp, which
  defines the macro itself and runs in every build, and gain the two cases
  above. unit-deserialization.cpp pins the default behavior instead.
- The docs describe the default behavior again and point to the new macro
  page; version history says "added in 3.13.0, planned default in 4.0.0".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/sax_parse.md

@@ -70,7 +70,8 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index
 
 `strict` (in)
 :   whether the input has to be consumed completely (optional, `#!cpp true` by default); when `#!cpp false` and the
-    input is a `#!cpp std::istream`, the stream is left positioned right after the parsed value
+    input is a `#!cpp std::istream`, the character that terminates a number is consumed unless
+    [`JSON_PRECISE_STREAM_POSITION`](../macros/json_precise_stream_position.md) is defined to `1`; see [`operator>>`](../operator_gtgt.md#notes)
 
 `ignore_comments` (in)
 :   whether comments should be ignored and treated like whitespace (`#!cpp true`) or yield a parse error
@@ -137,8 +138,8 @@ A UTF-8 byte order mark is silently ignored.
 - Added `ignore_trailing_commas` in version 3.13.0.
 - Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
 - Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
-- Changed in version 4.0.0 to leave a `#!cpp std::istream` positioned right after the parsed value when `strict` is
-  `#!cpp false`; see [`operator>>`](../operator_gtgt.md#notes).
+- `JSON_PRECISE_STREAM_POSITION` added in version 3.13.0 to optionally leave a `#!cpp std::istream` positioned right
+  after the parsed value when `strict` is `#!cpp false`.
 
 !!! warning "Deprecation"
 

docs/mkdocs/docs/api/macros/index.md

@@ -16,6 +16,8 @@ header. See also the [macro overview page](../../features/macros.md).
 
 ## Parsing
 
+- [**JSON_PRECISE_STREAM_POSITION**](json_precise_stream_position.md) - opt in to leaving an input stream positioned
+  right after a parsed number
 - [**JSON_STRICT_NUL_HANDLING**](json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input instead of
   treating it as end of input
 

docs/mkdocs/docs/api/macros/json_precise_stream_position.md

@@ -0,0 +1,131 @@
+# JSON_PRECISE_STREAM_POSITION
+
+```cpp
+#define JSON_PRECISE_STREAM_POSITION /* value */
+```
+
+When defined to `1`, [`operator>>`](../operator_gtgt.md) and [`sax_parse`](../basic_json/sax_parse.md) with
+`strict = false` leave a `#!cpp std::istream` positioned right after the parsed value for every value type. By default,
+the character that terminates a number is consumed as well.
+
+The macro only affects reading from a `#!cpp std::istream` when the rest of the stream is not required to be consumed.
+[`parse`](../basic_json/parse.md), [`accept`](../basic_json/accept.md), and all other inputs (strings, iterators,
+containers, `#!cpp FILE*`) are never affected.
+
+## Default definition
+
+The default value is `0` (disabled — existing behavior is preserved).
+
+```cpp
+#define JSON_PRECISE_STREAM_POSITION 0
+```
+
+## Notes
+
+!!! note "Background"
+
+    A number is the only JSON value whose end can be detected solely by reading the character that follows it. By
+    default, that character is consumed and not put back, so the stream is left one byte too far after a number, and
+    only after a number:
+
+    ```cpp
+    std::istringstream input("1true");
+    json j;
+    input >> j;  // j == 1, but the stream now starts at "rue"
+    ```
+
+    With this macro, the character is only looked at and left in the stream, so the stream starts at `true`. This
+    does not require the stream buffer to support putting a character back.
+
+    This was not changed unconditionally, because code can depend on the consumed character, even unknowingly (see
+    [#5340](https://github.com/nlohmann/json/issues/5340)). Both of the following work by default only because the
+    character after each number is swallowed, and behave differently with this macro:
+
+    ```cpp
+    std::istringstream input("1,2,3");
+    json j1, j2, j3;
+    input >> j1 >> j2 >> j3;  // default: 1, 2, 3
+                              // with the macro: throws parse_error.101 at the ','
+    ```
+
+    ```cpp
+    std::istringstream input("42\nfoo");
+    json j;
+    std::string line;
+    input >> j;
+    std::getline(input, line);  // default: "foo"
+                                // with the macro: "" (like after reading an int with >>)
+    ```
+
+    In both cases, the behavior with the macro is what you already get today when the value is not a number: `"a","b"`
+    fails at the `,`, and `std::getline` after `{}` returns an empty string. This macro offers an opt-in path to
+    the consistent behavior ahead of version 4.0.0, where it is planned to become the default.
+
+!!! warning "Opt-in only"
+
+    This macro must be defined **before** including `<nlohmann/json.hpp>`. Defining it after the include has no
+    effect.
+
+!!! note "ABI compatibility"
+
+    The value of this macro is encoded in the [namespace](../../features/namespace.md) (tag `_psp`), resulting in
+    distinct symbol names. Translation units compiled with and without it can therefore be linked into the same program
+    without One Definition Rule (ODR) violations, but they cannot exchange instances of library types.
+
+!!! tip "Workaround without the macro"
+
+    Separate the values in the stream with whitespace. The character consumed after a number is then the separator,
+    and whitespace before the next value is skipped anyway.
+
+## Examples
+
+??? example "Default behavior (macro not defined)"
+
+    Without the macro, the character after a number is consumed:
+
+    ```cpp
+    #include <iostream>
+    #include <sstream>
+    #include <nlohmann/json.hpp>
+
+    using json = nlohmann::json;
+
+    int main()
+    {
+        std::istringstream input("1true");
+        json j1, j2;
+        input >> j1;  // j1 == 1
+        input >> j2;  // throws parse_error.101: the stream now starts at "rue"
+    }
+    ```
+
+??? example "Opt-in precise stream position (macro defined to 1)"
+
+    With the macro, the stream is positioned right after the number:
+
+    ```cpp
+    #define JSON_PRECISE_STREAM_POSITION 1
+    #include <iostream>
+    #include <sstream>
+    #include <nlohmann/json.hpp>
+
+    using json = nlohmann::json;
+
+    int main()
+    {
+        std::istringstream input("1true");
+        json j1, j2;
+        input >> j1;  // j1 == 1
+        input >> j2;  // j2 == true
+    }
+    ```
+
+## See also
+
+- [**operator>>**](../operator_gtgt.md) - deserialize from stream
+- [**sax_parse**](../basic_json/sax_parse.md) - generate SAX events
+
+## Version history
+
+- Added in version 3.13.0.
+- Planned to become the default (with the macro removed) in version 4.0.0.

docs/mkdocs/docs/api/operator_gtgt.md

@@ -33,26 +33,43 @@ A UTF-8 byte order mark is silently ignored.
 Invalid Unicode escapes and unpaired surrogates in the input are reported as
 [`parse_error.101`](../home/exceptions.md#jsonexceptionparse_error101) with a detailed message.
 
-`operator>>` parses exactly one JSON value and leaves the stream positioned right after it, so it can be called
-repeatedly to read a sequence of concatenated JSON values from the same stream:
+`operator>>` parses exactly one JSON value, so it can be called repeatedly to read a sequence of concatenated JSON
+values from the same stream:
 
 ```cpp
-std::istringstream input("1true[2]");
-json j1, j2, j3;
-input >> j1;  // j1 == 1,      stream now positioned right after it
-input >> j2;  // j2 == true
-input >> j3;  // j3 == [2]
+json j1, j2;
+input >> j1;  // parses the first value
+input >> j2;  // parses the next value
 ```
 
-!!! note "Changed behavior for numbers"
+!!! warning "A number must be followed by whitespace"
 
-    A number is the only value whose end can be detected solely by reading the character that follows it. Up to
-    version 3.13.0 that character was consumed and not put back, so the stream was left one byte too far whenever a
-    number was immediately followed by another value: reading `1true` yielded `1` and left the stream at `rue`.
-    Values had to be separated by whitespace to work around this.
+    A number is only terminated by the character that follows it. That character is read from the stream to detect the
+    end of the number, and it is **not** put back. When a value that is a number is immediately followed by the next
+    value, the first character of that next value is lost:
 
-    The terminating character is now only looked at and left in the stream, so no separator is required. Code that
-    relied on the extra byte being swallowed will observe it again.
+    ```cpp
+    std::istringstream input("1true");
+    json j1, j2;
+    input >> j1;  // j1 == 1
+    input >> j2;  // throws parse_error.101: the stream now starts at "rue"
+    ```
+
+    Separating the values with whitespace avoids this, because the character that is eaten is then the separator:
+
+    ```cpp
+    std::istringstream input("1 true");
+    json j1, j2;
+    input >> j1;  // j1 == 1
+    input >> j2;  // j2 == true
+    ```
+
+    Only numbers are affected. Values ending in a self-delimiting character do not read past themselves, so
+    `truefalse`, `[1][2]`, `{"a":1}{"b":2}`, and `"a""b"` can be read back to back without a separator.
+
+    Define [`JSON_PRECISE_STREAM_POSITION`](macros/json_precise_stream_position.md) to `1` to leave the terminating character in the stream
+    instead, so that the stream is positioned right after the value for every value type and no separator is
+    needed. This is tracked in [#5340](https://github.com/nlohmann/json/issues/5340).
 
 Note that reading concatenated values does **not** work for [JSON Lines](../features/parsing/json_lines.md)
 (newline-delimited JSON) input -- see that page for why and for the recommended alternative.
@@ -92,11 +109,12 @@ being read.
 - [parse](basic_json/parse.md) - deserialize from a compatible input
 - [`JSON_STRICT_NUL_HANDLING`](macros/json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input
   instead of treating it as end of input
+- [`JSON_PRECISE_STREAM_POSITION`](macros/json_precise_stream_position.md) - opt in to leaving the stream positioned right after a number
 
 ## Version history
 
 - Added in version 1.0.0.
-- Changed in version 4.0.0 to leave the character that terminates a number in the stream, so that the stream is
-  positioned right after the parsed value for every value type.
 - `JSON_STRICT_NUL_HANDLING` added in version 3.13.0 to optionally reject a NUL byte in the input instead of treating
   it as end of input; planned to become the default in version 4.0.0.
+- `JSON_PRECISE_STREAM_POSITION` added in version 3.13.0 to optionally leave the character that terminates a number in
+  the stream; planned to become the default in version 4.0.0.

docs/mkdocs/docs/features/macros.md

@@ -98,6 +98,15 @@ rather than descending into a bounded number of levels first, which is slower bu
 
 See [full documentation of `JSON_NO_THREAD_LOCAL`](../api/macros/json_no_thread_local.md).
 
+## `JSON_PRECISE_STREAM_POSITION`
+
+When defined to `1`, [`operator>>`](../api/operator_gtgt.md) and non-strict
+[`sax_parse`](../api/basic_json/sax_parse.md) leave an input stream positioned right after the parsed value, instead of
+also consuming the character that terminates a number. The default value is `0`, which preserves the existing behavior;
+this is planned to become the default in version 4.0.0.
+
+See [full documentation of `JSON_PRECISE_STREAM_POSITION`](../api/macros/json_precise_stream_position.md).
+
 ## `JSON_SKIP_LIBRARY_VERSION_CHECK`
 
 When defined, the library will not create a compiler warning when a different version of the library was already

docs/mkdocs/docs/features/namespace.md

@@ -18,6 +18,7 @@ The complete default namespace name is derived as follows:
     - [`JSON_DIAGNOSTIC_POSITIONS`](../api/macros/json_diagnostic_positions.md) defined non-zero appends `_dp`.
     - [`JSON_BRACE_INIT_COPY_SEMANTICS`](../api/macros/json_brace_init_copy_semantics.md) defined non-zero appends
       `_bics`.
+    - [`JSON_PRECISE_STREAM_POSITION`](../api/macros/json_precise_stream_position.md) defined non-zero appends `_psp`.
 - The inline namespace ends with the suffix `_v` followed by the 3 components of the version number separated by
   underscores. To omit the version component, see [Disabling the version component](#disabling-the-version-component)
   below.

docs/mkdocs/docs/features/parsing/index.md

@@ -40,7 +40,9 @@ what makes it possible to read several concatenated values from the same stream,
 document followed by trailing bytes" is accepted rather than rejected. If you are validating conformance, or need to
 reject any input that is not exactly one JSON document, prefer `parse`.
 
-Values read this way do not need to be separated by whitespace; see the
+When using `operator>>` to read several concatenated values this way, a value that is a number must be followed by
+whitespace, because `operator>>` consumes the character that terminates a number, unless
+[`JSON_PRECISE_STREAM_POSITION`](../../api/macros/json_precise_stream_position.md) is defined to `1` — see the
 [`operator>>` notes](../../api/operator_gtgt.md#notes) for details and examples.
 
 ## SAX vs. DOM parsing

docs/mkdocs/docs/features/parsing/json_lines.md

@@ -49,4 +49,5 @@ JSON Lines input with more than one value is treated as invalid JSON by the [`pa
     with a JSON Lines input does not work, because the parser will try to parse one value after the last one.
 
     This is different from parsing a stream of *concatenated* (non-newline-delimited) JSON values, for which
-    `operator>>` does work -- see its [notes](../../api/operator_gtgt.md#notes) for details.
+    `operator>>` does work, provided that a value that is a number is followed by whitespace -- see its
+    [notes](../../api/operator_gtgt.md#notes) for details.

docs/mkdocs/mkdocs.yml

@@ -293,6 +293,7 @@ nav:
           - 'JSON_NOEXCEPTION': api/macros/json_noexception.md
           - 'JSON_NO_IO': api/macros/json_no_io.md
           - 'JSON_NO_THREAD_LOCAL': api/macros/json_no_thread_local.md
+          - 'JSON_PRECISE_STREAM_POSITION': api/macros/json_precise_stream_position.md
           - 'JSON_SKIP_LIBRARY_VERSION_CHECK': api/macros/json_skip_library_version_check.md
           - 'JSON_SKIP_UNSUPPORTED_COMPILER_CHECK': api/macros/json_skip_unsupported_compiler_check.md
           - 'JSON_STRICT_NUL_HANDLING': api/macros/json_strict_nul_handling.md