Merge branch 'develop' into bon8

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/get_to.md

@@ -21,6 +21,10 @@ This overload is chosen if:
 - `ValueType` is not `basic_json`,
 - `json_serializer<ValueType>` has a `from_json()` method of the form `void from_json(const basic_json&, ValueType&)`
 
+`v` must not be `const`. Passing a `const` object is a compile-time error. For types such as arithmetic types, enums,
+and C arrays, the error is a `static_assert` that names the problem. For other types, the overload is not viable, and
+the compiler reports that no matching `get_to` was found.
+
 ## Template parameters
 
 `ValueType`
@@ -67,3 +71,4 @@ Depends on the `json_serializer<ValueType>::from_json()` implementation.
 ## Version history
 
 - Since version 3.3.0.
+- Added a `static_assert` with a clear message for `const` arguments in version 3.13.0.

docs/mkdocs/docs/api/basic_json/sax_parse.md

@@ -70,7 +70,9 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index
      [`input_format_t`](input_format_t.md) for more information
 
 `strict` (in)
-:   whether the input has to be consumed completely (optional, `#!cpp true` by default)
+:   whether the input has to be consumed completely (optional, `#!cpp true` by default); when `#!cpp false` and the
+    input is a `#!cpp std::istream`, the character that terminates a number is consumed unless
+    [`JSON_PRECISE_STREAM_POSITION`](../macros/json_precise_stream_position.md) is defined to `1`; see [`operator>>`](../operator_gtgt.md#notes)
 
 `ignore_comments` (in)
 :   whether comments should be ignored and treated like whitespace (`#!cpp true`) or yield a parse error
@@ -137,6 +139,8 @@ A UTF-8 byte order mark is silently ignored.
 - Added `ignore_trailing_commas` in version 3.13.0.
 - Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
 - Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
+- `JSON_PRECISE_STREAM_POSITION` added in version 3.13.0 to optionally leave a `#!cpp std::istream` positioned right
+  after the parsed value when `strict` is `#!cpp false`.
 
 !!! warning "Deprecation"
 

docs/mkdocs/docs/api/macros/index.md

@@ -16,6 +16,8 @@ header. See also the [macro overview page](../../features/macros.md).
 
 ## Parsing
 
+- [**JSON_PRECISE_STREAM_POSITION**](json_precise_stream_position.md) - opt in to leaving an input stream positioned
+  right after a parsed number
 - [**JSON_STRICT_NUL_HANDLING**](json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input instead of
   treating it as end of input
 

docs/mkdocs/docs/api/macros/json_no_thread_local.md

@@ -7,16 +7,16 @@
 When defined, the library does not use `#!cpp thread_local` storage. This is relevant for the few environments whose
 toolchain does not support it.
 
-The copy constructor copies the first levels of a value by copying the containers, which copy their elements, and
-completes whatever is nested deeper than that without the call stack, so that copying a value cannot exhaust the stack
-however deeply it is nested. It counts the levels it has descended into in a `#!cpp thread_local` variable, as a counter
-shared between threads would be raced.
+Copying a value and comparing two values both descend into the first levels by letting the containers copy or compare
+themselves, and finish whatever is nested deeper than that without the call stack, so that neither can exhaust the stack
+however deeply the values are nested. Each counts the levels it has descended into in a `#!cpp thread_local` variable, as
+a counter shared between threads would be raced.
 
-Without that counter, no descent can be bounded safely, so objects and arrays are copied without the call stack right
-away. Copying keeps working exactly as it does otherwise - the same values come out, and deeply nested values are copied
-just as safely - but copying is slower, because the containers no longer copy themselves. Copying the benchmark
-documents takes 9% (`canada.json`) to 34% (`twitter.json`) longer; values built mostly from objects are affected the
-most.
+Without those counters, no descent can be bounded safely, so objects and arrays are copied and compared without the call
+stack right away. Both keep working exactly as they do otherwise - the same values come out, the same comparisons hold,
+and deeply nested values are handled just as safely - but both are slower, because the containers no longer copy or
+compare themselves. Copying the benchmark documents takes 9% (`canada.json`) to 34% (`twitter.json`) longer, and
+comparing two equal ones 10% (`citm_catalog.json`) to 90% (`canada.json`) longer.
 
 ## Default definition
 
@@ -28,6 +28,7 @@ By default, `#!cpp JSON_NO_THREAD_LOCAL` is not defined.
 
 The library defines it by itself for Clang targeting MinGW, which does not survive the `#!cpp thread_local` storage:
 copying a value segfaults there, with both old and current Clang versions, while GCC targeting MinGW is unaffected.
+Copying and comparing fall back to working without the call stack there, as they do whenever the macro is defined.
 
 ## Examples
 

docs/mkdocs/docs/api/macros/json_precise_stream_position.md

@@ -0,0 +1,131 @@
+# JSON_PRECISE_STREAM_POSITION
+
+```cpp
+#define JSON_PRECISE_STREAM_POSITION /* value */
+```
+
+When defined to `1`, [`operator>>`](../operator_gtgt.md) and [`sax_parse`](../basic_json/sax_parse.md) with
+`strict = false` leave a `#!cpp std::istream` positioned right after the parsed value for every value type. By default,
+the character that terminates a number is consumed as well.
+
+The macro only affects reading from a `#!cpp std::istream` when the rest of the stream is not required to be consumed.
+[`parse`](../basic_json/parse.md), [`accept`](../basic_json/accept.md), and all other inputs (strings, iterators,
+containers, `#!cpp FILE*`) are never affected.
+
+## Default definition
+
+The default value is `0` (disabled — existing behavior is preserved).
+
+```cpp
+#define JSON_PRECISE_STREAM_POSITION 0
+```
+
+## Notes
+
+!!! note "Background"
+
+    A number is the only JSON value whose end can be detected solely by reading the character that follows it. By
+    default, that character is consumed and not put back, so the stream is left one byte too far after a number, and
+    only after a number:
+
+    ```cpp
+    std::istringstream input("1true");
+    json j;
+    input >> j;  // j == 1, but the stream now starts at "rue"
+    ```
+
+    With this macro, the character is only looked at and left in the stream, so the stream starts at `true`. This
+    does not require the stream buffer to support putting a character back.
+
+    This was not changed unconditionally, because code can depend on the consumed character, even unknowingly (see
+    [#5340](https://github.com/nlohmann/json/issues/5340)). Both of the following work by default only because the
+    character after each number is swallowed, and behave differently with this macro:
+
+    ```cpp
+    std::istringstream input("1,2,3");
+    json j1, j2, j3;
+    input >> j1 >> j2 >> j3;  // default: 1, 2, 3
+                              // with the macro: throws parse_error.101 at the ','
+    ```
+
+    ```cpp
+    std::istringstream input("42\nfoo");
+    json j;
+    std::string line;
+    input >> j;
+    std::getline(input, line);  // default: "foo"
+                                // with the macro: "" (like after reading an int with >>)
+    ```
+
+    In both cases, the behavior with the macro is what you already get today when the value is not a number: `"a","b"`
+    fails at the `,`, and `std::getline` after `{}` returns an empty string. This macro offers an opt-in path to
+    the consistent behavior ahead of version 4.0.0, where it is planned to become the default.
+
+!!! warning "Opt-in only"
+
+    This macro must be defined **before** including `<nlohmann/json.hpp>`. Defining it after the include has no
+    effect.
+
+!!! note "ABI compatibility"
+
+    The value of this macro is encoded in the [namespace](../../features/namespace.md) (tag `_psp`), resulting in
+    distinct symbol names. Translation units compiled with and without it can therefore be linked into the same program
+    without One Definition Rule (ODR) violations, but they cannot exchange instances of library types.
+
+!!! tip "Workaround without the macro"
+
+    Separate the values in the stream with whitespace. The character consumed after a number is then the separator,
+    and whitespace before the next value is skipped anyway.
+
+## Examples
+
+??? example "Default behavior (macro not defined)"
+
+    Without the macro, the character after a number is consumed:
+
+    ```cpp
+    #include <iostream>
+    #include <sstream>
+    #include <nlohmann/json.hpp>
+
+    using json = nlohmann::json;
+
+    int main()
+    {
+        std::istringstream input("1true");
+        json j1, j2;
+        input >> j1;  // j1 == 1
+        input >> j2;  // throws parse_error.101: the stream now starts at "rue"
+    }
+    ```
+
+??? example "Opt-in precise stream position (macro defined to 1)"
+
+    With the macro, the stream is positioned right after the number:
+
+    ```cpp
+    #define JSON_PRECISE_STREAM_POSITION 1
+    #include <iostream>
+    #include <sstream>
+    #include <nlohmann/json.hpp>
+
+    using json = nlohmann::json;
+
+    int main()
+    {
+        std::istringstream input("1true");
+        json j1, j2;
+        input >> j1;  // j1 == 1
+        input >> j2;  // j2 == true
+    }
+    ```
+
+## See also
+
+- [**operator>>**](../operator_gtgt.md) - deserialize from stream
+- [**sax_parse**](../basic_json/sax_parse.md) - generate SAX events
+
+## Version history
+
+- Added in version 3.13.0.
+- Planned to become the default (with the macro removed) in version 4.0.0.

docs/mkdocs/docs/api/operator_gtgt.md

@@ -67,7 +67,9 @@ input >> j2;  // parses the next value
     Only numbers are affected. Values ending in a self-delimiting character do not read past themselves, so
     `truefalse`, `[1][2]`, `{"a":1}{"b":2}`, and `"a""b"` can be read back to back without a separator.
 
-    This is tracked in [#5340](https://github.com/nlohmann/json/issues/5340).
+    Define [`JSON_PRECISE_STREAM_POSITION`](macros/json_precise_stream_position.md) to `1` to leave the terminating character in the stream
+    instead, so that the stream is positioned right after the value for every value type and no separator is
+    needed. This is tracked in [#5340](https://github.com/nlohmann/json/issues/5340).
 
 Note that reading concatenated values does **not** work for [JSON Lines](../features/parsing/json_lines.md)
 (newline-delimited JSON) input -- see that page for why and for the recommended alternative.
@@ -107,9 +109,12 @@ being read.
 - [parse](basic_json/parse.md) - deserialize from a compatible input
 - [`JSON_STRICT_NUL_HANDLING`](macros/json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input
   instead of treating it as end of input
+- [`JSON_PRECISE_STREAM_POSITION`](macros/json_precise_stream_position.md) - opt in to leaving the stream positioned right after a number
 
 ## Version history
 
 - Added in version 1.0.0.
 - `JSON_STRICT_NUL_HANDLING` added in version 3.13.0 to optionally reject a NUL byte in the input instead of treating
   it as end of input; planned to become the default in version 4.0.0.
+- `JSON_PRECISE_STREAM_POSITION` added in version 3.13.0 to optionally leave the character that terminates a number in
+  the stream; planned to become the default in version 4.0.0.

docs/mkdocs/docs/community/assurance_case.md

@@ -0,0 +1,70 @@
+# Assurance case
+
+This page argues why the library meets its security requirements. It describes the threats the library faces, where the
+trust boundaries lie, and how the library's design and the [quality assurance](quality_assurance.md) counter these
+threats. To report a vulnerability, see the [security policy](security_policy.md).
+
+## Threat model
+
+The library parses, stores, and serializes JSON values in memory. It does not open network connections, does not open
+files (it only reads from streams or `std::FILE*` handles that the caller has already opened), does not read environment
+variables, and does not implement cryptography or handle credentials.
+
+The primary threat is therefore **untrusted input**: JSON text or binary data (BJData, BSON, CBOR, MessagePack, UBJSON)
+that an attacker controls, passed to [`parse`](../api/basic_json/parse.md), [`accept`](../api/basic_json/accept.md),
+[`sax_parse`](../api/basic_json/sax_parse.md), or one of the `from_*` functions such as
+[`from_cbor`](../api/basic_json/from_cbor.md). Such input may try to
+
+- make the library read or write out of bounds (malformed lengths, truncated input, invalid UTF-8),
+- trigger undefined behavior (integer overflow in sizes or numbers, invalid casts),
+- exhaust memory (huge announced sizes), or
+- exhaust the call stack (deeply nested arrays and objects).
+
+## Trust boundaries
+
+- **Untrusted:** all serialized input read by the parser, the SAX interface, and the binary readers. The library must
+  handle every possible input by either producing a value or throwing a [`parse_error`](../home/exceptions.md#parse-errors)
+  (or returning `false` when exceptions are disabled for the call).
+- **Trusted:** the C++ code that calls the library. Calling a function with violated preconditions, for instance
+  accessing an array with [`operator[]`](../api/basic_json/operator%5B%5D.md) out of range, is a programming error and
+  not a security boundary. Such preconditions are checked with [runtime assertions](../features/assertions.md) in debug
+  builds; functions such as [`at`](../api/basic_json/at.md) offer checked access with exceptions.
+
+## Secure design
+
+- **Strict parsing.** The parser accepts exactly the JSON grammar of [RFC 8259](https://datatracker.ietf.org/doc/html/rfc8259).
+  Extensions such as [comments](../features/comments.md) and [trailing commas](../features/trailing_commas.md) must be
+  enabled explicitly. Invalid UTF-8 is rejected.
+- **Errors are reported, not ignored.** Malformed input results in a [`parse_error`](../home/exceptions.md#parse-errors)
+  with the byte position of the error. Binary readers do not trust announced sizes: strings and binary values grow
+  only as bytes are actually read, arrays reserve at most a fixed number of elements up front, and sizes that no
+  container can hold are rejected.
+- **Memory is owned by values.** Each `basic_json` value owns its content, and there is no manual memory management in
+  user code. The destructor does not recurse, so destroying a deeply nested value does not exhaust the stack.
+- **Bounded recursion.** The JSON parser and the binary readers keep their state in explicit stacks instead of
+  recursing per nesting level. Operations that walk a value, such as [`dump`](../api/basic_json/dump.md), copying,
+  hashing, and [`merge_patch`](../api/basic_json/merge_patch.md), recurse only up to a fixed depth and continue with an
+  explicit stack below it. Some operations, such as comparison, [`diff`](../api/basic_json/diff.md),
+  [`flatten`](../api/basic_json/flatten.md), and the binary writers, still recurse once per nesting level; work on them
+  is in progress. Applications that process untrusted input can limit its nesting depth with a
+  [parser callback](../features/parsing/parser_callbacks.md).
+- **Invariants are checked.** The class invariant (for instance, that the pointer for the stored type is never null) is
+  checked with runtime assertions throughout the test suite.
+
+## Common weaknesses
+
+The following table maps the relevant classes of the [Common Weakness Enumeration](https://cwe.mitre.org) to the
+measures that counter them. The measures are described in detail in [Quality assurance](quality_assurance.md).
+
+| Weakness                                                                  | Countermeasures                                                                                  |
+|---------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------|
+| Out-of-bounds read/write ([CWE-125](https://cwe.mitre.org/data/definitions/125.html), [CWE-787](https://cwe.mitre.org/data/definitions/787.html)) | bounds checks on all reads from the input; AddressSanitizer and Valgrind on the test suite; OSS-Fuzz      |
+| Integer overflow ([CWE-190](https://cwe.mitre.org/data/definitions/190.html)) | UndefinedBehaviorSanitizer with integer overflow detection; Clang-Tidy; Cppcheck                 |
+| Use after free, double free ([CWE-416](https://cwe.mitre.org/data/definitions/416.html), [CWE-415](https://cwe.mitre.org/data/definitions/415.html)) | ownership of all memory by values; AddressSanitizer and Valgrind; Clang Static Analyzer          |
+| Memory leaks ([CWE-401](https://cwe.mitre.org/data/definitions/401.html)) | Valgrind (Memcheck) on the test suite                                                            |
+| Uncontrolled recursion ([CWE-674](https://cwe.mitre.org/data/definitions/674.html)) | iterative parser, binary readers, and destructor; bounded recursion in value operations; tests with deeply nested inputs |
+| Uncontrolled resource consumption ([CWE-400](https://cwe.mitre.org/data/definitions/400.html)) | allocations based on announced sizes are capped; OSS-Fuzz with memory limits                     |
+| Undefined behavior in general ([CWE-758](https://cwe.mitre.org/data/definitions/758.html)) | UndefinedBehaviorSanitizer; runtime assertions; Clang-Tidy, Cppcheck, Clang Static Analyzer, Infer |
+
+In addition, every line of the library is covered by the unit tests, and all parsers are fuzz-tested around the clock
+by [OSS-Fuzz](https://github.com/google/oss-fuzz/tree/master/projects/json).

docs/mkdocs/docs/community/index.md

@@ -5,4 +5,6 @@
 - [Contribution Guidelines](contribution_guidelines.md) - guidelines how to contribute to this project
 - [Governance](governance.md) - the governance model of this project
 - [Quality Assurance](quality_assurance.md) - how the quality of this project is assured
+- [Roadmap](roadmap.md) - what the project will and will not do
 - [Security Policy](security_policy.md) - the security policy of the project
+- [Assurance Case](assurance_case.md) - why the library meets its security requirements

docs/mkdocs/docs/community/quality_assurance.md

@@ -164,6 +164,9 @@ Note: Some modern features (like C++20 ranges or filesystem support) may be disa
 - [x] The parser is tested against extensive correctness suites for JSON compliance.
 - [x] In addition, the library is continuously fuzz-tested at [OSS-Fuzz](https://google.github.io/oss-fuzz/) where the
   library is checked against billions of inputs.
+- [x] Every crash reported by OSS-Fuzz is fixed together with a unit test that reproduces it, and the fix references
+  the OSS-Fuzz issue. The round-trip checks of the fuzzer drivers are also part of the unit tests. See the
+  [fuzz testing documentation](https://github.com/nlohmann/json/blob/develop/tests/fuzzing.md#handling-oss-fuzz-reports).
 
 ## Static analysis
 

docs/mkdocs/docs/community/roadmap.md

@@ -0,0 +1,43 @@
+# Roadmap
+
+This page describes what the project intends to do, and what it does not intend to do, over the next year. Concrete
+work items are tracked in the [GitHub milestones](https://github.com/nlohmann/json/milestones) and the
+[issue tracker](https://github.com/nlohmann/json/issues).
+
+## What the project will do
+
+- **Keep the C++11 baseline.** The library will continue to compile with every
+  [supported C++11 compiler](https://github.com/nlohmann/json/blob/develop/README.md#supported-compilers). Features of
+  later standards are only used when they are guarded by the `JSON_HAS_CPP_*` macros.
+- **Stay conformant to JSON.** The parser and serializer follow [RFC 8259](https://datatracker.ietf.org/doc/html/rfc8259).
+  Extensions such as [comments](../features/comments.md) or [trailing commas](../features/trailing_commas.md) remain
+  opt-in.
+- **Keep the 3.x public API stable.** Releases follow [semantic versioning](https://semver.org). Changes that would
+  break existing code are only added behind a feature macro, so users can opt in and test their code before a next
+  major release.
+- **Support a broad range of compilers and platforms.** The [CI](quality_assurance.md) keeps testing old and new
+  versions of GCC, Clang, MSVC, and other compilers on Linux, macOS, and Windows.
+- **Keep the quality assurance up.** Every change keeps the test coverage at 100%, passes the static and dynamic
+  analysis, and is fuzz-tested by OSS-Fuzz, see [Quality assurance](quality_assurance.md).
+- **Harden the library against hostile input.** Handling deeply nested values without exhausting the call stack is
+  ongoing work.
+- **Fix bugs and security issues** reported through the issue tracker and the [security policy](security_policy.md).
+
+## What the project will not do
+
+- **Break the public API of version 3.x.** See the
+  [contribution guidelines](https://github.com/nlohmann/json/blob/develop/.github/CONTRIBUTING.md#break-the-public-api)
+  for what counts as a breaking change.
+- **Require a newer C++ standard than C++11.**
+- **Break JSON conformance** or enable non-standard extensions by default.
+- **Add dependencies** or require a build step. The library remains header-only, and the single header
+  `json.hpp` remains a complete distribution.
+- **Trade simplicity for speed or memory efficiency.** Performance improvements are welcome, but the library is not
+  meant to compete with the fastest JSON libraries, see [Design goals](../home/design_goals.md).
+
+## Version 4.0
+
+There is no decision yet on whether or when a version 4.0 with breaking changes will be released. Proposals that need
+a major version, for instance stricter type conversions, are collected in issue
+[#3453](https://github.com/nlohmann/json/issues/3453). Until then, such changes are only added as opt-in behavior
+behind feature macros.

docs/mkdocs/docs/features/binary_formats/bjdata.md

@@ -208,6 +208,16 @@ The library maps BJData types to JSON value types as follows:
 
     The mapping is **complete** in the sense that any BJData value can be converted to a JSON value.
 
+!!! info "Round trips"
+
+    A value returned by [`from_bjdata`](../../api/basic_json/from_bjdata.md) can be serialized with
+    [`to_bjdata`](../../api/basic_json/to_bjdata.md) using any combination of options and parsed back into an equal
+    value, and serializing that value again with the same options produces the same bytes. The exception is binary
+    values: they are only written as an optimized binary array (`[$B`) if Draft 3 is enabled and both `use_size` and
+    `use_type` are set. Otherwise, they are written as arrays of integers and parsed back as such (see the notes on
+    binary values above), and serializing such an array again may choose different, but equally valid, type markers.
+    The bytes can then differ, but parsing them again yields the same value.
+
 ??? example
 
     ```cpp

docs/mkdocs/docs/features/macros.md

@@ -93,11 +93,21 @@ See [full documentation of `JSON_NO_IO`](../api/macros/json_no_io.md).
 
 ## `JSON_NO_THREAD_LOCAL`
 
-When defined, the library does not use `#!cpp thread_local` storage. Copying a value then always avoids the call stack
-rather than descending into a bounded number of levels first, which is slower but yields the same values.
+When defined, the library does not use `#!cpp thread_local` storage. Copying a value and comparing two values then
+always avoid the call stack rather than descending into a bounded number of levels first, which is slower but yields the
+same values and the same comparisons.
 
 See [full documentation of `JSON_NO_THREAD_LOCAL`](../api/macros/json_no_thread_local.md).
 
+## `JSON_PRECISE_STREAM_POSITION`
+
+When defined to `1`, [`operator>>`](../api/operator_gtgt.md) and non-strict
+[`sax_parse`](../api/basic_json/sax_parse.md) leave an input stream positioned right after the parsed value, instead of
+also consuming the character that terminates a number. The default value is `0`, which preserves the existing behavior;
+this is planned to become the default in version 4.0.0.
+
+See [full documentation of `JSON_PRECISE_STREAM_POSITION`](../api/macros/json_precise_stream_position.md).
+
 ## `JSON_SKIP_LIBRARY_VERSION_CHECK`
 
 When defined, the library will not create a compiler warning when a different version of the library was already

docs/mkdocs/docs/features/namespace.md

@@ -18,6 +18,7 @@ The complete default namespace name is derived as follows:
     - [`JSON_DIAGNOSTIC_POSITIONS`](../api/macros/json_diagnostic_positions.md) defined non-zero appends `_dp`.
     - [`JSON_BRACE_INIT_COPY_SEMANTICS`](../api/macros/json_brace_init_copy_semantics.md) defined non-zero appends
       `_bics`.
+    - [`JSON_PRECISE_STREAM_POSITION`](../api/macros/json_precise_stream_position.md) defined non-zero appends `_psp`.
 - The inline namespace ends with the suffix `_v` followed by the 3 components of the version number separated by
   underscores. To omit the version component, see [Disabling the version component](#disabling-the-version-component)
   below.

docs/mkdocs/docs/features/parsing/index.md

@@ -41,7 +41,8 @@ document followed by trailing bytes" is accepted rather than rejected. If you ar
 reject any input that is not exactly one JSON document, prefer `parse`.
 
 When using `operator>>` to read several concatenated values this way, a value that is a number must be followed by
-whitespace, because `operator>>` consumes the character that terminates a number — see the
+whitespace, because `operator>>` consumes the character that terminates a number, unless
+[`JSON_PRECISE_STREAM_POSITION`](../../api/macros/json_precise_stream_position.md) is defined to `1` — see the
 [`operator>>` notes](../../api/operator_gtgt.md#notes) for details and examples.
 
 ## SAX vs. DOM parsing

docs/mkdocs/docs/home/architecture.md

@@ -1,34 +1,125 @@
 # Architecture
 
-!!! info
-
-    This page is still under construction. Its goal is to provide a high-level overview of the library's architecture.
-    This should help new contributors to get an idea of the used concepts and where to make changes.
+This page gives a high-level overview of the library's architecture. It should help new contributors to get an idea of
+the used concepts and where to make changes.
 
 ## Overview
 
-The main structure is class [nlohmann::basic_json](../api/basic_json/index.md).
+The library is built around a single class template, [`nlohmann::basic_json`](../api/basic_json/index.md). A
+`basic_json` value is a node in a tree of JSON values. All other components either create such a tree from an input
+(parsing), write a tree to an output (serialization), or give access to it (iterators, JSON Pointer, conversions).
 
-- public API
-- container interface
-- iterators
+```mermaid
+flowchart LR
+    input[/"input<br>(string, stream,<br>iterator range, file)"/]
+    ia["input adapter"]
+    lexer["lexer"]
+    parser["parser"]
+    breader["binary_reader"]
+    sax["SAX interface"]
+    value[("basic_json<br>value tree")]
+    serializer["serializer"]
+    bwriter["binary_writer"]
+    oa["output adapter"]
+    output[/"output<br>(string, stream,<br>vector)"/]
 
-## Template specializations
+    input --> ia
+    ia --> lexer --> parser --> sax
+    ia --> breader --> sax
+    sax --> value
+    value --> serializer --> oa
+    value --> bwriter --> oa
+    oa --> output
+```
 
-- describe template parameters of `basic_json`
-- [`json`](../api/json.md)
-- [`ordered_json`](../api/ordered_json.md) via [`ordered_map`](../api/ordered_map.md)
+- **JSON text** is read by an [input adapter](#input-adapters), tokenized by the lexer, and turned into SAX events by
+  the parser.
+- **Binary formats** (BJData, BSON, CBOR, MessagePack, UBJSON) are read by an input adapter and turned into the same SAX
+  events by the `binary_reader`.
+- A [SAX consumer](#sax-interface) receives the events. The one used by [`parse`](../api/basic_json/parse.md) builds a
+  `basic_json` value tree.
+- The `serializer` (JSON text) or the `binary_writer` (binary formats) writes a value tree to an
+  [output adapter](#output-adapters).
+
+## Source layout
+
+The public headers are in [`include/nlohmann`](https://github.com/nlohmann/json/tree/develop/include/nlohmann):
+
+- [`json.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/json.hpp) defines class [`basic_json`](../api/basic_json/index.md).
+- [`json_fwd.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/json_fwd.hpp) contains forward declarations.
+- [`adl_serializer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/adl_serializer.hpp), [`byte_container_with_subtype.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/byte_container_with_subtype.hpp), and [`ordered_map.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/ordered_map.hpp) define
+  [`adl_serializer`](../api/adl_serializer/index.md),
+  [`byte_container_with_subtype`](../api/byte_container_with_subtype/index.md), and
+  [`ordered_map`](../api/ordered_map.md).
+
+Everything else lives in [`detail/`](https://github.com/nlohmann/json/tree/develop/include/nlohmann/detail) and namespace `nlohmann::detail`, which is not part of the public API. Paths
+below are relative to `include/nlohmann`.
+
+| Component | Location |
+|-----------|----------|
+| Value type enumeration | [`detail/value_t.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/value_t.hpp) |
+| Input adapters | [`detail/input/input_adapters.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/input_adapters.hpp) |
+| Lexer | [`detail/input/lexer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/lexer.hpp), [`detail/input/number_parse.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/number_parse.hpp), [`detail/input/string_scan.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/string_scan.hpp) |
+| Parser | [`detail/input/parser.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/parser.hpp) |
+| SAX interface and DOM builders | [`detail/input/json_sax.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/json_sax.hpp) |
+| Binary format readers | [`detail/input/binary_reader.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/binary_reader.hpp) |
+| JSON serializer | [`detail/output/serializer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/output/serializer.hpp), [`detail/conversions/to_chars.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/conversions/to_chars.hpp) |
+| Binary format writers | [`detail/output/binary_writer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/output/binary_writer.hpp) |
+| Output adapters | [`detail/output/output_adapters.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/output/output_adapters.hpp) |
+| Iterators | [`detail/iterators/`](https://github.com/nlohmann/json/tree/develop/include/nlohmann/detail/iterators) |
+| Conversions from/to arbitrary types | [`detail/conversions/from_json.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/conversions/from_json.hpp), [`detail/conversions/to_json.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/conversions/to_json.hpp) |
+| JSON Pointer | [`detail/json_pointer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/json_pointer.hpp) |
+| Exceptions | [`detail/exceptions.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/exceptions.hpp) |
+| Type traits and C++ feature backports | [`detail/meta/`](https://github.com/nlohmann/json/tree/develop/include/nlohmann/detail/meta) |
+| Macros | [`detail/macro_scope.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/macro_scope.hpp), [`detail/macro_unscope.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/macro_unscope.hpp), [`detail/abi_macros.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/abi_macros.hpp) |
+
+The single-header version [`single_include/nlohmann/json.hpp`](https://github.com/nlohmann/json/blob/develop/single_include/nlohmann/json.hpp)
+is generated from these files with `make amalgamate` and must not be edited by hand.
+
+## Template parameters
+
+[`basic_json`](../api/basic_json/index.md) is parameterized by the types it uses to store values and to convert from and to other types:
+
+| Template parameter   | Default                     | Used for                                                          |
+|----------------------|-----------------------------|-------------------------------------------------------------------|
+| `ObjectType`         | `std::map`                  | objects, see [`object_t`](../api/basic_json/object_t.md)          |
+| `ArrayType`          | `std::vector`               | arrays, see [`array_t`](../api/basic_json/array_t.md)             |
+| `StringType`         | `std::string`               | strings and object keys, see [`string_t`](../api/basic_json/string_t.md) |
+| `BooleanType`        | `bool`                      | Booleans, see [`boolean_t`](../api/basic_json/boolean_t.md)       |
+| `NumberIntegerType`  | `std::int64_t`              | signed integers, see [`number_integer_t`](../api/basic_json/number_integer_t.md) |
+| `NumberUnsignedType` | `std::uint64_t`             | unsigned integers, see [`number_unsigned_t`](../api/basic_json/number_unsigned_t.md) |
+| `NumberFloatType`    | `double`                    | floating-point numbers, see [`number_float_t`](../api/basic_json/number_float_t.md) |
+| `AllocatorType`      | `std::allocator`            | allocating objects, arrays, strings, and binary values            |
+| `JSONSerializer`     | `adl_serializer`            | conversions from/to other types, see [`adl_serializer`](../api/adl_serializer/index.md) |
+| `BinaryType`         | `std::vector<std::uint8_t>` | binary values, see [`binary_t`](../api/basic_json/binary_t.md)  |
+| `CustomBaseClass`    | `void`                      | an optional base class, see [`json_base_class_t`](../api/basic_json/json_base_class_t.md) |
+
+The library provides two specializations:
+
+- [`json`](../api/json.md) uses all default template arguments.
+- [`ordered_json`](../api/ordered_json.md) uses [`ordered_map`](../api/ordered_map.md) as `ObjectType` to keep the
+  insertion order of object keys.
+
+The requirements on the template arguments are listed in
+[Template Parameter Requirements](../features/types/template_parameters.md).
 
 ## Value storage
 
-Values are stored as a tagged union of [value_t](../api/basic_json/value_t.md) and json_value.
+Each [`basic_json`](../api/basic_json/index.md) value stores its content as a tagged union: an enumeration [`value_t`](../api/basic_json/value_t.md)
+names the type of the value, and a union `json_value` holds the value itself. Both are members of the nested struct
+`data`, which is the only data member `m_data` of `basic_json`:
 
 ```cpp
-/// the type of the current element
-value_t m_type = value_t::null;
+struct data
+{
+    /// the type of the current element
+    value_t m_type = value_t::null;
 
-/// the value of the current element
-json_value m_value = {};
+    /// the value of the current element
+    json_value m_value = {};
+};
+
+data m_data = {};
 ```
 
 with
@@ -68,42 +159,83 @@ union json_value {
 };
 ```
 
-## Parsing inputs (deserialization)
+Objects, arrays, strings, and binary values are allocated on the heap with `AllocatorType`, and the union only stores a
+pointer to them. This keeps a `basic_json` value small: one pointer-sized union and one byte for the type. The class
+maintains the invariant that the pointer matching `m_type` is never null; `assert_invariant()` checks it with
+[runtime assertions](../features/assertions.md).
 
-Input is read via **input adapters** that abstract a source with a common interface:
+## Input adapters
+
+Input is read via **input adapters** that abstract a source. Every input adapter provides this interface:
 
 ```cpp
-/// read a single character
-std::char_traits<char>::int_type get_character() noexcept;
+/// the type of the characters in the input
+using char_type = ...;
 
-/// read multiple characters to a destination buffer and
-/// returns the number of characters successfully read
+/// read a single character; returns std::char_traits<char_type>::eof() at the end of the input
+typename std::char_traits<char_type>::int_type get_character();
+
+/// read up to count * sizeof(T) bytes into dest and return the number of bytes read
+/// (used by the binary readers)
 template<class T>
 std::size_t get_elements(T* dest, std::size_t count = 1);
 ```
 
-List examples of input adapters.
+The lexer detects two optional extensions at compile time. Only `iterator_input_adapter` provides them, and only for
+random-access input of single-byte characters:
 
-## SAX Interface
+- `supports_seek`, `get_consumed_count()`, and `copy_consumed_range()` let the lexer reconstruct already consumed input
+  for error messages instead of copying every character it reads.
+- `supports_bulk_scan`, `bulk_data()`, `bulk_remaining()`, and `bulk_skip()` let the lexer scan strings directly in
+  contiguous memory, several bytes at a time.
 
-TODO
+The function `input_adapter` picks the right adapter for the argument passed to `parse`, `accept`, `sax_parse`, or the
+`from_*` functions:
 
-## Writing outputs (serialization)
+- `iterator_input_adapter` reads from an iterator range, which also covers strings, containers, and pointers.
+- `wide_string_input_adapter` reads from ranges of `wchar_t`, `char16_t`, or `char32_t` and converts them to UTF-8.
+  It cannot be used for binary formats; its `get_elements()` throws.
+- `input_stream_adapter` reads from a `std::istream`.
+- `file_input_adapter` reads from a `std::FILE*`.
+
+## SAX interface
+
+The parser does not build values itself. It reports what it reads as events to a [SAX](../features/parsing/sax_interface.md)
+consumer, which implements the interface [`json_sax`](../api/json_sax/index.md): `null`, `boolean`, `number_integer`,
+`number_unsigned`, `number_float`, `string`, `binary`, `start_object`, `key`, `end_object`, `start_array`, `end_array`,
+and `parse_error`.
+
+The library comes with two consumers in `detail/input/json_sax.hpp`:
+
+- `json_sax_dom_parser` builds a [`basic_json`](../api/basic_json/index.md) value tree. [`parse`](../api/basic_json/parse.md) uses it.
+- `json_sax_dom_callback_parser` does the same, but calls a [parser callback](../features/parsing/parser_callbacks.md)
+  for each event, which can skip values. `parse` uses it when a callback is given.
+
+The `binary_reader` emits the same events for binary formats, so [`sax_parse`](../api/basic_json/sax_parse.md) works
+with a user-defined consumer for JSON and for all binary formats alike.
+
+## Output adapters
 
 Output is written via **output adapters**:
 
 ```cpp
-template<typename T>
 void write_character(CharType c);
 
-template<typename CharType>
 void write_characters(const CharType* s, std::size_t length);
 ```
 
-List examples of output adapters.
+The `serializer` (used by [`dump`](../api/basic_json/dump.md) and [`operator<<`](../api/operator_ltlt.md)) and the
+`binary_writer` (used by the `to_*` functions) write to one of these adapters:
+
+- `output_vector_adapter` appends to a `std::vector`.
+- `output_stream_adapter` writes to a `std::ostream`.
+- `output_string_adapter` appends to a string.
 
 ## Value conversion
 
+Values are converted from and to other types with the `JSONSerializer` template parameter. The default,
+[`adl_serializer`](../api/adl_serializer/index.md), calls the free functions
+
 ```cpp
 template<class T>
 void to_json(basic_json& j, const T& t);
@@ -112,13 +244,23 @@ template<class T>
 void from_json(const basic_json& j, T& t);
 ```
 
+found by argument-dependent lookup. The library defines them for standard types in `detail/conversions`; users add them
+for their own types, see [Arbitrary Type Conversions](../features/arbitrary_types.md). The
+[serialization macros](../features/macros.md) generate these functions.
+
 ## Additional features
 
-- JSON Pointers
-- Binary formats
-- Custom base class
-- Conversion macros
+- [JSON Pointer](../features/json_pointer.md) (class `json_pointer`) addresses values inside a tree. It is also the
+  basis of [JSON Patch](../features/json_patch.md).
+- [Binary formats](../features/binary_formats/index.md) are read by `binary_reader` and written by `binary_writer`.
+- A [custom base class](../api/basic_json/json_base_class_t.md) can add members to every [`basic_json`](../api/basic_json/index.md) value.
+- [Serialization macros](../features/macros.md) generate `to_json` and `from_json` functions for user-defined types.
 
 ## Details namespace
 
-- C++ feature backports
+Namespace `nlohmann::detail` contains all implementation details. It is not part of the public API and may change in any
+release. Besides the components above, it contains:
+
+- type traits to detect the capabilities of user-defined types (`detail/meta/type_traits.hpp`),
+- backports of C++14/17 features to C++11 (`detail/meta/cpp_future.hpp`), and
+- helpers such as `string_concat` and `string_escape`.

docs/mkdocs/mkdocs.yml

@@ -296,6 +296,7 @@ nav:
           - 'JSON_NOEXCEPTION': api/macros/json_noexception.md
           - 'JSON_NO_IO': api/macros/json_no_io.md
           - 'JSON_NO_THREAD_LOCAL': api/macros/json_no_thread_local.md
+          - 'JSON_PRECISE_STREAM_POSITION': api/macros/json_precise_stream_position.md
           - 'JSON_SKIP_LIBRARY_VERSION_CHECK': api/macros/json_skip_library_version_check.md
           - 'JSON_SKIP_UNSUPPORTED_COMPILER_CHECK': api/macros/json_skip_unsupported_compiler_check.md
           - 'JSON_STRICT_NUL_HANDLING': api/macros/json_strict_nul_handling.md
@@ -320,7 +321,9 @@ nav:
       - community/contribution_guidelines.md
       - community/quality_assurance.md
       - community/governance.md
+      - community/roadmap.md
       - community/security_policy.md
+      - community/assurance_case.md
 
 # Extras
 extra: