Files
json/docs/mkdocs/docs/api/basic_json/sax_parse.md
T
Niels Lohmann abbe52d6de Add JSON_PRECISE_STREAM_POSITION to leave the character that terminates a number in the stream (#5344)
* docs: qualify the operator>> stream positioning guarantee

operator>>'s notes state that it leaves the stream positioned right
after the parsed value, so that concatenated JSON values can be read
back to back. That does not hold when the value is a number: a number
is only terminated by the character that follows it, and the lexer's
unget() is simulated (it rewinds only the lexer's own bookkeeping),
so that character stays consumed from the stream.

Document the actual behaviour: the guarantee holds for all value types
except numbers, which must be followed by whitespace. Also qualify the
cross-reference on the JSON Lines page, which repeated the unqualified
claim.

Documentation only; the behaviour itself is tracked in #5340.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* fix: restore the character that terminates a number (#5340)

operator>> is documented to leave the stream positioned right after the
parsed value, so that concatenated JSON values can be read back to back.
That did not hold for numbers: a number is only terminated by the
character following it, and lexer::scan_number() reads that character
and calls unget() -- which is simulated and rewinds only the lexer's own
bookkeeping. input_stream_adapter consumes via sbumpc() with no matching
sungetc(), so the terminating character stayed consumed and the next
extraction started one byte too late ('1true' left the stream at 'rue').

Propagating unget() to the adapter directly does not work: next_unget
makes the following get() replay the cached character, so the terminator
would be delivered twice. Instead, restore the still-pending character
once at the end of a non-strict parse, where the input is handed back to
the caller:

- input_stream_adapter gains unget_character() (sungetc()) and advertises
  it via supports_unget, detected the same way as supports_seek.
- lexer::restore_pending_unget() turns a pending simulated unget of a
  real (non-EOF) character into a real one and clears next_unget so the
  character is not also replayed. It is a no-op for adapters that cannot
  unget, and reports failure when sungetc() fails, in which case the
  input is left as it was before.
- parser calls it on the three non-strict paths, i.e. for operator>> and
  sax_parse(strict = false).

Strict parse()/accept() are unaffected: they require the input to end
after the value, so the character is consumed by the end-of-input check
anyway. Parse error messages and reported positions are unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* tests: fix CI failures in the #5340 test helpers

Four CI failures, all in the new test code:

- GCC (-Werror=useless-cast): drop the `json(...)` wrapper around
  `json::parse(...)`, which already returns a `json`.
- GCC (-Werror=unused-result): assign the discarded `json::parse()`
  result to a dummy, the idiom used elsewhere in the test suite, and
  catch `json::parse_error&` for consistency.
- clang-tidy (google-default-arguments): remove the default argument
  from the `pbackfail()` override; `sungetc()` supplies the base
  declaration's default.
- MSVC (bad allocation): `no_putback_streambuf::underflow()` set a
  one-character get area without advancing `m_pos`, so an implementation
  whose `istream::get` peeks before it bumps re-read the same character
  forever. Keep no get area at all: `underflow()` peeks, `uflow()`
  consumes, and `sungetc()` still always lands in `pbackfail()`, which
  is what the test needs.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* fix: leave the character that terminates a number in the input

Read the character following a number without consuming it, instead of
consuming it and putting it back. input_stream_adapter now peeks with
sgetc() and only steps over the character when the next one is requested
or when the adapter is destroyed, so releasing it cannot fail - no
putback position is required from the streambuf.

Suggested by gregmarr in #5344.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: match the version history wording to the peek-based fix

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* docs: drop the whitespace-separator caveat from the parsing pages

The caveat added in #5343 describes the behavior this branch fixes: a
number no longer consumes the character that terminates it, so
concatenated values need no separator.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* refactor: split the strict and non-strict paths in parser

Folding the release_lookahead() call into the existing strict check left
the "in strict mode" comment on an else-if branch, and made the strict
condition in sax_parse() redundant with the branch it followed.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Put the stream position fix behind JSON_PRECISE_STREAM_POSITION

Leaving the character that terminates a number in the stream is observable:
reading "1,2,3" with repeated operator>> works today only because the comma
after each number is swallowed, and std::getline after a number skips the
line break. Both break with the fix, so make it opt-in for 3.x, as suggested
by @gregmarr in the review.

- JSON_PRECISE_STREAM_POSITION (default 0) selects the peek-based
  input_stream_adapter. Without it, the adapter is the consuming one from
  develop and has no supports_lookahead, so lexer::release_lookahead() and
  the parser's calls to it compile to nothing.
- The macro changes input_stream_adapter's layout and member functions, so
  it gets the ABI tag _psp, after _bics. The ABI config tests, the natvis
  generator, and nlohmann_json.natvis (regenerated) know the tag.
- The tests for the fix move to unit-precise-stream-position.cpp, which
  defines the macro itself and runs in every build, and gain the two cases
  above. unit-deserialization.cpp pins the default behavior instead.
- The docs describe the default behavior again and point to the new macro
  page; version history says "added in 3.13.0, planned default in 4.0.0".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-25 17:56:18 +02:00

5.6 KiB

nlohmann::basic_json::sax_parse

// (1)
template <typename InputType, typename SAX>
static bool sax_parse(InputType&& i,
                      SAX* sax,
                      input_format_t format = input_format_t::json,
                      const bool strict = true,
                      const bool ignore_comments = false,
                      const bool ignore_trailing_commas = false);

// (2)
template<class IteratorType, class SAX, class SentinelType = IteratorType>
static bool sax_parse(IteratorType first, SentinelType last,
                      SAX* sax,
                      input_format_t format = input_format_t::json,
                      const bool strict = true,
                      const bool ignore_comments = false,
                      const bool ignore_trailing_commas = false);

Read from input and generate SAX events

  1. Read from a compatible input.

  2. Read from a pair of character iterators, or an iterator and a sentinel of a different type (C++20 ranges support)

    The value_type of the iterator must be an integral type with a size of 1, 2, or 4 bytes, which will be interpreted respectively as UTF-8, UTF-16, and UTF-32. If SentinelType differs from IteratorType, it must be comparable to the iterator type with operator!=.

The SAX event lister must follow the interface of json_sax.

Template parameters

InputType
A compatible input, for instance:
  • an std::istream object
  • a FILE pointer
  • a C-style array of characters
  • a pointer to a null-terminated string of single byte characters
  • a container obj for which begin(obj) and end(obj) produce a valid pair of iterators (as found via ADL or member functions, with semantics compatible to std::begin and std::end)
IteratorType
a compatible iterator type for overload (2); a pair of character iterators whose value_type is an integral type with a size of 1, 2, or 4 bytes (interpreted respectively as UTF-8, UTF-16, and UTF-32)
SentinelType
defaults to IteratorType; may be a different type comparable to IteratorType via operator!=, for overload (2), for instance.
  • a custom sentinel type for C++20 ranges
  • std::default_sentinel_t, when IteratorType is std::counted_iterator
SAX
a class fulfilling the SAX event listener interface; see json_sax

Parameters

i (in)
Input to parse from
sax (in)
SAX event listener (must not be null)
format (in)
the format to parse (JSON, CBOR, MessagePack, or UBJSON) (optional, input_format_t::json by default), see input_format_t for more information
strict (in)
whether the input has to be consumed completely (optional, #!cpp true by default); when #!cpp false and the input is a #!cpp std::istream, the character that terminates a number is consumed unless JSON_PRECISE_STREAM_POSITION is defined to 1; see operator>>
ignore_comments (in)
whether comments should be ignored and treated like whitespace (#!cpp true) or yield a parse error (#!cpp false); (optional, #!cpp false by default)
ignore_trailing_commas (in)
whether trailing commas in arrays or objects should be ignored and treated like whitespace (#!cpp true) or yield a parse error (#!cpp false); (optional, #!cpp false by default)
first (in)
iterator to the start of a character range
last (in)
iterator to the end of a character range, or a sentinel value that compares equal to the end iterator with operator!=

Return value

return value of the last processed SAX event

Exception safety

Strong guarantee: if an exception is thrown, there are no changes in the JSON value.

Exceptions

  • Throws parse_error.101 in case of an unexpected token, or empty input like a null FILE* or char* pointer.

Complexity

Linear in the length of the input. The parser is a predictive LL(1) parser. The complexity can be higher if the SAX consumer sax has a super-linear complexity.

Notes

A UTF-8 byte order mark is silently ignored.

Examples

??? example

The example below demonstrates the `sax_parse()` function reading from string and processing the events with a
user-defined SAX event consumer.

```cpp
--8<-- "examples/sax_parse.cpp"
```

Output:

```json
--8<-- "examples/sax_parse.output"
```

See also

  • parse - deserialize from a compatible input
  • accept - check if the input is valid JSON

Version history

  • Added in version 3.2.0.
  • Ignoring comments via ignore_comments added in version 3.9.0.
  • Added ignore_trailing_commas in version 3.13.0.
  • Extended container support (1) to include types with lvalue-only ADL begin/end (matching std::begin/std::end semantics) in version 3.13.0.
  • Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
  • JSON_PRECISE_STREAM_POSITION added in version 3.13.0 to optionally leave a #!cpp std::istream positioned right after the parsed value when strict is #!cpp false.

!!! warning "Deprecation"

Overload (2) replaces calls to `sax_parse` with a pair of iterators as their first parameter which has been
deprecated in version 3.8.0. This overload will be removed in version 4.0.0. Please replace all calls like
`#!cpp sax_parse({ptr, ptr+len});` with `#!cpp sax_parse(ptr, ptr+len);`.