* Make JSON_STRICT_NUL_HANDLING part of the ABI tag
JSON_STRICT_NUL_HANDLING (#5534) changes the bodies of inline functions:
the lexer's handling of '\0' and input_adapter() for char arrays. So
translation units compiled with and without it define the same functions
differently, an ODR violation - the case the ABI tag exists for, as with
JSON_BRACE_INIT_COPY_SEMANTICS (_bics). It now appends _snul to the inline
namespace. The macro is new in 3.13.0, so no existing namespace changes.
Its default moves to abi_macros.hpp, and it is only #undef'd without
JSON_TEST_KEEP_MACROS, as for the other ABI macros. The ABI config tests,
the namespace docs, the macro's docs and the Natvis file cover the new tag.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Amalgamate
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: document that a NUL byte in the input is treated as end of input
A NUL byte anywhere in the input - trailing, or embedded ahead of more
otherwise well-formed JSON - is currently treated the same as genuine
end of input, so parsing silently stops there instead of raising the
parse_error.101 any other unexpected byte triggers. This mirrors the
NUL-terminated-C-string convention already used when no explicit input
length is given (json::parse(const char*) already stops at strlen()),
just applied uniformly rather than only when a length is genuinely
unavailable.
This behavior predates this change and is not being altered here -
changing it would be an observable, backwards-incompatible behavior
change for any caller that (knowingly or not) depends on it, which is
not something to do silently in a patch. Documenting the current,
verified behavior as a new FAQ entry instead, so it's an intentional
and discoverable part of the contract rather than a surprise.
Fixes#5530.
Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4RQ1Ahan5YAGbnAQGjZTY
* Add JSON_STRICT_NUL_HANDLING opt-in macro for issue #5530
A NUL byte anywhere in the input is currently treated the same as real
end of input, rather than raising parse_error.101 like any other
unexpected byte (documented in the previous commit's FAQ entry). A full
unconditional fix was tried in PR #5532 but rejected as too risky to
ship by default: any caller could depend on the current behavior, even
unknowingly (e.g. a zero-padded buffer). On PR #5534, gregmarr proposed
a compile-time opt-in flag instead, and the maintainer agreed, wanting
it available now and defaulting to the corrected behavior in 4.0.0.
This mirrors the existing JSON_BRACE_INIT_COPY_SEMANTICS precedent as
closely as sensible:
- JSON_STRICT_NUL_HANDLING defaults to 0 (off); the three lexer sites
that treat '\0' as EOF/comment-terminator are gated with
`#if !JSON_STRICT_NUL_HANDLING` so the default-off behavior is
byte-for-byte identical to today's.
- input_adapters.hpp's `T (&array)[N]` overload additionally trims a
single trailing '\0' from a `char` array (e.g. a string literal like
`json::parse("123")`) when the macro is on, so that case keeps
working; every other element type (unsigned char, std::uint8_t, ...)
always keeps its full extent. This intentionally does *not* reuse the
existing strlen()-based pointer overload via SFINAE-excluding `char`
from the array overload, as originally sketched for this change: that
approach is ambiguous against the newer generic container overload
added since PR #5532, and even where it compiles, strlen()-scanning a
`char` array that is not NUL-terminated within its bounds reads past
the end of the array (confirmed with AddressSanitizer). Trimming only
a single trailing byte, without scanning, avoids both problems.
- Documented via docs/mkdocs/docs/api/macros/json_strict_nul_handling.md,
linked from the macros index/nav/features page, the FAQ entry, and
the parse/accept/operator>> reference pages.
- Tested in unit-class_parser.cpp and unit-deserialization.cpp, default
state unguarded and opt-in state guarded. Since the library itself
#undefs the macro at the end of json.hpp (as JSON_BRACE_INIT_COPY_SEMANTICS
already does), a plain `#if defined(JSON_STRICT_NUL_HANDLING)` guard
after the include never actually triggers; the tests instead capture
the command-line value into a test-local macro before including the
header. A few pre-existing fixtures elsewhere (std::array<uint8_t, N>
sized one larger than their literal, relying on value-initialization
to silently add a trailing zero byte) needed the same one-byte
adjustment to keep passing under the opt-in behavior.
Unlike the precedent, this adds a proper `JSON_StrictNulHandling` CMake
option (rather than a raw -DCMAKE_CXX_FLAGS injection) and wires its
ci_test_strict_nul_handling target into the ci_cmake_options job matrix
in .github/workflows/ubuntu.yml, so the opt-in build is actually
exercised in CI -- closing the one gap in the precedent's own CI setup
(ci_test_brace_init_copy_semantics is defined but never referenced by
any workflow, so it has never actually run).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Clarify where JSON_STRICT_NUL_HANDLING does not reject NUL bytes
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The characterization test added in #5482 asserted that basic_json::swap()
does NOT exchange start_position/end_position, based on a misreading of
the code cited for #5420. In fact swap() (json.hpp, around line 3637)
does swap start_position/end_position along with the value, consistent
with copy-assignment. The test's assumption was backwards, so it failed
on every CI job across every branch/PR since the commit landed. Correct
the assertions to match the actual (and correct) behavior: positions are
exchanged together with values.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Add missing diagnostic-positions test coverage (lifetime, input adapters, SAX)
Building on the merged unit-class_parser.cpp from #5417, add
characterization tests (regression protection for existing behavior, not a
behavior change) for JSON_DIAGNOSTIC_POSITIONS:
- value lifetime: copy ctor copies positions recursively, move ctor resets
the moved-from value to npos, and mutating a parsed document (operator[],
push_back, erase) leaves the parent's stale span and siblings' positions
untouched while new values get npos.
- input adapters: wide-string input positions count transcoded UTF-8 bytes
(not wide characters), BOM-prefixed input's start_pos() reflects the
skipped 3-byte BOM, istringstream/ifstream/iterator-pair inputs report
consistent (non-npos) positions, and binary formats (CBOR, MessagePack,
UBJSON, BSON) always report npos.
- a user-constructed json_sax_dom_parser with no lexer (as used when driving
json::sax_parse() directly) reports npos for every value, since it has no
m_lexer_ref to source positions from.
While characterizing swap(), found that basic_json::swap() (and the friend
swap() that forwards to it) does not swap start_position/end_position,
unlike copy-assignment's operator=(basic_json), which does as part of its
copy-and-swap implementation. This looks like a real inconsistency/bug, but
per the scope of this test-only change it is only pinned (not fixed) here;
see the comment at the "swap() does NOT exchange positions" section.
Fixes#5420
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix MSVC source-encoding portability in the wide-string position test
Use é escapes instead of a literal UTF-8-encoded 'é' inside the L""
literal, so the wide string's content does not depend on the compiler's
assumed source character set (MSVC without /utf-8 decodes raw non-ASCII
source bytes using the system code page rather than as UTF-8, which was
producing a wstring of unexpected length/content and failing the
ws.size()/end_pos() assertions on Windows CI).
Also reworded a comment that unintentionally embedded the literal
substring "TODO check", which clang-tidy's google-readability-todo check
flags regardless of quoting context.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
basic_json::swap() (and the friend swap() that forwards to it) only
exchanged m_data.m_type/m_data.m_value, leaving start_position/end_position
untouched under JSON_DIAGNOSTIC_POSITIONS. This is inconsistent with
copy-assignment's operator=(basic_json), which swaps positions as part of
its copy-and-swap implementation, so after swap(a, b) each value ended up
with the other value's content but its own original position.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
When a parser callback rejects a value, the placeholder stored for it has to
be removed from its parent again. remove_discarded_value() found it by
scanning the parent from the beginning, so filtering a container cost one
scan per rejected member - quadratic in the number of members of a single
container.
A rejected value can only ever be the one most recently added to its parent:
the last element of an array, or the placeholder key() stored under the
current key in an object. Record that key alongside the existing
key_keep_stack, and for a container record it again alongside ref_stack so
end_object()/end_array() can find it in the parent. Removal is then O(1) for
an array and O(log n) for an object, and finding nothing there means nothing
was stored, so there is nothing to remove.
The key for a container is read before handle_value() may consume it, so it
is also correct when the callback rejects the container at its start event
and it never reaches its parent at all.
Discarding half the members of one object, before -> after:
members value rejected container rejected at start
16 000 392 ms -> 3.7 ms 803 ms -> 8.2 ms
64 000 6238 ms -> 14.6 ms 12651 ms -> 32.2 ms
128 000 25339 ms -> 30.6 ms
Results are unchanged: 48 000 randomized documents parsed under 12 different
filtering callbacks - covering duplicate keys, empty keys, rejected keys and
containers rejected at both their start and end events - produce byte
identical output before and after.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
tests/src/unit-class_parser_diagnostic_positions.cpp and
tests/src/unit-diagnostic-positions-only.cpp were maintained as near-copies
of unit-class_parser.cpp and unit-diagnostic-positions.cpp respectively, and
had drifted: trailing-comma handling, the #5342 filter-array/filter-value
sections, and the cross-input-adapter diagnostics test were never ported to
the positions-enabled copy.
Fold the position-specific assertions into the base files, guarded by
file a second time with the relevant macro set via CMake COMPILE_DEFINITIONS
(mirroring the existing test-comparison_legacy pattern) instead of
maintaining a separate source file. This removes the duplication and, as a
side effect, closes the coverage gaps above since the full test file now
compiles under JSON_DIAGNOSTIC_POSITIONS=1 as well.
Fixes#5417
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Broaden JSON_HEDLEY_WARN_UNUSED_RESULT coverage to pure query functions
Add JSON_HEDLEY_WARN_UNUSED_RESULT to the unambiguous, const,
side-effect-free observer functions whose return value is the entire
purpose of the call:
- dump()
- type(), type_name()
- all is_* predicates (is_primitive, is_structured, is_null,
is_boolean, is_number, is_number_integer, is_number_unsigned,
is_number_float, is_object, is_array, is_string, is_binary,
is_discarded)
- empty(), size(), max_size()
- count(...) (both overloads) and contains(...) (all overloads,
including the deprecated json_pointer<BasicJsonType> overload)
This mirrors the direction the standard library has taken with
[[nodiscard]] on the analogous std::vector/std::map members, and
catches real bugs such as `j.empty();` (meant `j.clear();`) or
`j.contains(k);` with the result thrown away.
Deliberately out of scope (left for a separate, later policy
decision, per the issue): at(), value(), get*(), flatten(),
unflatten(), patch(), merge_patch(), begin()/end(), comparison
operators, erase(), and emplace().
Compiling the full test suite (tests/src/unit-*.cpp) with
-Wunused-result -Werror uncovered one real hit: a regression test in
unit-regression2.cpp called dump() purely to check it does not throw,
discarding the result. Fixed by explicitly casting to void, since the
call is intentionally result-less there.
Fixes#5410
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix discarded nodiscard results across the test suite for GCC's warn_unused_result
A plain (void) cast on a call expression suppresses the C++17 [[nodiscard]]
warning but not GCC's warning for functions annotated via the GNU
__attribute__((warn_unused_result)) form -- which is what
JSON_HEDLEY_WARN_UNUSED_RESULT expands to on GCC. Several existing tests
that call a newly-annotated function (dump(), empty()) purely to check
that it throws/does not throw, discarding the result via (void), newly
warned (and failed -Werror builds) once the annotation was broadened.
Route those discards through a small ignore_return_value() helper
instead, which actually consumes the value and suppresses the warning
on both attribute forms.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use utils::ignore_return_value() for the issue #1445 dump() discard too
Addresses review feedback from @gregmarr on PR #5477: this call site was
still using the older "capture in a variable, then (void) it" pattern
from before this PR introduced utils::ignore_return_value(), instead of
the helper now used at every other discarded-nodiscard-result call site
this PR touches.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Speed up whitespace skipping in the lexer
lexer::skip_whitespace() called get() for every whitespace byte, and
get() checks the (almost always false, once past the first character)
next_unget flag on every call. skip_whitespace() now reads its first
character with get() (needed to honor a pending unget() left over from
finishing the previous token, e.g. scan_number() always ungets the
character that terminated the number) and every further whitespace
character with a new get_ignoring_pending_unget() variant that skips
that branch, since nothing in the loop calls unget().
This is a narrower fix than the full contiguous-buffer bulk-skip
suggested in the issue (scan a run of whitespace directly in the
adapter's buffer and update position counters once per run). That
approach depends on bulk-scan adapter infrastructure
(supports_bulk_scan/bulk_data()/bulk_skip()) introduced by the open,
unmerged parser-performance PR #5283, which this change intentionally
does not depend on or replicate. Building new bulk-scan adapter
infrastructure from scratch was judged out of scope/riskier than
warranted here, so this change is limited to the safe, always-correct
improvement of removing redundant per-character bookkeeping from the
existing byte-at-a-time loop; full bulk-skipping is left as future
work once #5283 (or equivalent adapter support) lands.
Line/column/byte-offset bookkeeping is untouched and verified
bit-for-bit identical before and after this change, including for
pretty-printed (dump(4)) input with embedded newlines.
Fixes#5412
Stacked on top of the PR for #5411 (branch
issue-5411-lexer-skip-conversion).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix codegen regression in skip_whitespace() from #5490
Benchmarking found the get()/get_ignoring_pending_unget() split in
skip_whitespace() made long whitespace runs (e.g. indentation in
pretty-printed JSON) 1.75x-3.2x SLOWER instead of faster, reproducible
with both Apple Clang and GCC.
Root cause: rewriting the loop from a plain do-while into an initial
get() followed by a while-loop defeated the compiler's ability to keep
the input adapter's read/end pointers in registers across iterations;
both compilers instead reloaded them from memory on every character.
The function split itself was not the problem (it still fully
inlines); the loop's control-flow shape was.
The fix keeps the same two-function structure but restores a
do-while shape (guarded by an if for the "first char not whitespace"
case), which lets both compilers hoist the pointers back into
registers, matching or beating pre-#5490 performance.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Share the position-counter bump between get() and get_ignoring_pending_unget()
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Extract current_is_whitespace() to deduplicate skip_whitespace()'s two whitespace checks
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use a raw string literal for the multi-line error-position test input
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy raw-string-literal finding and guard a new test against JSON_NOEXCEPTION
The issue #5412 whitespace-skipping test added a check_error() helper that
relies on catching json::parse_error to verify the exception message; under
JSON_NOEXCEPTION, JSON_THROW aborts instead of throwing, which crashed
ci_test_noexceptions (and cascaded into the other ci_cmake_options jobs).
Guard the whole section with #if !defined(JSON_NOEXCEPTION), matching the
existing pattern used by sibling tests in this file.
Also switch one escaped string literal to a raw string literal to satisfy
clang-tidy's modernize-raw-string-literal check.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip integer conversion in accept()/SAX validation when the value is unused
lexer::scan_number() always converted every numeric token with
strtoull()/strtoll() before returning, even though accept() (and any
consumer using json_sax_acceptor) immediately discards the converted
value. For value_unsigned/value_integer tokens whose digit count
already guarantees the value fits into 64 bits, the conversion cannot
change the accept/reject decision (such tokens are always finite and
unconditionally accepted), so scan_number() can skip strtoull()/
strtoll() entirely in that case when the caller signals it does not
need the value. Numbers with more digits keep using the exact,
unmodified conversion path, so overflow reclassification to
value_float (and the finiteness check on it) is unaffected.
parse() and value_float handling are completely unchanged.
Fixes#5411
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document why the digit-count fast path is safe regardless of number_unsigned_t/number_integer_t width
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid temporary-string concatenation flagged by clang-tidy in the differential test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
`lexer::get()` copied every scanned character into `token_string` on the
whole successful-parse hot path, yet that buffer is consumed only by
`get_token_string()` when rendering the "last read" fragment of a parse
error. On well-formed input the per-byte copy (plus the `unget()` pop)
is pure overhead that is always discarded.
For seekable input adapters - random-access, single-byte iterators such
as those backing `std::string`, `const char*`, and `std::vector<char>` -
the offending token is now reconstructed on demand from the input when
an error is reported, using a saved start offset, and the eager copy is
skipped. Streaming adapters (file, istream, wide-string, and user-defined
adapters) keep the eager copy; the strategy is chosen at compile time via
`input_adapter_supports_seek`, so adapters without the capability are
unaffected.
Error messages are byte-for-byte identical across all adapters, verified
by a new parity regression test. Microbenchmark (4 MB mixed JSON, parsed
from a std::string): ~149 -> ~160 MB/s, about +8%.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
PR #4873 introduced a safety check in sax_parse functions to catch
nullptr passed as SAX parser object, which had been already annotated by
JSON_HEDLEY_NON_NULL macro.
Compilers (e.g. clang) which respected the non-null annotation tended to
eliminate the safety check completely in optimized builds, while
compilers which did not, compiled the safety check in. This led to
different behaviors accross different compilers/platforms and/or build
types (debug, release).
This commit reverts PR #4873 to remove this discrepancy. Passing null to
non-null annotated parameter is considered to be undefined behavior.
Fixes#5048
Signed-off-by: Richard Musil <risa2000x@gmail.com>
Co-authored-by: Richard Musil <risa2000x@gmail.com>
* Add implementation to retrieve start and end positions of json during parse
* Add more unit tests and add start/stop parsing for arrays
* Add raw value for all types
* Add more tests and fix compiler warning
* Amalgamate
* Fix CLang GCC warnings
* Fix error in build
* Style using astyle 3.1
* Fix whitespace changes
* revert
* more whitespace reverts
* Address PR comments
* Fix failing issues
* More whitespace reverts
* Address remaining PR comments
* Address comments
* Switch to using custom base class instead of default basic_json
* Adding a basic using for a json using the new base class. Also address PR comments and fix CI failures
* Address decltype comments
* Diagnostic positions macro (#4)
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Fix missed include deletion
* Add docs and address other PR comments (#5)
* Add docs and address other PR comments
---------
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Address new PR comments and fix CI tests for documentation
* Update documentation based on feedback (#6)
---------
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Address std::size_t and other comments
* Fix new CI issues
* Fix lcov
* Improve lcov case with update to handle_diagnostic_positions call for discarded values
* Fix indentation of LCOV_EXCL_STOP comments
* fix amalgamation astyle issue
---------
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Move UDLs into nlohmann::literals::json_literals namespace
* Add 'using namespace' to unit tests
* Add 'using namespace' to examples
* Add 'using namespace' to README
* Move UDL mkdocs pages out of basic_json/
* Update documentation
* Update docset index
* Add JSON_GlobalUDLs CMake option
* Add unit test
* Build examples without global UDLs
* Add CI target
* Add C++20 3-way comparison operator and fix broken comparisons
Fixes#3207.
Fixes#3409.
* Fix iterators to meet (more) std::ranges requirements
Fixes#3130.
Related discussion: #3408
* Add note about CMake standard version selection to unit tests
Document how CMake chooses which C++ standard version to use when
building tests.
* Update documentation
* CI: add legacy discarded value comparison
* Fix internal linkage errors when building a module
Niels Lohmann
Angadi56
risa2000andRichard Musil
Nikhil
chirszandNiels Lohmann
Sushrut ShringarputaleandSush Shringarputale
laterlaugh
