Read BON8 strings in bulk from contiguous input

- copy the valid UTF-8 of a string in one step when the input is
  contiguous (twitter.json is read in 1.68 instead of 2.52 ms,
  jeopardy.json in 196 instead of 297 ms, close to CBOR and MessagePack)
- share the new valid_utf8_prefix() with the writer's UTF-8 check, which
  now skips ASCII 8 bytes at a time
- let the fuzzer check that contiguous and stream input give the same
  value or error, and test both paths in the unit tests
- clarify that a second 0xFF after a string is an empty string

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann
2026-09-25 22:06:54 +02:00
parent 31db00f2b4
commit 43a346cf99
7 changed files with 269 additions and 35 deletions
@@ -137,8 +137,8 @@ Non-negative integers are read as number_unsigned, negative integers as number_i
!!! info
Values that do not use the canonical representation, such as integers with a longer encoding than necessary,
arrays and objects with up to four elements that are terminated by 0xFE, unsorted object keys, or unneeded 0xFF
bytes after a string, are accepted.
arrays and objects with up to four elements that are terminated by 0xFE, unsorted object keys, or a 0xFF after a
string that would also end without it, are accepted. A second 0xFF is not a terminator but an empty string.
Strings must be valid UTF-8, and the last string of a message must be terminated by 0xFF.