mirror of
https://github.com/nlohmann/json.git
synced 2026-11-11 18:08:22 +00:00
Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Niels Lohmann
|
44b1e7b2b9 | ||
|
Niels Lohmann
|
71ec3b4306 |
No files matched your search
@@ -89,8 +89,10 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
|
||||
- Throws [out_of_range.406](../../home/exceptions.md#jsonexceptionout_of_range406) if a high-precision number, or a
|
||||
float64 when `number_float_t` is `float`, does not fit into `number_float_t`
|
||||
- Throws [out_of_range.408](../../home/exceptions.md#jsonexceptionout_of_range408) if the count of an optimized
|
||||
container exceeds the capacity of the container (or of `std::size_t`), or if an optimized array of `Z`, `T`, or `F` has
|
||||
more than 1,048,576 elements
|
||||
container exceeds the capacity of the container (or of `std::size_t`; an array of type `N` is empty, so only the
|
||||
latter applies to it), if an optimized array of `Z`, `T`, or `F` has more than 1,048,576 elements, or if the
|
||||
optimized arrays of `Z`, `T`, and `F` in the input together have more than 1,048,576 elements plus 4 for every byte
|
||||
read so far
|
||||
|
||||
## Complexity
|
||||
|
||||
@@ -132,6 +134,9 @@ Linear in the size of the input.
|
||||
- Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
|
||||
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
|
||||
- Added `error_handler` parameter in version 3.13.0.
|
||||
- Report `start_array(0)` to a SAX handler for an optimized array of type `N` (`[$N#n`), whatever its count, in version
|
||||
3.13.0.
|
||||
- Limited the total number of elements of optimized arrays of `Z`, `T`, and `F` in one input in version 3.13.0.
|
||||
|
||||
!!! warning "Deprecation"
|
||||
|
||||
|
||||
@@ -117,5 +117,8 @@ Linear in the size of the JSON value `j`.
|
||||
- Added `error_handler` parameter in version 3.13.0. Its default, `keep`, writes the bytes of a string or object key
|
||||
that is not valid UTF-8 unchanged, as before; `strict` (the default if
|
||||
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316`.
|
||||
- Writes arrays of `Z`, `T`, or `F` without the `$` type annotation (`[#n Z Z ...`) once their elements in one call
|
||||
add up to more than 1,048,576, in version 3.13.0, so that `from_ubjson` can read the output back. Such output is
|
||||
larger: 270,000 rows of 100 `null` values take 26.8 MB instead of 1.35 MB.
|
||||
- Throws `type_error.321` for a discarded value since version 3.13.0; previously, a discarded value nested in an
|
||||
array or object was silently skipped, producing invalid UBJSON.
|
||||
@@ -122,9 +122,18 @@ The library uses the following mapping from JSON values types to UBJSON types ac
|
||||
already is the value. Its declared count is therefore the only thing that decides how much memory the receiving side
|
||||
allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
|
||||
[`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) when the count exceeds 1,048,576
|
||||
(`1 << 20`), and `to_ubjson` writes longer arrays of these types in the counted but untyped form: the `$` type marker is
|
||||
dropped, the `#` count is kept, and every element is written with its own marker (`[#l 1048577 Z Z ...`), so any value
|
||||
it produces can be read back. Objects and untyped counted arrays are not subject to this limit.
|
||||
(`1 << 20`), or when the counts of all such arrays in the input add up to more than 1,048,576 plus 4 for every byte
|
||||
read so far. `to_ubjson` therefore writes an array of these types in the counted but untyped form when it is longer
|
||||
than that, or when it would bring the total of the typed arrays written so far above 1,048,576: the `$` type marker
|
||||
is dropped, the `#` count is kept, and every element is written with its own marker (`[#l 1048577 Z Z ...`), so
|
||||
everything it produces can be read back. Only input from another encoder, or written by version 3.12.0 and earlier of
|
||||
this library, can exceed the limit: for example, 1,100 rows of 1,000 `null` values or 210,000 rows of 25, each with
|
||||
a `$Z` annotation. Version 3.12.0 writes such a row in 5 bytes (6 above 255 elements), so rows of 20 or fewer such
|
||||
values are never rejected. Objects and untyped counted arrays are not subject to this limit.
|
||||
|
||||
An optimized array of type `N` (no-op) is empty whatever its count is, because a no-op is not a value. Its count is
|
||||
not passed on: `sax_parse` reports `start_array(0)` (versions before 3.13.0 reported the declared count), and the DOM
|
||||
parser reserves nothing for it.
|
||||
|
||||
!!! info "Binary values"
|
||||
|
||||
@@ -180,10 +189,11 @@ The library maps UBJSON types to JSON value types as follows:
|
||||
|
||||
Every well-formed UBJSON value can be converted to a JSON value, with these exceptions: a high-precision number
|
||||
outside the range of `number_float_t` ([`out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406)),
|
||||
an optimized array of `Z`, `T`, or `F` with more than 1,048,576 elements, and a count that exceeds the capacity of the
|
||||
container ([`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408)). High-precision numbers that
|
||||
do not fit an integer type are stored as floating-point numbers and may lose precision. Ill-formed input is rejected
|
||||
with a parse error.
|
||||
optimized arrays of `Z`, `T`, or `F` with more elements than the limit allows (1,048,576 plus 4 per byte read),
|
||||
and a count that exceeds the capacity of the container
|
||||
([`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408)). High-precision numbers that do not
|
||||
fit an integer type are stored as floating-point numbers and may lose precision. Ill-formed input is rejected with a
|
||||
parse error.
|
||||
|
||||
!!! info "Duplicate object keys"
|
||||
|
||||
|
||||
@@ -72,10 +72,22 @@ For the binary formats, an announced size is never trusted outright:
|
||||
(reported as a parse error) instead of triggering an oversized allocation.
|
||||
- When an array announces its number of elements and the array container supports `reserve()` (as `#!cpp std::vector`,
|
||||
the default, does), the library reserves storage for at most 16384 of them upfront, regardless of how large the
|
||||
announced count is; further elements still grow the container normally as they are read.
|
||||
announced count is; further elements still grow the container normally as they are read. A count that no
|
||||
input backs, as in a truncated input, reserves at most that much until the parse error is reported. A parse that
|
||||
completes keeps storage for such a count only where the elements take no input at all, which the two
|
||||
[UBJSON](../binary_formats/ubjson.md) cases below cover.
|
||||
- An announced array or object size that exceeds what the target container could ever hold (its `max_size()`) is
|
||||
rejected immediately as [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408), without
|
||||
attempting to allocate anything.
|
||||
- A [UBJSON](../binary_formats/ubjson.md) array of the no-op type (`[$N#<n>`) is empty whatever count it declares: a
|
||||
SAX handler is told `start_array(0)` and the DOM parser reserves nothing.
|
||||
- A UBJSON array of type `Z`, `T` or `F` has no payload, so its announced count is not bounded by the input. One array
|
||||
may announce at most 1,048,576 elements, and all such arrays of one input together may contain at most 1,048,576
|
||||
elements plus 4 for every byte read so far (checked when each array's header has been read); beyond that the input is
|
||||
rejected as [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408). For a DOM that bounds the memory these
|
||||
arrays can need to about 80 bytes per byte of input plus about 33 MB for the first 1,048,576 elements; a SAX handler
|
||||
receives at most that many events. Every other optimized array element, and every UBJSON object member (it has a
|
||||
key), takes at least one byte of input.
|
||||
|
||||
## Strings
|
||||
|
||||
|
||||
@@ -1050,11 +1050,17 @@ The size of an array or object in a [binary format](../features/binary_formats/i
|
||||
the size following `#` for [UBJSON](../features/binary_formats/ubjson.md)/[BJData](../features/binary_formats/bjdata.md),
|
||||
or the encoded length for [CBOR](../features/binary_formats/cbor.md).
|
||||
|
||||
The exception is also thrown for a [UBJSON](../features/binary_formats/ubjson.md) array of a type that is encoded by its
|
||||
marker alone (`Z`, `T` or `F`) whose declared count exceeds 1,048,576 (`1 << 20`). Such an array has no payload, so its
|
||||
count alone decides how much memory is allocated, and a handful of bytes would otherwise describe billions of values.
|
||||
[`to_ubjson`](../api/basic_json/to_ubjson.md) writes longer arrays of these types without the `$` type annotation (the `#`
|
||||
count and one marker per element are kept), so any value it produces can still be read back.
|
||||
The exception is also thrown for [UBJSON](../features/binary_formats/ubjson.md) arrays of a type that is encoded by
|
||||
its marker alone (`Z`, `T` or `F`). Such an array has no payload, so its count alone decides how much memory is
|
||||
allocated, and a handful of bytes would otherwise describe billions of values. It is thrown if the declared count of one
|
||||
array exceeds 1,048,576 (`1 << 20`), and if the counts of all such arrays read so far in one input add up to more than
|
||||
1,048,576 plus 4 for every byte of the input read when the header of the last array has been read.
|
||||
[`to_ubjson`](../api/basic_json/to_ubjson.md) writes an array of these types without the `$` type annotation (the `#`
|
||||
count and one marker per element are kept) when it is longer, or when it would bring the total of the typed arrays
|
||||
written so far above 1,048,576, so everything it produces can be read back. Only input written by another encoder, or
|
||||
by version 3.12.0 and earlier of this library, can exceed the limit: for example, 1,100 rows of 1,000 `null` values or
|
||||
210,000 rows of 25, each with a `$Z` annotation. Version 3.12.0 writes such a row in 5 bytes (6 above 255 elements),
|
||||
so rows of 20 or fewer such values are never rejected.
|
||||
|
||||
!!! failure "Example messages"
|
||||
|
||||
@@ -1070,6 +1076,9 @@ count and one marker per element are kept), so any value it produces can still b
|
||||
```
|
||||
[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size
|
||||
```
|
||||
```
|
||||
[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive number of elements without payload
|
||||
```
|
||||
|
||||
### json.exception.out_of_range.409
|
||||
|
||||
|
||||
@@ -73,13 +73,46 @@ the input.
|
||||
|
||||
This is a sanity bound rather than a security boundary, and it is far above
|
||||
any container met in practice. @ref binary_writer falls back to the
|
||||
unoptimized encoding for longer containers, so that a value serialized by
|
||||
this library can always be read back.
|
||||
unoptimized encoding for longer containers, and for containers that together
|
||||
with the ones before them would exceed @ref valueless_elements_base, so that a
|
||||
value serialized by this library can always be read back.
|
||||
|
||||
@sa https://github.com/nlohmann/json/issues/2793
|
||||
*/
|
||||
JSON_INLINE_VARIABLE constexpr std::size_t max_valueless_container_size = 1 << 20;
|
||||
|
||||
/*!
|
||||
@brief number of elements of valueless UBJSON containers accepted on top of
|
||||
@ref valueless_elements_per_byte
|
||||
|
||||
The limit @ref max_valueless_container_size applies to each container on its
|
||||
own, so on its own it lets 9 bytes announce 1048576 values, and every further
|
||||
9 bytes another 1048576. The elements of all containers of a valueless type
|
||||
in one input are therefore added up, and the sum must stay below this base
|
||||
plus @ref valueless_elements_per_byte for every byte read so far. The base is
|
||||
the per-container limit, so a container that was accepted on its own before
|
||||
the sum was introduced is still accepted at the start of an input.
|
||||
*/
|
||||
JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_base = max_valueless_container_size;
|
||||
|
||||
/*!
|
||||
@brief number of elements of valueless UBJSON containers accepted per byte of
|
||||
input, in addition to @ref valueless_elements_base
|
||||
|
||||
An element of a value that has a payload costs at least one byte of input.
|
||||
Valueless elements are accepted at a rate of 4 per byte. A value takes 16
|
||||
bytes of memory in a DOM, so these arrays add about 80 bytes of memory per
|
||||
byte of input (measured: 77; one `Z` per byte needs about 32), beyond what the
|
||||
same bytes cost anyway, plus the base. This bounds the arrays of valueless
|
||||
type, not the memory of a whole document. Containers written by
|
||||
@ref binary_writer never need more than the base, because it writes an array
|
||||
in the unoptimized form once the typed arrays written so far and this one add
|
||||
up to more than that. The rate matters for input written by other encoders and
|
||||
by versions before 3.13.0 only; in the data this was measured on, no input has
|
||||
more than 0.17 elements per byte.
|
||||
*/
|
||||
JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_per_byte = 4;
|
||||
|
||||
///////////////////
|
||||
// binary reader //
|
||||
///////////////////
|
||||
@@ -3700,6 +3733,37 @@ class binary_reader
|
||||
return emit_unsigned(number);
|
||||
}
|
||||
|
||||
/*!
|
||||
@brief add the elements of a container of a valueless type to the total of
|
||||
the input and check the total against the input read so far
|
||||
|
||||
A container of type 'Z', 'T' or 'F' has no payload, so its elements are the
|
||||
only ones for which the input does not bound the count. The check is made
|
||||
once per container, not per element.
|
||||
|
||||
@param[in] count number of elements the container declares; at most
|
||||
@ref max_valueless_container_size
|
||||
@return false if the total is more than @ref valueless_elements_base plus
|
||||
@ref valueless_elements_per_byte for every byte read so far; the
|
||||
total is not changed then
|
||||
*/
|
||||
bool charge_valueless_elements(const std::size_t count) noexcept
|
||||
{
|
||||
// the limit saturates instead of overflowing when size_t is 32 bits
|
||||
// wide and a very large input has been read; valueless_elements
|
||||
// never exceeds the limit, so the difference cannot underflow
|
||||
constexpr std::size_t max_size = (std::numeric_limits<std::size_t>::max)();
|
||||
const std::size_t limit = chars_read <= (max_size - valueless_elements_base) / valueless_elements_per_byte
|
||||
? valueless_elements_base + (valueless_elements_per_byte * chars_read)
|
||||
: max_size;
|
||||
if (JSON_HEDLEY_UNLIKELY(count > limit - valueless_elements))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
valueless_elements += count;
|
||||
return true;
|
||||
}
|
||||
|
||||
/*!
|
||||
@return whether array creation completed
|
||||
*/
|
||||
@@ -3785,24 +3849,27 @@ class binary_reader
|
||||
// declared count alone decides how much is allocated; the check is
|
||||
// made before the start event so that no container is opened that
|
||||
// is then abandoned. See @ref max_valueless_container_size.
|
||||
if (JSON_HEDLEY_UNLIKELY((size_and_type.second == 'Z' || size_and_type.second == 'T' || size_and_type.second == 'F')
|
||||
&& size_and_type.first > max_valueless_container_size))
|
||||
if (JSON_HEDLEY_UNLIKELY(size_and_type.second == 'Z' || size_and_type.second == 'T' || size_and_type.second == 'F'))
|
||||
{
|
||||
return out_of_range_error(exception_id::container_too_large, "excessive array size", "size");
|
||||
if (JSON_HEDLEY_UNLIKELY(size_and_type.first > max_valueless_container_size))
|
||||
{
|
||||
return out_of_range_error(exception_id::container_too_large, "excessive array size", "size");
|
||||
}
|
||||
if (JSON_HEDLEY_UNLIKELY(!charge_valueless_elements(size_and_type.first)))
|
||||
{
|
||||
return out_of_range_error(exception_id::container_too_large, "excessive number of elements without payload", "size");
|
||||
}
|
||||
}
|
||||
|
||||
if (JSON_HEDLEY_UNLIKELY(!enter_array(size_and_type.first, size_and_type.second)))
|
||||
// a no-op is not a value, so an array of them is empty whatever
|
||||
// count it declares: the SAX parser is told so, instead of being
|
||||
// made to reserve storage for a count that no input backs
|
||||
const std::size_t announced = size_and_type.second == 'N' ? 0 : size_and_type.first;
|
||||
if (JSON_HEDLEY_UNLIKELY(!enter_array(announced, size_and_type.second)))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
if (size_and_type.second == 'N')
|
||||
{
|
||||
// a no-op is not a value, so a container of them holds none;
|
||||
// the declared size has already been passed to the SAX parser
|
||||
container_stack.back().remaining = 0;
|
||||
}
|
||||
|
||||
return size_and_type.second == 0
|
||||
|| parse_ubjson_typed_run(container_stack.back().remaining, size_and_type.second, std::integral_constant<bool, bulk_scan> {});
|
||||
}
|
||||
@@ -5114,6 +5181,10 @@ class binary_reader
|
||||
/// the number of characters read
|
||||
std::size_t chars_read = 0;
|
||||
|
||||
/// UBJSON: the sum of the element counts of the containers of a valueless
|
||||
/// type read so far; see @ref charge_valueless_elements
|
||||
std::size_t valueless_elements = 0;
|
||||
|
||||
/// whether we can assume little endianness
|
||||
const bool is_little_endian = little_endianness();
|
||||
|
||||
|
||||
@@ -1204,20 +1204,27 @@ class binary_writer
|
||||
|
||||
// an optimized array of a valueless type carries no payload, so a
|
||||
// reader has nothing but the declared count to bound the allocation
|
||||
// by and refuses an excessive one. Write the unoptimized form for
|
||||
// those, at one byte per element, so the result can be read back.
|
||||
// Objects are not affected: every element is preceded by its key.
|
||||
const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
|
||||
const bool excessive_valueless = IsArray
|
||||
&& valueless_type
|
||||
&& size > detail::max_valueless_container_size;
|
||||
|
||||
if (same_prefix && !excessive_valueless
|
||||
&& !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
|
||||
// by. It refuses one array of more than max_valueless_container_size
|
||||
// elements, and all such arrays of one input together when they add
|
||||
// up to more than valueless_elements_base plus a number per byte
|
||||
// read. Write the unoptimized form, at one byte per element, for
|
||||
// an array that would exceed the base with the arrays written so
|
||||
// far, so that the result can be read back: the limit of a reader
|
||||
// is never below the base. Objects are not affected: every element
|
||||
// is preceded by its key.
|
||||
if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
|
||||
{
|
||||
prefix_required = false;
|
||||
oa.write_character(to_char_type('$'));
|
||||
oa.write_character(first_prefix);
|
||||
const bool valueless_type = IsArray && (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
|
||||
if (!valueless_type || size <= detail::valueless_elements_base - valueless_elements_written)
|
||||
{
|
||||
prefix_required = false;
|
||||
oa.write_character(to_char_type('$'));
|
||||
oa.write_character(first_prefix);
|
||||
if (valueless_type)
|
||||
{
|
||||
valueless_elements_written += size;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5441,6 +5448,10 @@ class binary_writer
|
||||
/// how to treat a string value or object key that is not valid UTF-8
|
||||
/// (CBOR, MessagePack, UBJSON, BJData, and BSON; not BON8)
|
||||
const error_handler_t error_handler = binary_writer_default_error_handler();
|
||||
|
||||
/// UBJSON: the number of elements written so far in optimized arrays of a
|
||||
/// valueless type, which a reader adds up and bounds
|
||||
std::size_t valueless_elements_written = 0;
|
||||
};
|
||||
|
||||
} // namespace detail
|
||||
|
||||
Generated
+108
-26
@@ -18352,13 +18352,46 @@ the input.
|
||||
|
||||
This is a sanity bound rather than a security boundary, and it is far above
|
||||
any container met in practice. @ref binary_writer falls back to the
|
||||
unoptimized encoding for longer containers, so that a value serialized by
|
||||
this library can always be read back.
|
||||
unoptimized encoding for longer containers, and for containers that together
|
||||
with the ones before them would exceed @ref valueless_elements_base, so that a
|
||||
value serialized by this library can always be read back.
|
||||
|
||||
@sa https://github.com/nlohmann/json/issues/2793
|
||||
*/
|
||||
JSON_INLINE_VARIABLE constexpr std::size_t max_valueless_container_size = 1 << 20;
|
||||
|
||||
/*!
|
||||
@brief number of elements of valueless UBJSON containers accepted on top of
|
||||
@ref valueless_elements_per_byte
|
||||
|
||||
The limit @ref max_valueless_container_size applies to each container on its
|
||||
own, so on its own it lets 9 bytes announce 1048576 values, and every further
|
||||
9 bytes another 1048576. The elements of all containers of a valueless type
|
||||
in one input are therefore added up, and the sum must stay below this base
|
||||
plus @ref valueless_elements_per_byte for every byte read so far. The base is
|
||||
the per-container limit, so a container that was accepted on its own before
|
||||
the sum was introduced is still accepted at the start of an input.
|
||||
*/
|
||||
JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_base = max_valueless_container_size;
|
||||
|
||||
/*!
|
||||
@brief number of elements of valueless UBJSON containers accepted per byte of
|
||||
input, in addition to @ref valueless_elements_base
|
||||
|
||||
An element of a value that has a payload costs at least one byte of input.
|
||||
Valueless elements are accepted at a rate of 4 per byte. A value takes 16
|
||||
bytes of memory in a DOM, so these arrays add about 80 bytes of memory per
|
||||
byte of input (measured: 77; one `Z` per byte needs about 32), beyond what the
|
||||
same bytes cost anyway, plus the base. This bounds the arrays of valueless
|
||||
type, not the memory of a whole document. Containers written by
|
||||
@ref binary_writer never need more than the base, because it writes an array
|
||||
in the unoptimized form once the typed arrays written so far and this one add
|
||||
up to more than that. The rate matters for input written by other encoders and
|
||||
by versions before 3.13.0 only; in the data this was measured on, no input has
|
||||
more than 0.17 elements per byte.
|
||||
*/
|
||||
JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_per_byte = 4;
|
||||
|
||||
///////////////////
|
||||
// binary reader //
|
||||
///////////////////
|
||||
@@ -21979,6 +22012,37 @@ class binary_reader
|
||||
return emit_unsigned(number);
|
||||
}
|
||||
|
||||
/*!
|
||||
@brief add the elements of a container of a valueless type to the total of
|
||||
the input and check the total against the input read so far
|
||||
|
||||
A container of type 'Z', 'T' or 'F' has no payload, so its elements are the
|
||||
only ones for which the input does not bound the count. The check is made
|
||||
once per container, not per element.
|
||||
|
||||
@param[in] count number of elements the container declares; at most
|
||||
@ref max_valueless_container_size
|
||||
@return false if the total is more than @ref valueless_elements_base plus
|
||||
@ref valueless_elements_per_byte for every byte read so far; the
|
||||
total is not changed then
|
||||
*/
|
||||
bool charge_valueless_elements(const std::size_t count) noexcept
|
||||
{
|
||||
// the limit saturates instead of overflowing when size_t is 32 bits
|
||||
// wide and a very large input has been read; valueless_elements
|
||||
// never exceeds the limit, so the difference cannot underflow
|
||||
constexpr std::size_t max_size = (std::numeric_limits<std::size_t>::max)();
|
||||
const std::size_t limit = chars_read <= (max_size - valueless_elements_base) / valueless_elements_per_byte
|
||||
? valueless_elements_base + (valueless_elements_per_byte * chars_read)
|
||||
: max_size;
|
||||
if (JSON_HEDLEY_UNLIKELY(count > limit - valueless_elements))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
valueless_elements += count;
|
||||
return true;
|
||||
}
|
||||
|
||||
/*!
|
||||
@return whether array creation completed
|
||||
*/
|
||||
@@ -22064,24 +22128,27 @@ class binary_reader
|
||||
// declared count alone decides how much is allocated; the check is
|
||||
// made before the start event so that no container is opened that
|
||||
// is then abandoned. See @ref max_valueless_container_size.
|
||||
if (JSON_HEDLEY_UNLIKELY((size_and_type.second == 'Z' || size_and_type.second == 'T' || size_and_type.second == 'F')
|
||||
&& size_and_type.first > max_valueless_container_size))
|
||||
if (JSON_HEDLEY_UNLIKELY(size_and_type.second == 'Z' || size_and_type.second == 'T' || size_and_type.second == 'F'))
|
||||
{
|
||||
return out_of_range_error(exception_id::container_too_large, "excessive array size", "size");
|
||||
if (JSON_HEDLEY_UNLIKELY(size_and_type.first > max_valueless_container_size))
|
||||
{
|
||||
return out_of_range_error(exception_id::container_too_large, "excessive array size", "size");
|
||||
}
|
||||
if (JSON_HEDLEY_UNLIKELY(!charge_valueless_elements(size_and_type.first)))
|
||||
{
|
||||
return out_of_range_error(exception_id::container_too_large, "excessive number of elements without payload", "size");
|
||||
}
|
||||
}
|
||||
|
||||
if (JSON_HEDLEY_UNLIKELY(!enter_array(size_and_type.first, size_and_type.second)))
|
||||
// a no-op is not a value, so an array of them is empty whatever
|
||||
// count it declares: the SAX parser is told so, instead of being
|
||||
// made to reserve storage for a count that no input backs
|
||||
const std::size_t announced = size_and_type.second == 'N' ? 0 : size_and_type.first;
|
||||
if (JSON_HEDLEY_UNLIKELY(!enter_array(announced, size_and_type.second)))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
if (size_and_type.second == 'N')
|
||||
{
|
||||
// a no-op is not a value, so a container of them holds none;
|
||||
// the declared size has already been passed to the SAX parser
|
||||
container_stack.back().remaining = 0;
|
||||
}
|
||||
|
||||
return size_and_type.second == 0
|
||||
|| parse_ubjson_typed_run(container_stack.back().remaining, size_and_type.second, std::integral_constant<bool, bulk_scan> {});
|
||||
}
|
||||
@@ -23393,6 +23460,10 @@ class binary_reader
|
||||
/// the number of characters read
|
||||
std::size_t chars_read = 0;
|
||||
|
||||
/// UBJSON: the sum of the element counts of the containers of a valueless
|
||||
/// type read so far; see @ref charge_valueless_elements
|
||||
std::size_t valueless_elements = 0;
|
||||
|
||||
/// whether we can assume little endianness
|
||||
const bool is_little_endian = little_endianness();
|
||||
|
||||
@@ -30546,20 +30617,27 @@ class binary_writer
|
||||
|
||||
// an optimized array of a valueless type carries no payload, so a
|
||||
// reader has nothing but the declared count to bound the allocation
|
||||
// by and refuses an excessive one. Write the unoptimized form for
|
||||
// those, at one byte per element, so the result can be read back.
|
||||
// Objects are not affected: every element is preceded by its key.
|
||||
const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
|
||||
const bool excessive_valueless = IsArray
|
||||
&& valueless_type
|
||||
&& size > detail::max_valueless_container_size;
|
||||
|
||||
if (same_prefix && !excessive_valueless
|
||||
&& !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
|
||||
// by. It refuses one array of more than max_valueless_container_size
|
||||
// elements, and all such arrays of one input together when they add
|
||||
// up to more than valueless_elements_base plus a number per byte
|
||||
// read. Write the unoptimized form, at one byte per element, for
|
||||
// an array that would exceed the base with the arrays written so
|
||||
// far, so that the result can be read back: the limit of a reader
|
||||
// is never below the base. Objects are not affected: every element
|
||||
// is preceded by its key.
|
||||
if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
|
||||
{
|
||||
prefix_required = false;
|
||||
oa.write_character(to_char_type('$'));
|
||||
oa.write_character(first_prefix);
|
||||
const bool valueless_type = IsArray && (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
|
||||
if (!valueless_type || size <= detail::valueless_elements_base - valueless_elements_written)
|
||||
{
|
||||
prefix_required = false;
|
||||
oa.write_character(to_char_type('$'));
|
||||
oa.write_character(first_prefix);
|
||||
if (valueless_type)
|
||||
{
|
||||
valueless_elements_written += size;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -34783,6 +34861,10 @@ class binary_writer
|
||||
/// how to treat a string value or object key that is not valid UTF-8
|
||||
/// (CBOR, MessagePack, UBJSON, BJData, and BSON; not BON8)
|
||||
const error_handler_t error_handler = binary_writer_default_error_handler();
|
||||
|
||||
/// UBJSON: the number of elements written so far in optimized arrays of a
|
||||
/// valueless type, which a reader adds up and bounds
|
||||
std::size_t valueless_elements_written = 0;
|
||||
};
|
||||
|
||||
} // namespace detail
|
||||
|
||||
@@ -19,8 +19,12 @@
|
||||
#include <nlohmann/json.hpp>
|
||||
using nlohmann::json;
|
||||
|
||||
#include <cstdio>
|
||||
#include <iostream>
|
||||
#include <sstream>
|
||||
#include <utility>
|
||||
#include <fstream>
|
||||
#include <limits>
|
||||
#include <list>
|
||||
#include <set>
|
||||
#include "make_test_data_available.hpp"
|
||||
@@ -2436,6 +2440,341 @@ TEST_CASE("UBJSON optimized arrays of a valueless type are bounded")
|
||||
}
|
||||
}
|
||||
|
||||
namespace
|
||||
{
|
||||
/// SAX consumer that counts the values and keeps the id of a parse error
|
||||
struct ValuelessEventCounter : public nlohmann::json_sax<json>
|
||||
{
|
||||
bool null() override
|
||||
{
|
||||
return ++values < stop_after;
|
||||
}
|
||||
bool boolean(bool /*val*/) override
|
||||
{
|
||||
return ++values < stop_after;
|
||||
}
|
||||
bool number_integer(json::number_integer_t /*val*/) override
|
||||
{
|
||||
++values;
|
||||
return true;
|
||||
}
|
||||
bool number_unsigned(json::number_unsigned_t /*val*/) override
|
||||
{
|
||||
++values;
|
||||
return true;
|
||||
}
|
||||
bool number_float(json::number_float_t /*val*/, const std::string& /*s*/) override
|
||||
{
|
||||
++values;
|
||||
return true;
|
||||
}
|
||||
bool string(std::string& /*val*/) override
|
||||
{
|
||||
++values;
|
||||
return true;
|
||||
}
|
||||
bool binary(json::binary_t& /*val*/) override
|
||||
{
|
||||
++values;
|
||||
return true;
|
||||
}
|
||||
bool start_object(std::size_t /*elements*/) override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool key(std::string& /*val*/) override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool end_object() override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool start_array(std::size_t /*elements*/) override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool end_array() override
|
||||
{
|
||||
return true;
|
||||
}
|
||||
bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const json::exception& ex) override
|
||||
{
|
||||
error_id = ex.id;
|
||||
return false;
|
||||
}
|
||||
|
||||
/// the handler refuses the value that makes this many
|
||||
std::size_t stop_after = (std::numeric_limits<std::size_t>::max)();
|
||||
std::size_t values = 0;
|
||||
int error_id = 0;
|
||||
};
|
||||
|
||||
/// appends `[$<marker>#l<count>` (nine bytes) to @a v
|
||||
void append_valueless_array(std::vector<std::uint8_t>& v, const char marker, const std::uint32_t count)
|
||||
{
|
||||
v.push_back('[');
|
||||
v.push_back('$');
|
||||
v.push_back(static_cast<std::uint8_t>(marker));
|
||||
v.push_back('#');
|
||||
v.push_back('l');
|
||||
v.push_back(static_cast<std::uint8_t>(count >> 24));
|
||||
v.push_back(static_cast<std::uint8_t>(count >> 16));
|
||||
v.push_back(static_cast<std::uint8_t>(count >> 8));
|
||||
v.push_back(static_cast<std::uint8_t>(count));
|
||||
}
|
||||
|
||||
/// `[` followed by one `[$<marker>#l<count>` per entry of @a counts and `]`
|
||||
std::vector<std::uint8_t> valueless_arrays(const char marker, const std::vector<std::uint32_t>& counts)
|
||||
{
|
||||
std::vector<std::uint8_t> v = {'['};
|
||||
for (const auto count : counts)
|
||||
{
|
||||
append_valueless_array(v, marker, count);
|
||||
}
|
||||
v.push_back(']');
|
||||
return v;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
TEST_CASE("UBJSON the valueless elements of a whole input are bounded")
|
||||
{
|
||||
// The per-container limit alone let nine bytes (`[$Z#l` and a count of
|
||||
// 1048576) announce a million values, and every further nine bytes another
|
||||
// million. The elements of all containers of type 'Z', 'T' or 'F' in one
|
||||
// input are now added up: they may not exceed 1048576 plus 4 for every
|
||||
// byte read when the container's header has been read.
|
||||
//
|
||||
// The tests count events with a SAX handler where they can: building and
|
||||
// comparing values of a million elements is what makes them slow under
|
||||
// Valgrind, so there is one rejection and one round trip with a DOM.
|
||||
json _;
|
||||
const char* const budget_message = "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive number of elements without payload";
|
||||
|
||||
// two arrays that exceed the limit: the second one, with its header read
|
||||
// at byte 19, may hold 4 * 19 = 76 values on top of the base of 1048576
|
||||
const std::uint32_t over_first = 524288;
|
||||
const std::uint32_t over_second = 1048576 + 76 - over_first + 1;
|
||||
|
||||
SECTION("the input of the original report is rejected")
|
||||
{
|
||||
// 362 bytes (40 arrays of 1048576 values) used to allocate 661 MB; the
|
||||
// second array is the one that is refused
|
||||
const auto input = valueless_arrays('Z', {over_first, over_second});
|
||||
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), budget_message, json::out_of_range&);
|
||||
}
|
||||
|
||||
SECTION("every input adapter applies the budget")
|
||||
{
|
||||
const auto input = valueless_arrays('Z', {over_first, over_second});
|
||||
|
||||
// stream; not strict, which does not matter
|
||||
std::stringstream stream;
|
||||
stream.write(reinterpret_cast<const char*>(input.data()), static_cast<std::streamsize>(input.size()));
|
||||
ValuelessEventCounter from_stream;
|
||||
CHECK(!json::sax_parse(stream, &from_stream, json::input_format_t::ubjson, false));
|
||||
CHECK(from_stream.error_id == 408);
|
||||
|
||||
// a container that is not contiguous; the counts of arrays of
|
||||
// different types are added up
|
||||
std::vector<std::uint8_t> mixed = {'['};
|
||||
append_valueless_array(mixed, 'T', over_first);
|
||||
append_valueless_array(mixed, 'F', over_second);
|
||||
mixed.push_back(']');
|
||||
const std::list<std::uint8_t> as_list(mixed.begin(), mixed.end());
|
||||
ValuelessEventCounter from_list;
|
||||
CHECK(!json::sax_parse(as_list.begin(), as_list.end(), &from_list, json::input_format_t::ubjson));
|
||||
CHECK(from_list.error_id == 408);
|
||||
CHECK(from_list.values == over_first);
|
||||
|
||||
// FILE*
|
||||
FILE* const file = std::tmpfile(); // NOLINT(cppcoreguidelines-owning-memory)
|
||||
REQUIRE(file != nullptr);
|
||||
CHECK(std::fwrite(input.data(), 1, input.size(), file) == input.size());
|
||||
std::rewind(file);
|
||||
ValuelessEventCounter from_file;
|
||||
CHECK(!json::sax_parse(file, &from_file, json::input_format_t::ubjson));
|
||||
CHECK(from_file.error_id == 408);
|
||||
std::fclose(file); // NOLINT(cert-err33-c,cppcoreguidelines-owning-memory)
|
||||
}
|
||||
|
||||
SECTION("the limit is exact and the bytes read raise it")
|
||||
{
|
||||
// [ [$Z#l 524288] [$U#l 4096 + 4096 bytes] [$Z#l count] ]: chars_read
|
||||
// is 1 + 9 + (9 + 4096) + 9 = 4124 when the last header has been read,
|
||||
// so the last array may hold 4 * 4124 = 16496 values on top of the
|
||||
// base of 1048576 less the 524288 of the first array
|
||||
const std::uint32_t last_allowed = 1048576 + 16496 - 524288;
|
||||
for (const std::uint32_t count :
|
||||
{
|
||||
last_allowed, last_allowed + 1
|
||||
})
|
||||
{
|
||||
CAPTURE(count)
|
||||
std::vector<std::uint8_t> input = {'['};
|
||||
append_valueless_array(input, 'Z', 524288);
|
||||
input.insert(input.end(), {'[', '$', 'U', '#', 'l', 0x00, 0x00, 0x10, 0x00});
|
||||
input.insert(input.end(), 4096, 0x01);
|
||||
append_valueless_array(input, 'Z', count);
|
||||
input.push_back(']');
|
||||
ValuelessEventCounter sax;
|
||||
const bool accepted = count == last_allowed;
|
||||
CHECK(json::sax_parse(input, &sax, json::input_format_t::ubjson) == accepted);
|
||||
CHECK(sax.error_id == (accepted ? 0 : 408));
|
||||
CHECK(sax.values == 524288u + 4096u + (accepted ? count : 0u));
|
||||
}
|
||||
|
||||
// an array of 1048576 values on its own is accepted as before
|
||||
const std::vector<std::uint8_t> single = {'[', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
|
||||
// (the handler ends the parse at the first value, so that the million
|
||||
// values are not read)
|
||||
ValuelessEventCounter sax;
|
||||
sax.stop_after = 1;
|
||||
CHECK(!json::sax_parse(single, &sax, json::input_format_t::ubjson));
|
||||
CHECK(sax.error_id == 0);
|
||||
CHECK(sax.values == 1);
|
||||
}
|
||||
|
||||
SECTION("to_ubjson writes the typed form up to the base only")
|
||||
{
|
||||
// 1048 rows of 1000 values fit under the base of 1048576, the 1049th
|
||||
// would not: its elements are written one by one. Written all typed,
|
||||
// 1100 rows of 6 bytes would be refused by a reader (1075 rows are the
|
||||
// most it accepts). The rows hold null, true or false, and the
|
||||
// fallback does not depend on which
|
||||
std::vector<std::uint8_t> encoded;
|
||||
{
|
||||
json table = json::array();
|
||||
for (int row = 0; row < 1100; ++row)
|
||||
{
|
||||
const json cell = row % 3 == 0 ? json(nullptr) : json(row % 3 == 1);
|
||||
table.push_back(json(1000, cell));
|
||||
}
|
||||
encoded = json::to_ubjson(table, true, true);
|
||||
}
|
||||
// outer array: `[$[#I` and the count (7 bytes), then 1048 typed rows of 6
|
||||
// bytes (`$Z#I` and two bytes) and 52 rows of `#I`, two bytes, and 1000 `Z`
|
||||
CHECK(encoded.size() == 7 + (1048 * 6) + (52 * (4 + 1000)));
|
||||
|
||||
// the output is read back
|
||||
ValuelessEventCounter sax;
|
||||
CHECK(json::sax_parse(encoded, &sax, json::input_format_t::ubjson));
|
||||
CHECK(sax.error_id == 0);
|
||||
CHECK(sax.values == 1100u * 1000u);
|
||||
}
|
||||
|
||||
SECTION("BJData is not affected")
|
||||
{
|
||||
// BJData has no optimized arrays of these types, and the writer does
|
||||
// not charge them
|
||||
json table = json::array();
|
||||
for (int row = 0; row < 3; ++row)
|
||||
{
|
||||
table.push_back(json(10, nullptr));
|
||||
}
|
||||
const auto encoded = json::to_bjdata(table, true, true);
|
||||
CHECK(json::from_bjdata(encoded) == table);
|
||||
|
||||
const std::vector<std::uint8_t> input = {'[', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
|
||||
CHECK_THROWS_AS(_ = json::from_bjdata(input), json::parse_error&);
|
||||
}
|
||||
|
||||
SECTION("an object of a valueless type needs a key for every value")
|
||||
{
|
||||
// an object member costs a key, so the count is bounded by the input
|
||||
// and is not charged to the budget
|
||||
const std::vector<std::uint8_t> input = {'{', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
|
||||
ValuelessEventCounter sax;
|
||||
CHECK(!json::sax_parse(input, &sax, json::input_format_t::ubjson));
|
||||
CHECK(sax.error_id == 110);
|
||||
CHECK(sax.values == 0);
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("UBJSON an optimized array of no-ops is empty whatever its count")
|
||||
{
|
||||
// A no-op ('N') is not a value, so `[$N#<n>` is an array without elements
|
||||
// for every n. The count used to be passed on to the SAX parser, which
|
||||
// told it to expect n elements, and the DOM parser reserved storage for
|
||||
// them: 16384 values (256 KB) for a count of 2^31 in 9 bytes of input.
|
||||
SECTION("the SAX parser is told that the array is empty")
|
||||
{
|
||||
for (const std::vector<std::uint8_t>& input :
|
||||
{
|
||||
std::vector<std::uint8_t> {'[', '$', 'N', '#', 'i', 2},
|
||||
std::vector<std::uint8_t> {'[', '$', 'N', '#', 'I', 0x7F, 0xFF},
|
||||
std::vector<std::uint8_t> {'[', '$', 'N', '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF},
|
||||
std::vector<std::uint8_t> {'[', '$', 'N', '#', 'i', 0}
|
||||
})
|
||||
{
|
||||
SaxEventLogger logger;
|
||||
CHECK(json::sax_parse(input, &logger, json::input_format_t::ubjson));
|
||||
CHECK(logger.events == std::vector<std::string>({"start_array(0)", "end_array()"}));
|
||||
}
|
||||
|
||||
// between other values, the next value is read as usual
|
||||
const std::vector<std::uint8_t> nested = {'[', '[', '$', 'N', '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF, 'T', '[', '$', 'N', '#', 'i', 3, ']'};
|
||||
SaxEventLogger logger;
|
||||
CHECK(json::sax_parse(nested, &logger, json::input_format_t::ubjson));
|
||||
CHECK(logger.events == std::vector<std::string>({"start_array()", "start_array(0)", "end_array()", "boolean(true)", "start_array(0)", "end_array()", "end_array()"}));
|
||||
}
|
||||
|
||||
SECTION("the DOM parser reserves nothing")
|
||||
{
|
||||
// 2000 arrays of 9 bytes that announce 2^31 elements each: with a
|
||||
// reservation of 16384 values per array this took about 500 MB
|
||||
std::vector<std::uint8_t> input = {'['};
|
||||
for (int i = 0; i < 2000; ++i)
|
||||
{
|
||||
input.insert(input.end(), {'[', '$', 'N', '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF});
|
||||
}
|
||||
input.push_back(']');
|
||||
|
||||
const json j = json::from_ubjson(input);
|
||||
REQUIRE(j.size() == 2000);
|
||||
std::size_t capacity = 0;
|
||||
for (const auto& element : j)
|
||||
{
|
||||
CHECK(element.is_array());
|
||||
CHECK(element.empty());
|
||||
capacity += element.get_ref<const json::array_t&>().capacity();
|
||||
}
|
||||
CHECK(capacity == 0);
|
||||
|
||||
// the 7-byte form inside an optimized array of arrays
|
||||
std::vector<std::uint8_t> typed = {'[', '$', '[', '#', 'I', 0x07, 0xD0};
|
||||
for (int i = 0; i < 2000; ++i)
|
||||
{
|
||||
typed.insert(typed.end(), {'$', 'N', '#', 'I', 0x7F, 0xFF});
|
||||
}
|
||||
const json k = json::from_ubjson(typed);
|
||||
REQUIRE(k.size() == 2000);
|
||||
capacity = 0;
|
||||
for (const auto& element : k)
|
||||
{
|
||||
CHECK(element.empty());
|
||||
capacity += element.get_ref<const json::array_t&>().capacity();
|
||||
}
|
||||
CHECK(capacity == 0);
|
||||
}
|
||||
|
||||
SECTION("a count that no container could hold is no longer an error")
|
||||
{
|
||||
// the array is empty, so nothing has to fit: 0x7FFFFFFF used to be
|
||||
// reported as out_of_range.408 where it exceeds the maximum size of
|
||||
// an array, and is accepted now
|
||||
const std::vector<std::uint8_t> input = {'[', '$', 'N', '#', 'l', 0x7F, 0xFF, 0xFF, 0xFF};
|
||||
CHECK(json::from_ubjson(input) == json::array());
|
||||
|
||||
// a count that does not fit into std::size_t still is
|
||||
#if SIZE_MAX == 0xffffffff
|
||||
const std::vector<std::uint8_t> too_large = {'[', '$', 'N', '#', 'L', 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00};
|
||||
json _;
|
||||
CHECK_THROWS_AS(_ = json::from_ubjson(too_large), json::out_of_range&);
|
||||
#endif
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("issue #5405 - array reserve for definite-length UBJSON arrays")
|
||||
{
|
||||
#if !defined(JSON_NOEXCEPTION)
|
||||
|
||||
Reference in new issue
Block a user