Compare commits

..
Author SHA1 Message Date
Niels Lohmann 017a3ce001 ci: revert importlib-metadata to 8.7.1 to fix Semgrep install (#6064)
The Dependabot bump to 10.1.0 (#6059) conflicts with the pinned
opentelemetry-api 1.37.0 (importlib-metadata<8.8.0), so the Semgrep job
failed with ResolutionImpossible. Ignore importlib-metadata in Dependabot
since semgrep pins it indirectly.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-11 09:18:40 +01:00
11 changed files with 54 additions and 505 deletions

No files matched your search

+1
View File
@@ -67,3 +67,4 @@ updates:
- dependency-name: mcp
- dependency-name: boltons
- dependency-name: "opentelemetry-*"
- dependency-name: importlib-metadata
+3 -3
View File
@@ -391,9 +391,9 @@ httpx-sse==0.4.3 \
idna==3.20 \
--hash=sha256:a7db850025b95ded1eae8a46181a1a6c56c92c96f0e2b005d9ff8dc0210cab44 \
--hash=sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c
importlib-metadata==10.1.0 \
--hash=sha256:82b04047fa09792c1a8cfeddfe08bec79e4b13c29c8469b0d3949860f61017e8 \
--hash=sha256:cf45ce66a929c64e994185a809c04840d0501db17e663810d993d0f4829265c7
importlib-metadata==8.7.1 \
--hash=sha256:49fef1ae6440c182052f407c8d34a68f72efc36db9ca90dc0113398f2fdde8bb \
--hash=sha256:5a1f80bf1daa489495071efbb095d75a634cf28a8bc299581244063b53176151
jsonschema==4.25.1 \
--hash=sha256:3fba0169e345c7175110351d456342c364814cfcf3b964ba4587f22915230a63 \
--hash=sha256:e4a9655ce0da0c0b67a085847e00a3a51449e1157f4f75e9fb5aa545e122eb85
@@ -89,9 +89,8 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
- Throws [out_of_range.406](../../home/exceptions.md#jsonexceptionout_of_range406) if a high-precision number, or a
float64 when `number_float_t` is `float`, does not fit into `number_float_t`
- Throws [out_of_range.408](../../home/exceptions.md#jsonexceptionout_of_range408) if the count of an optimized
container exceeds the capacity of the container (or of `std::size_t`), if an optimized array of `Z`, `T`, or `F` has
more than 1,048,576 elements, or if the optimized arrays of `Z`, `T`, and `F` in the input together have more than
1,048,576 elements plus 4 for every byte read so far
container exceeds the capacity of the container (or of `std::size_t`), or if an optimized array of `Z`, `T`, or `F` has
more than 1,048,576 elements
## Complexity
@@ -133,7 +132,6 @@ Linear in the size of the input.
- Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
- Added `error_handler` parameter in version 3.13.0.
- Limited the total number of elements of optimized arrays of `Z`, `T`, and `F` in one input in version 3.13.0.
!!! warning "Deprecation"
@@ -117,8 +117,5 @@ Linear in the size of the JSON value `j`.
- Added `error_handler` parameter in version 3.13.0. Its default, `keep`, writes the bytes of a string or object key
that is not valid UTF-8 unchanged, as before; `strict` (the default if
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316`.
- Writes arrays of `Z`, `T`, or `F` without the `$` type annotation (`[#n Z Z ...`) once their elements in one call
add up to more than 1,048,576, in version 3.13.0, so that `from_ubjson` can read the output back. Such output is
larger: 270,000 rows of 100 `null` values take 26.8 MB instead of 1.35 MB.
- Throws `type_error.321` for a discarded value since version 3.13.0; previously, a discarded value nested in an
array or object was silently skipped, producing invalid UBJSON.
@@ -122,14 +122,9 @@ The library uses the following mapping from JSON values types to UBJSON types ac
already is the value. Its declared count is therefore the only thing that decides how much memory the receiving side
allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
[`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) when the count exceeds 1,048,576
(`1 << 20`), or when the counts of all such arrays in the input add up to more than 1,048,576 plus 4 for every byte
read so far. `to_ubjson` therefore writes an array of these types in the counted but untyped form when it is longer
than that, or when it would bring the total of the typed arrays written so far above 1,048,576: the `$` type marker
is dropped, the `#` count is kept, and every element is written with its own marker (`[#l 1048577 Z Z ...`), so
everything it produces can be read back. Only input from another encoder, or written by version 3.12.0 and earlier of
this library, can exceed the limit: for example, 1,100 rows of 1,000 `null` values or 210,000 rows of 25, each with
a `$Z` annotation. Version 3.12.0 writes such a row in 5 bytes (6 above 255 elements), so rows of 20 or fewer such
values are never rejected. Objects and untyped counted arrays are not subject to this limit.
(`1 << 20`), and `to_ubjson` writes longer arrays of these types in the counted but untyped form: the `$` type marker is
dropped, the `#` count is kept, and every element is written with its own marker (`[#l 1048577 Z Z ...`), so any value
it produces can be read back. Objects and untyped counted arrays are not subject to this limit.
!!! info "Binary values"
@@ -185,11 +180,10 @@ The library maps UBJSON types to JSON value types as follows:
Every well-formed UBJSON value can be converted to a JSON value, with these exceptions: a high-precision number
outside the range of `number_float_t` ([`out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406)),
optimized arrays of `Z`, `T`, or `F` with more elements than the limit allows (1,048,576 plus 4 per byte read),
and a count that exceeds the capacity of the container
([`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408)). High-precision numbers that do not
fit an integer type are stored as floating-point numbers and may lose precision. Ill-formed input is rejected with a
parse error.
an optimized array of `Z`, `T`, or `F` with more than 1,048,576 elements, and a count that exceeds the capacity of the
container ([`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408)). High-precision numbers that
do not fit an integer type are stored as floating-point numbers and may lose precision. Ill-formed input is rejected
with a parse error.
!!! info "Duplicate object keys"
@@ -76,13 +76,6 @@ For the binary formats, an announced size is never trusted outright:
- An announced array or object size that exceeds what the target container could ever hold (its `max_size()`) is
rejected immediately as [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408), without
attempting to allocate anything.
- A [UBJSON](../binary_formats/ubjson.md) array of type `Z`, `T` or `F` has no payload, so its announced count is not
bounded by the input. One array may announce at most 1,048,576 elements, and all such arrays of one input together
may contain at most 1,048,576 elements plus 4 for every byte read so far (checked when each array's header has been
read); beyond that the input is rejected as
[`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408). For a DOM that bounds the memory these
arrays can need to about 80 bytes per byte of input plus about 33 MB for the first 1,048,576 elements; a SAX handler
receives at most that many events. A UBJSON object member always costs a key, so it needs input too.
## Strings
+5 -14
View File
@@ -1050,17 +1050,11 @@ The size of an array or object in a [binary format](../features/binary_formats/i
the size following `#` for [UBJSON](../features/binary_formats/ubjson.md)/[BJData](../features/binary_formats/bjdata.md),
or the encoded length for [CBOR](../features/binary_formats/cbor.md).
The exception is also thrown for [UBJSON](../features/binary_formats/ubjson.md) arrays of a type that is encoded by
its marker alone (`Z`, `T` or `F`). Such an array has no payload, so its count alone decides how much memory is
allocated, and a handful of bytes would otherwise describe billions of values. It is thrown if the declared count of one
array exceeds 1,048,576 (`1 << 20`), and if the counts of all such arrays read so far in one input add up to more than
1,048,576 plus 4 for every byte of the input read when the header of the last array has been read.
[`to_ubjson`](../api/basic_json/to_ubjson.md) writes an array of these types without the `$` type annotation (the `#`
count and one marker per element are kept) when it is longer, or when it would bring the total of the typed arrays
written so far above 1,048,576, so everything it produces can be read back. Only input written by another encoder, or
by version 3.12.0 and earlier of this library, can exceed the limit: for example, 1,100 rows of 1,000 `null` values or
210,000 rows of 25, each with a `$Z` annotation. Version 3.12.0 writes such a row in 5 bytes (6 above 255 elements),
so rows of 20 or fewer such values are never rejected.
The exception is also thrown for a [UBJSON](../features/binary_formats/ubjson.md) array of a type that is encoded by its
marker alone (`Z`, `T` or `F`) whose declared count exceeds 1,048,576 (`1 << 20`). Such an array has no payload, so its
count alone decides how much memory is allocated, and a handful of bytes would otherwise describe billions of values.
[`to_ubjson`](../api/basic_json/to_ubjson.md) writes longer arrays of these types without the `$` type annotation (the `#`
count and one marker per element are kept), so any value it produces can still be read back.
!!! failure "Example messages"
@@ -1076,9 +1070,6 @@ so rows of 20 or fewer such values are never rejected.
```
[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size
```
```
[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive number of elements without payload
```
### json.exception.out_of_range.409
@@ -73,46 +73,13 @@ the input.
This is a sanity bound rather than a security boundary, and it is far above
any container met in practice. @ref binary_writer falls back to the
unoptimized encoding for longer containers, and for containers that together
with the ones before them would exceed @ref valueless_elements_base, so that a
value serialized by this library can always be read back.
unoptimized encoding for longer containers, so that a value serialized by
this library can always be read back.
@sa https://github.com/nlohmann/json/issues/2793
*/
JSON_INLINE_VARIABLE constexpr std::size_t max_valueless_container_size = 1 << 20;
/*!
@brief number of elements of valueless UBJSON containers accepted on top of
@ref valueless_elements_per_byte
The limit @ref max_valueless_container_size applies to each container on its
own, so on its own it lets 9 bytes announce 1048576 values, and every further
9 bytes another 1048576. The elements of all containers of a valueless type
in one input are therefore added up, and the sum must stay below this base
plus @ref valueless_elements_per_byte for every byte read so far. The base is
the per-container limit, so a container that was accepted on its own before
the sum was introduced is still accepted at the start of an input.
*/
JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_base = max_valueless_container_size;
/*!
@brief number of elements of valueless UBJSON containers accepted per byte of
input, in addition to @ref valueless_elements_base
An element of a value that has a payload costs at least one byte of input.
Valueless elements are accepted at a rate of 4 per byte. A value takes 16
bytes of memory in a DOM, so these arrays add about 80 bytes of memory per
byte of input (measured: 77; one `Z` per byte needs about 32), beyond what the
same bytes cost anyway, plus the base. This bounds the arrays of valueless
type, not the memory of a whole document. Containers written by
@ref binary_writer never need more than the base, because it writes an array
in the unoptimized form once the typed arrays written so far and this one add
up to more than that. The rate matters for input written by other encoders and
by versions before 3.13.0 only; in the data this was measured on, no input has
more than 0.17 elements per byte.
*/
JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_per_byte = 4;
///////////////////
// binary reader //
///////////////////
@@ -3733,37 +3700,6 @@ class binary_reader
return emit_unsigned(number);
}
/*!
@brief add the elements of a container of a valueless type to the total of
the input and check the total against the input read so far
A container of type 'Z', 'T' or 'F' has no payload, so its elements are the
only ones for which the input does not bound the count. The check is made
once per container, not per element.
@param[in] count number of elements the container declares; at most
@ref max_valueless_container_size
@return false if the total is more than @ref valueless_elements_base plus
@ref valueless_elements_per_byte for every byte read so far; the
total is not changed then
*/
bool charge_valueless_elements(const std::size_t count) noexcept
{
// the limit saturates instead of overflowing when size_t is 32 bits
// wide and a very large input has been read; valueless_elements
// never exceeds the limit, so the difference cannot underflow
constexpr std::size_t max_size = (std::numeric_limits<std::size_t>::max)();
const std::size_t limit = chars_read <= (max_size - valueless_elements_base) / valueless_elements_per_byte
? valueless_elements_base + (valueless_elements_per_byte * chars_read)
: max_size;
if (JSON_HEDLEY_UNLIKELY(count > limit - valueless_elements))
{
return false;
}
valueless_elements += count;
return true;
}
/*!
@return whether array creation completed
*/
@@ -3849,16 +3785,10 @@ class binary_reader
// declared count alone decides how much is allocated; the check is
// made before the start event so that no container is opened that
// is then abandoned. See @ref max_valueless_container_size.
if (JSON_HEDLEY_UNLIKELY(size_and_type.second == 'Z' || size_and_type.second == 'T' || size_and_type.second == 'F'))
if (JSON_HEDLEY_UNLIKELY((size_and_type.second == 'Z' || size_and_type.second == 'T' || size_and_type.second == 'F')
&& size_and_type.first > max_valueless_container_size))
{
if (JSON_HEDLEY_UNLIKELY(size_and_type.first > max_valueless_container_size))
{
return out_of_range_error(exception_id::container_too_large, "excessive array size", "size");
}
if (JSON_HEDLEY_UNLIKELY(!charge_valueless_elements(size_and_type.first)))
{
return out_of_range_error(exception_id::container_too_large, "excessive number of elements without payload", "size");
}
return out_of_range_error(exception_id::container_too_large, "excessive array size", "size");
}
if (JSON_HEDLEY_UNLIKELY(!enter_array(size_and_type.first, size_and_type.second)))
@@ -5184,10 +5114,6 @@ class binary_reader
/// the number of characters read
std::size_t chars_read = 0;
/// UBJSON: the sum of the element counts of the containers of a valueless
/// type read so far; see @ref charge_valueless_elements
std::size_t valueless_elements = 0;
/// whether we can assume little endianness
const bool is_little_endian = little_endianness();
@@ -1204,27 +1204,20 @@ class binary_writer
// an optimized array of a valueless type carries no payload, so a
// reader has nothing but the declared count to bound the allocation
// by. It refuses one array of more than max_valueless_container_size
// elements, and all such arrays of one input together when they add
// up to more than valueless_elements_base plus a number per byte
// read. Write the unoptimized form, at one byte per element, for
// an array that would exceed the base with the arrays written so
// far, so that the result can be read back: the limit of a reader
// is never below the base. Objects are not affected: every element
// is preceded by its key.
if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
// by and refuses an excessive one. Write the unoptimized form for
// those, at one byte per element, so the result can be read back.
// Objects are not affected: every element is preceded by its key.
const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
const bool excessive_valueless = IsArray
&& valueless_type
&& size > detail::max_valueless_container_size;
if (same_prefix && !excessive_valueless
&& !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
{
const bool valueless_type = IsArray && (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
if (!valueless_type || size <= detail::valueless_elements_base - valueless_elements_written)
{
prefix_required = false;
oa.write_character(to_char_type('$'));
oa.write_character(first_prefix);
if (valueless_type)
{
valueless_elements_written += size;
}
}
prefix_required = false;
oa.write_character(to_char_type('$'));
oa.write_character(first_prefix);
}
}
@@ -5448,10 +5441,6 @@ class binary_writer
/// how to treat a string value or object key that is not valid UTF-8
/// (CBOR, MessagePack, UBJSON, BJData, and BSON; not BON8)
const error_handler_t error_handler = binary_writer_default_error_handler();
/// UBJSON: the number of elements written so far in optimized arrays of a
/// valueless type, which a reader adds up and bounds
std::size_t valueless_elements_written = 0;
};
} // namespace detail
+18 -103
View File
@@ -18352,46 +18352,13 @@ the input.
This is a sanity bound rather than a security boundary, and it is far above
any container met in practice. @ref binary_writer falls back to the
unoptimized encoding for longer containers, and for containers that together
with the ones before them would exceed @ref valueless_elements_base, so that a
value serialized by this library can always be read back.
unoptimized encoding for longer containers, so that a value serialized by
this library can always be read back.
@sa https://github.com/nlohmann/json/issues/2793
*/
JSON_INLINE_VARIABLE constexpr std::size_t max_valueless_container_size = 1 << 20;
/*!
@brief number of elements of valueless UBJSON containers accepted on top of
@ref valueless_elements_per_byte
The limit @ref max_valueless_container_size applies to each container on its
own, so on its own it lets 9 bytes announce 1048576 values, and every further
9 bytes another 1048576. The elements of all containers of a valueless type
in one input are therefore added up, and the sum must stay below this base
plus @ref valueless_elements_per_byte for every byte read so far. The base is
the per-container limit, so a container that was accepted on its own before
the sum was introduced is still accepted at the start of an input.
*/
JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_base = max_valueless_container_size;
/*!
@brief number of elements of valueless UBJSON containers accepted per byte of
input, in addition to @ref valueless_elements_base
An element of a value that has a payload costs at least one byte of input.
Valueless elements are accepted at a rate of 4 per byte. A value takes 16
bytes of memory in a DOM, so these arrays add about 80 bytes of memory per
byte of input (measured: 77; one `Z` per byte needs about 32), beyond what the
same bytes cost anyway, plus the base. This bounds the arrays of valueless
type, not the memory of a whole document. Containers written by
@ref binary_writer never need more than the base, because it writes an array
in the unoptimized form once the typed arrays written so far and this one add
up to more than that. The rate matters for input written by other encoders and
by versions before 3.13.0 only; in the data this was measured on, no input has
more than 0.17 elements per byte.
*/
JSON_INLINE_VARIABLE constexpr std::size_t valueless_elements_per_byte = 4;
///////////////////
// binary reader //
///////////////////
@@ -22012,37 +21979,6 @@ class binary_reader
return emit_unsigned(number);
}
/*!
@brief add the elements of a container of a valueless type to the total of
the input and check the total against the input read so far
A container of type 'Z', 'T' or 'F' has no payload, so its elements are the
only ones for which the input does not bound the count. The check is made
once per container, not per element.
@param[in] count number of elements the container declares; at most
@ref max_valueless_container_size
@return false if the total is more than @ref valueless_elements_base plus
@ref valueless_elements_per_byte for every byte read so far; the
total is not changed then
*/
bool charge_valueless_elements(const std::size_t count) noexcept
{
// the limit saturates instead of overflowing when size_t is 32 bits
// wide and a very large input has been read; valueless_elements
// never exceeds the limit, so the difference cannot underflow
constexpr std::size_t max_size = (std::numeric_limits<std::size_t>::max)();
const std::size_t limit = chars_read <= (max_size - valueless_elements_base) / valueless_elements_per_byte
? valueless_elements_base + (valueless_elements_per_byte * chars_read)
: max_size;
if (JSON_HEDLEY_UNLIKELY(count > limit - valueless_elements))
{
return false;
}
valueless_elements += count;
return true;
}
/*!
@return whether array creation completed
*/
@@ -22128,16 +22064,10 @@ class binary_reader
// declared count alone decides how much is allocated; the check is
// made before the start event so that no container is opened that
// is then abandoned. See @ref max_valueless_container_size.
if (JSON_HEDLEY_UNLIKELY(size_and_type.second == 'Z' || size_and_type.second == 'T' || size_and_type.second == 'F'))
if (JSON_HEDLEY_UNLIKELY((size_and_type.second == 'Z' || size_and_type.second == 'T' || size_and_type.second == 'F')
&& size_and_type.first > max_valueless_container_size))
{
if (JSON_HEDLEY_UNLIKELY(size_and_type.first > max_valueless_container_size))
{
return out_of_range_error(exception_id::container_too_large, "excessive array size", "size");
}
if (JSON_HEDLEY_UNLIKELY(!charge_valueless_elements(size_and_type.first)))
{
return out_of_range_error(exception_id::container_too_large, "excessive number of elements without payload", "size");
}
return out_of_range_error(exception_id::container_too_large, "excessive array size", "size");
}
if (JSON_HEDLEY_UNLIKELY(!enter_array(size_and_type.first, size_and_type.second)))
@@ -23463,10 +23393,6 @@ class binary_reader
/// the number of characters read
std::size_t chars_read = 0;
/// UBJSON: the sum of the element counts of the containers of a valueless
/// type read so far; see @ref charge_valueless_elements
std::size_t valueless_elements = 0;
/// whether we can assume little endianness
const bool is_little_endian = little_endianness();
@@ -30620,27 +30546,20 @@ class binary_writer
// an optimized array of a valueless type carries no payload, so a
// reader has nothing but the declared count to bound the allocation
// by. It refuses one array of more than max_valueless_container_size
// elements, and all such arrays of one input together when they add
// up to more than valueless_elements_base plus a number per byte
// read. Write the unoptimized form, at one byte per element, for
// an array that would exceed the base with the arrays written so
// far, so that the result can be read back: the limit of a reader
// is never below the base. Objects are not affected: every element
// is preceded by its key.
if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
// by and refuses an excessive one. Write the unoptimized form for
// those, at one byte per element, so the result can be read back.
// Objects are not affected: every element is preceded by its key.
const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
const bool excessive_valueless = IsArray
&& valueless_type
&& size > detail::max_valueless_container_size;
if (same_prefix && !excessive_valueless
&& !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
{
const bool valueless_type = IsArray && (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
if (!valueless_type || size <= detail::valueless_elements_base - valueless_elements_written)
{
prefix_required = false;
oa.write_character(to_char_type('$'));
oa.write_character(first_prefix);
if (valueless_type)
{
valueless_elements_written += size;
}
}
prefix_required = false;
oa.write_character(to_char_type('$'));
oa.write_character(first_prefix);
}
}
@@ -34864,10 +34783,6 @@ class binary_writer
/// how to treat a string value or object key that is not valid UTF-8
/// (CBOR, MessagePack, UBJSON, BJData, and BSON; not BON8)
const error_handler_t error_handler = binary_writer_default_error_handler();
/// UBJSON: the number of elements written so far in optimized arrays of a
/// valueless type, which a reader adds up and bounds
std::size_t valueless_elements_written = 0;
};
} // namespace detail
-255
View File
@@ -19,12 +19,8 @@
#include <nlohmann/json.hpp>
using nlohmann::json;
#include <cstdio>
#include <iostream>
#include <sstream>
#include <utility>
#include <fstream>
#include <limits>
#include <list>
#include <set>
#include "make_test_data_available.hpp"
@@ -2440,257 +2436,6 @@ TEST_CASE("UBJSON optimized arrays of a valueless type are bounded")
}
}
namespace
{
/// SAX consumer that counts the values and keeps the id of a parse error
struct ValuelessEventCounter : public nlohmann::json_sax<json>
{
bool null() override
{
return ++values < stop_after;
}
bool boolean(bool /*val*/) override
{
return ++values < stop_after;
}
bool number_integer(json::number_integer_t /*val*/) override
{
++values;
return true;
}
bool number_unsigned(json::number_unsigned_t /*val*/) override
{
++values;
return true;
}
bool number_float(json::number_float_t /*val*/, const std::string& /*s*/) override
{
++values;
return true;
}
bool string(std::string& /*val*/) override
{
++values;
return true;
}
bool binary(json::binary_t& /*val*/) override
{
++values;
return true;
}
bool start_object(std::size_t /*elements*/) override
{
return true;
}
bool key(std::string& /*val*/) override
{
return true;
}
bool end_object() override
{
return true;
}
bool start_array(std::size_t /*elements*/) override
{
return true;
}
bool end_array() override
{
return true;
}
bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const json::exception& ex) override
{
error_id = ex.id;
return false;
}
/// the handler refuses the value that makes this many
std::size_t stop_after = (std::numeric_limits<std::size_t>::max)();
std::size_t values = 0;
int error_id = 0;
};
/// appends `[$<marker>#l<count>` (nine bytes) to @a v
void append_valueless_array(std::vector<std::uint8_t>& v, const char marker, const std::uint32_t count)
{
v.push_back('[');
v.push_back('$');
v.push_back(static_cast<std::uint8_t>(marker));
v.push_back('#');
v.push_back('l');
v.push_back(static_cast<std::uint8_t>(count >> 24));
v.push_back(static_cast<std::uint8_t>(count >> 16));
v.push_back(static_cast<std::uint8_t>(count >> 8));
v.push_back(static_cast<std::uint8_t>(count));
}
/// `[` followed by one `[$<marker>#l<count>` per entry of @a counts and `]`
std::vector<std::uint8_t> valueless_arrays(const char marker, const std::vector<std::uint32_t>& counts)
{
std::vector<std::uint8_t> v = {'['};
for (const auto count : counts)
{
append_valueless_array(v, marker, count);
}
v.push_back(']');
return v;
}
} // namespace
TEST_CASE("UBJSON the valueless elements of a whole input are bounded")
{
// The per-container limit alone let nine bytes (`[$Z#l` and a count of
// 1048576) announce a million values, and every further nine bytes another
// million. The elements of all containers of type 'Z', 'T' or 'F' in one
// input are now added up: they may not exceed 1048576 plus 4 for every
// byte read when the container's header has been read.
//
// The tests count events with a SAX handler where they can: building and
// comparing values of a million elements is what makes them slow under
// Valgrind, so there is one rejection and one round trip with a DOM.
json _;
const char* const budget_message = "[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive number of elements without payload";
// two arrays that exceed the limit: the second one, with its header read
// at byte 19, may hold 4 * 19 = 76 values on top of the base of 1048576
const std::uint32_t over_first = 524288;
const std::uint32_t over_second = 1048576 + 76 - over_first + 1;
SECTION("the input of the original report is rejected")
{
// 362 bytes (40 arrays of 1048576 values) used to allocate 661 MB; the
// second array is the one that is refused
const auto input = valueless_arrays('Z', {over_first, over_second});
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(input), budget_message, json::out_of_range&);
}
SECTION("every input adapter applies the budget")
{
const auto input = valueless_arrays('Z', {over_first, over_second});
// stream; not strict, which does not matter
std::stringstream stream;
stream.write(reinterpret_cast<const char*>(input.data()), static_cast<std::streamsize>(input.size()));
ValuelessEventCounter from_stream;
CHECK(!json::sax_parse(stream, &from_stream, json::input_format_t::ubjson, false));
CHECK(from_stream.error_id == 408);
// a container that is not contiguous; the counts of arrays of
// different types are added up
std::vector<std::uint8_t> mixed = {'['};
append_valueless_array(mixed, 'T', over_first);
append_valueless_array(mixed, 'F', over_second);
mixed.push_back(']');
const std::list<std::uint8_t> as_list(mixed.begin(), mixed.end());
ValuelessEventCounter from_list;
CHECK(!json::sax_parse(as_list.begin(), as_list.end(), &from_list, json::input_format_t::ubjson));
CHECK(from_list.error_id == 408);
CHECK(from_list.values == over_first);
// FILE*
FILE* const file = std::tmpfile(); // NOLINT(cppcoreguidelines-owning-memory)
REQUIRE(file != nullptr);
CHECK(std::fwrite(input.data(), 1, input.size(), file) == input.size());
std::rewind(file);
ValuelessEventCounter from_file;
CHECK(!json::sax_parse(file, &from_file, json::input_format_t::ubjson));
CHECK(from_file.error_id == 408);
std::fclose(file); // NOLINT(cert-err33-c,cppcoreguidelines-owning-memory)
}
SECTION("the limit is exact and the bytes read raise it")
{
// [ [$Z#l 524288] [$U#l 4096 + 4096 bytes] [$Z#l count] ]: chars_read
// is 1 + 9 + (9 + 4096) + 9 = 4124 when the last header has been read,
// so the last array may hold 4 * 4124 = 16496 values on top of the
// base of 1048576 less the 524288 of the first array
const std::uint32_t last_allowed = 1048576 + 16496 - 524288;
for (const std::uint32_t count :
{
last_allowed, last_allowed + 1
})
{
CAPTURE(count)
std::vector<std::uint8_t> input = {'['};
append_valueless_array(input, 'Z', 524288);
input.insert(input.end(), {'[', '$', 'U', '#', 'l', 0x00, 0x00, 0x10, 0x00});
input.insert(input.end(), 4096, 0x01);
append_valueless_array(input, 'Z', count);
input.push_back(']');
ValuelessEventCounter sax;
const bool accepted = count == last_allowed;
CHECK(json::sax_parse(input, &sax, json::input_format_t::ubjson) == accepted);
CHECK(sax.error_id == (accepted ? 0 : 408));
CHECK(sax.values == 524288u + 4096u + (accepted ? count : 0u));
}
// an array of 1048576 values on its own is accepted as before
const std::vector<std::uint8_t> single = {'[', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
// (the handler ends the parse at the first value, so that the million
// values are not read)
ValuelessEventCounter sax;
sax.stop_after = 1;
CHECK(!json::sax_parse(single, &sax, json::input_format_t::ubjson));
CHECK(sax.error_id == 0);
CHECK(sax.values == 1);
}
SECTION("to_ubjson writes the typed form up to the base only")
{
// 1048 rows of 1000 values fit under the base of 1048576, the 1049th
// would not: its elements are written one by one. Written all typed,
// 1100 rows of 6 bytes would be refused by a reader (1075 rows are the
// most it accepts). The rows hold null, true or false, and the
// fallback does not depend on which
std::vector<std::uint8_t> encoded;
{
json table = json::array();
for (int row = 0; row < 1100; ++row)
{
const json cell = row % 3 == 0 ? json(nullptr) : json(row % 3 == 1);
table.push_back(json(1000, cell));
}
encoded = json::to_ubjson(table, true, true);
}
// outer array: `[$[#I` and the count (7 bytes), then 1048 typed rows of 6
// bytes (`$Z#I` and two bytes) and 52 rows of `#I`, two bytes, and 1000 `Z`
CHECK(encoded.size() == 7 + (1048 * 6) + (52 * (4 + 1000)));
// the output is read back
ValuelessEventCounter sax;
CHECK(json::sax_parse(encoded, &sax, json::input_format_t::ubjson));
CHECK(sax.error_id == 0);
CHECK(sax.values == 1100u * 1000u);
}
SECTION("BJData is not affected")
{
// BJData has no optimized arrays of these types, and the writer does
// not charge them
json table = json::array();
for (int row = 0; row < 3; ++row)
{
table.push_back(json(10, nullptr));
}
const auto encoded = json::to_bjdata(table, true, true);
CHECK(json::from_bjdata(encoded) == table);
const std::vector<std::uint8_t> input = {'[', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
CHECK_THROWS_AS(_ = json::from_bjdata(input), json::parse_error&);
}
SECTION("an object of a valueless type needs a key for every value")
{
// an object member costs a key, so the count is bounded by the input
// and is not charged to the budget
const std::vector<std::uint8_t> input = {'{', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x00};
ValuelessEventCounter sax;
CHECK(!json::sax_parse(input, &sax, json::input_format_t::ubjson));
CHECK(sax.error_id == 110);
CHECK(sax.values == 0);
}
}
TEST_CASE("issue #5405 - array reserve for definite-length UBJSON arrays")
{
#if !defined(JSON_NOEXCEPTION)