mirror of https://github.com/nlohmann/json.git synced 2026-10-01 14:15:50 +01:00

Files

T

Niels Lohmann f7972970a4 Throw instead of writing MessagePack lengths beyond UINT32_MAX (#5584 )

* Throw instead of writing MessagePack lengths beyond UINT32_MAX

MessagePack stores the length of a string, binary value, array, or
object in at most 32 bits. For a larger value, to_msgpack wrote no length
at all, so the output could not be read back. It now throws
out_of_range.412, which BSON already uses for its 32-bit length fields.

The check lives in one function, so each length is written by an
if/else chain that ends in a plain else, without a condition that can
never be false. It is tested with string and binary types that report a
size beyond UINT32_MAX without allocating it, like the BSON tests do.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix the CI failures of the MessagePack length check

- mark to_msgpack_length's value as used when exceptions are disabled
  (-Wunused-parameter, misc-unused-parameters)
- put "Exception safety" before "Exceptions" in to_msgpack.md, as the
  documentation style check requires
- create the test's string value from its type: constructing it from a
  beyond_uint32_string_t considers the std::filesystem::path conversion,
  which libstdc++ 10 reports as ambiguous for a class derived from
  std::string (clang 13)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Skip the MessagePack string length test for clang with libstdc++ 10

C++17 builds consider the std::filesystem::path conversion for the
string type, and with clang and libstdc++ 10 that conversion is
ambiguous for a class derived from std::string. Creating the value from
its type did not avoid it, since any basic_json with that string type
instantiates the check. The binary and ext cases are still tested there.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Keep the MessagePack string test type and its alias in one block

astyle indented the alias oddly when it had an #ifdef of its own after
the binary alias; declare it right after the string type, in the same
block.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

2026-09-27 14:28:16 +02:00

8.1 KiB

Raw Blame History

MessagePack

MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.

!!! abstract "References"

- [MessagePack website](https://msgpack.org)
- [MessagePack specification](https://github.com/msgpack/msgpack/blob/master/spec.md)

Serialization

The library uses the following mapping from JSON values types to MessagePack types according to the MessagePack specification:

JSON value type	value/range	MessagePack type	first byte
null	`null`	nil	0xC0
boolean	`true`	true	0xC3
boolean	`false`	false	0xC2
number_integer	-9223372036854775808..-2147483649	int64	0xD3
number_integer	-2147483648..-32769	int32	0xD2
number_integer	-32768..-129	int16	0xD1
number_integer	-128..-33	int8	0xD0
number_integer	-32..-1	negative fixint	0xE0..0xFF
number_integer	0..127	positive fixint	0x00..0x7F
number_integer	128..255	uint 8	0xCC
number_integer	256..65535	uint 16	0xCD
number_integer	65536..4294967295	uint 32	0xCE
number_integer	4294967296..18446744073709551615	uint 64	0xCF
number_unsigned	0..127	positive fixint	0x00..0x7F
number_unsigned	128..255	uint 8	0xCC
number_unsigned	256..65535	uint 16	0xCD
number_unsigned	65536..4294967295	uint 32	0xCE
number_unsigned	4294967296..18446744073709551615	uint 64	0xCF
number_float	any value representable by a float	float 32	0xCA
number_float	any value NOT representable by a float	float 64	0xCB
string	length: 0..31	fixstr	0xA0..0xBF
string	length: 32..255	str 8	0xD9
string	length: 256..65535	str 16	0xDA
string	length: 65536..4294967295	str 32	0xDB
array	size: 0..15	fixarray	0x90..0x9F
array	size: 16..65535	array 16	0xDC
array	size: 65536..4294967295	array 32	0xDD
object	size: 0..15	fix map	0x80..0x8F
object	size: 16..65535	map 16	0xDE
object	size: 65536..4294967295	map 32	0xDF
binary	size: 0..255	bin 8	0xC4
binary	size: 256..65535	bin 16	0xC5
binary	size: 65536..4294967295	bin 32	0xC6

!!! success "Complete mapping"

The mapping is **complete** in the sense that any JSON value type can be converted to a MessagePack value.

Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.

!!! warning "Size constraints"

The following values can **not** be converted to a MessagePack value:

  - strings with more than 4294967295 bytes
  - byte strings with more than 4294967295 bytes
  - arrays with more than 4294967295 elements
  - objects with more than 4294967295 elements

Serializing such a value throws [`out_of_range.412`](../../home/exceptions.md#jsonexceptionout_of_range412).

!!! info "NaN/infinity handling"

`NaN`, `Infinity`, and `-Infinity` are serialized as a MessagePack float 32 (type 0xCA, 5 bytes total),
regardless of magnitude, in contrast to the [dump](../../api/basic_json/dump.md) function which serializes NaN
or Infinity to `null`.

!!! note

Prior to version 3.13.0, NaN and Infinity were instead serialized as a MessagePack float 64 (type 0xCB, 9 bytes
total), because the check used to select the smaller float 32 encoding compared magnitudes with NaN, which is
always `false` and caused the float 32 path to be skipped.

??? example

```cpp
--8<-- "examples/to_msgpack.cpp"
```

Output:

```c
--8<-- "examples/to_msgpack.output"
```

Deserialization

The library maps MessagePack types to JSON value types as follows:

MessagePack type	JSON value type	first byte
positive fixint	number_unsigned	0x00..0x7F
fixmap	object	0x80..0x8F
fixarray	array	0x90..0x9F
fixstr	string	0xA0..0xBF
nil	`null`	0xC0
false	`false`	0xC2
true	`true`	0xC3
float 32	number_float	0xCA
float 64	number_float	0xCB
uint 8	number_unsigned	0xCC
uint 16	number_unsigned	0xCD
uint 32	number_unsigned	0xCE
uint 64	number_unsigned	0xCF
int 8	number_integer	0xD0
int 16	number_integer	0xD1
int 32	number_integer	0xD2
int 64	number_integer	0xD3
str 8	string	0xD9
str 16	string	0xDA
str 32	string	0xDB
array 16	array	0xDC
array 32	array	0xDD
map 16	object	0xDE
map 32	object	0xDF
bin 8	binary	0xC4
bin 16	binary	0xC5
bin 32	binary	0xC6
ext 8	binary	0xC7
ext 16	binary	0xC8
ext 32	binary	0xC9
fixext 1	binary	0xD4
fixext 2	binary	0xD5
fixext 4	binary	0xD6
fixext 8	binary	0xD7
fixext 16	binary	0xD8
negative fixint	number_integer	0xE0-0xFF

!!! info

Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.

!!! warning "UTF-8 validation of string values"

The MessagePack specification requires `str` values (`fixstr`, `str 8`, `str 16`, `str 32`) to be valid UTF-8.
This library validates the bytes of every such string (object keys included) at decode time and rejects
ill-formed UTF-8 with a [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113) exception (or,
with `allow_exceptions` set to `false`, a discarded value), rather than only failing later when the resulting
value is dumped. `bin`/`ext`/`fixext` values are unaffected and are never validated, since they are not required
to hold text.

??? example

```cpp
--8<-- "examples/from_msgpack.cpp"
```

Output:

```json
--8<-- "examples/from_msgpack.output"
```

8.1 KiB Raw Blame History

MessagePack

Serialization

Deserialization

8.1 KiB

Raw Blame History