Write BSON in linear time, without recursing per nesting level

to_bson() had two problems with nested values:

- It recursed once per nesting level, so a value nested deeply enough -
  100,000 levels on an 8 MiB stack - exhausted the call stack and
  terminated the process, although parse() accepts such values without
  complaint.
- BSON prefixes every document and array with its length. The writer
  computed that length by walking the entire value below it, again for
  every nested document it wrote, which made serializing O(size x depth).
  A 200-level document took 30 ms instead of 1.

Both passes are now iterative, and each length is computed exactly once:

- calc_bson_sizes() computes the length of every document and array in
  one pass, each from the lengths of its entries, into a table ordered
  the way they are written.
- write_bson_document() then writes the document, taking each length from
  the table.

Everything observable is unchanged, as a differential test against
develop confirms byte for byte:

- The same bytes are written.
- A key containing U+0000 still throws out_of_range.409 for the same
  first key, with the same diagnostics path, before anything is written.
- A document too large for BSON still throws out_of_range.412 before
  anything is written.
- A binary subtype above 255 still throws out_of_range.415 after the
  same partial output.

Only the enclosing objects and arrays are kept on a stack, so a flat
document allocates nothing for it. Measured against develop (clang -O3,
median of 201 runs): flat objects unchanged, flat arrays 37% faster (the
array length was computed twice), a nested 3,000-object document 2x
faster, a 200-level document 33x faster.

to_bson.md documented the quadratic complexity since #5334; it is linear
again.

Fixes #5392 for BSON, and #5308.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

tests/src/unit-bson.cpp

@@ -49,7 +49,7 @@ using huge_binary_json = nlohmann::basic_json <
 // for *object keys* (e.g. "s" or "nested" below). Only the designated test
 // value is meant to lie about its size - if every huge_string_t (including
 // keys) reported a huge size, the running totals computed while walking the
-// BSON document (see calc_bson_object_size & friends in binary_writer.hpp)
+// BSON document (see calc_bson_sizes in binary_writer.hpp)
 // would need more than 32 bits, and on platforms where std::size_t is only
 // 32 bits wide that arithmetic would silently wrap around, producing wrong
 // (or even unguarded) lengths. The fake size is therefore opt-in via
@@ -1697,3 +1697,67 @@ TEST_CASE("BSON roundtrips" * doctest::skip())
         }
     }
 }
+
+TEST_CASE("BSON: deeply nested values")
+{
+    SECTION("documents and arrays round-trip at every depth")
+    {
+        // nested documents and arrays, with siblings on every level, so
+        // every length prefix covers entries of both kinds
+        json value = "leaf";
+        for (std::size_t depth = 0; depth <= 300; ++depth)
+        {
+            CAPTURE(depth);
+            const json document = {{"value", value}, {"n", depth}};
+            CHECK(json::from_bson(json::to_bson(document)) == document);
+
+value = depth % 2 == 0 ? json{{"a", std::move(value)}, {"b", {1, "x"}}} :
+            json::array({std::move(value), depth, json::object()});
+        }
+    }
+
+    SECTION("a key containing U+0000 is rejected before anything is written")
+    {
+        json value = json::object({{std::string("bad\0key", 7), 1}});
+        for (std::size_t depth = 0; depth < 200; ++depth)
+        {
+            value = json{{"a", {{"b", 1}}}, {"z", std::move(value)}};
+        }
+        std::vector<std::uint8_t> output;
+        CHECK_THROWS_AS(json::to_bson(value, output), json::out_of_range&);
+        CHECK(output.empty());
+    }
+
+    SECTION("values nested too deeply for the call stack (#5392)")
+    {
+        // serializing recursed once per nesting level, and computed every
+        // nested document's length by walking everything below it again.
+        // The values are only parsed, serialized and walked, never copied or
+        // compared, since those recurse too.
+        const std::size_t depth = 100000;
+        for (const bool objects :
+                {
+                    false, true
+                })
+        {
+            CAPTURE(objects);
+            std::string text = "{\"a\":";
+            for (std::size_t i = 0; i < depth; ++i)
+            {
+                text += objects ? "{\"a\":" : "[";
+            }
+            text += "1";
+            text.append(depth, objects ? '}' : ']');
+            text += "}";
+
+            const auto bson = json::to_bson(json::parse(text));
+            const auto result = json::from_bson(bson);
+            const json* p = &result.at("a");
+            for (std::size_t i = 0; i < depth; ++i)
+            {
+                p = objects ? &p->at("a") : &p->at(0);
+            }
+            CHECK(*p == 1);
+        }
+    }
+}