Diff deeply nested values without recursing per nesting level

diff() descended into both values once per nesting level, and compared
them with operator== on every level on the way, which recurses as well.
Values nested deeply enough - 25,000 levels on an 8 MiB stack - exhausted
the call stack and terminated the process, although parse() accepts
them without complaint. On such a chain the per-level comparisons and
path strings also made diff() quadratic in time and memory.

Both the recursion and operator== only descend as far as the source is
nested. So diff() first checks, recursing at most diff_depth_limit()
(128) levels, whether the source is nested more deeply than that. If not
- all but a vanishing minority of values - the recursive algorithm
diffs it exactly as before, now as diff_recursively(). Otherwise
diff_iteratively() walks the two values on an explicit stack, emitting
the same operations in the same order. It does not compare arrays and
objects with operator== up front (equal ones yield no operations
anyway), keeps the path in one buffer instead of a new string per
level, and hands every subtree that is not nested too deeply back to
diff_recursively(), so equal parts are still skipped quickly.

The check costs one pass over the source. On a 3,000-object document
that is about 30% of diffing two equal values (which is just an
operator== call), about 10% of diffing values that differ in a few
places, and noise when arrays change length. Once operator== no longer
recurses (#5390), the check can go.

Tests check that the patch reproduces the target at every depth up to
300, for json and ordered_json, including reordered members. They also
check the exact operation for a difference deep inside, and diff values
nested 100,000 levels deep.

Fixes #5393 for diff().

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

tests/src/unit-json_patch.cpp

@@ -15,8 +15,64 @@ using nlohmann::json;
 #endif
 
 #include <fstream>
+#include <string>
 #include "make_test_data_available.hpp"
 
+namespace
+{
+// alternating objects and arrays nested `depth` levels deep, with members that
+// depend on `variant` at some levels, so diffing two variants yields
+// operations on many levels: replacing the innermost value, adding, removing,
+// and (for ordered_json) reordering members, and changing array lengths
+template<typename BasicJsonType>
+BasicJsonType nested(const std::size_t depth, const int variant)
+{
+    BasicJsonType value = variant;
+    for (std::size_t i = 0; i < depth; ++i)
+    {
+        if (i % 2 == 0)
+        {
+            BasicJsonType object = BasicJsonType::object();
+            if ((i + static_cast<std::size_t>(variant)) % 7 == 0)
+            {
+                object["x"] = i;
+            }
+            if (variant == 2 && i % 11 == 0)
+            {
+                object["z"] = "z";
+            }
+            object["a"] = std::move(value);
+            if (variant == 1 && i % 5 == 0)
+            {
+                object["y"] = 1;
+            }
+            value = std::move(object);
+        }
+        else
+        {
+            BasicJsonType array = BasicJsonType::array({std::move(value)});
+            if ((i + static_cast<std::size_t>(variant)) % 3 == 0)
+            {
+                array.push_back(i);
+            }
+            value = std::move(array);
+        }
+    }
+    return value;
+}
+
+// a path of `depth` reference tokens, as nested() nests its values
+std::string nested_path(const std::size_t depth)
+{
+    std::string path;
+    for (std::size_t i = depth; i > 0; --i)
+    {
+        path += (i - 1) % 2 == 0 ? "/a" : "/0";
+    }
+    return path;
+}
+} // namespace
+
 TEST_CASE("JSON patch")
 {
     SECTION("examples from RFC 6902")
@@ -1751,3 +1807,87 @@ TEST_CASE("JSON patch - diff emits array removals in descending index order")
         CHECK(source.patch(patch) == target);
     }
 }
+
+TEST_CASE("JSON patch: diff of deeply nested values")
+{
+    SECTION("the diff reproduces the target at every depth")
+    {
+        // every depth on either side of the nesting depth up to which diff()
+        // recurses (basic_json::diff_depth_limit(), 128)
+        for (std::size_t depth = 0; depth <= 300; ++depth)
+        {
+            CAPTURE(depth);
+            for (int from = 0; from < 3; ++from)
+            {
+                for (int to = 0; to < 3; ++to)
+                {
+                    CAPTURE(from);
+                    CAPTURE(to);
+                    const auto source = nested<json>(depth, from);
+                    const auto target = nested<json>(depth, to);
+                    const auto patch = json::diff(source, target);
+                    CHECK(source.patch(patch) == target);
+                    CHECK(patch.empty() == (from == to));
+
+                    const auto ordered_source = nested<nlohmann::ordered_json>(depth, from);
+                    const auto ordered_target = nested<nlohmann::ordered_json>(depth, to);
+                    CHECK(ordered_source.patch(nlohmann::ordered_json::diff(ordered_source, ordered_target)) == ordered_target);
+                }
+            }
+        }
+    }
+
+    SECTION("a difference only in the innermost value is one replace operation")
+    {
+        for (std::size_t depth = 0; depth <= 300; ++depth)
+        {
+            CAPTURE(depth);
+            json source = 1;
+            json target = 2;
+            for (std::size_t i = 0; i < depth; ++i)
+            {
+                source = i % 2 == 0 ? json::object({{"a", std::move(source)}}) : json::array({std::move(source)});
+                target = i % 2 == 0 ? json::object({{"a", std::move(target)}}) : json::array({std::move(target)});
+            }
+            CHECK(json::diff(source, target, "/root") == json::array({{{"op", "replace"}, {"path", "/root" + nested_path(depth)}, {"value", 2}}}));
+        }
+    }
+
+    SECTION("values nested too deeply for the call stack (#5393)")
+    {
+        // diff() used to recurse once per nesting level, and compared the
+        // values with operator== on every level. The values are only
+        // parsed and diffed, never copied or compared, since those recurse
+        // too.
+        const std::size_t depth = 100000;
+        for (const bool objects :
+                {
+                    false, true
+                })
+        {
+            CAPTURE(objects);
+            std::string source_text;
+            std::string target_text;
+            std::string equal_text;
+            std::string path;
+            for (std::size_t i = 0; i < depth; ++i)
+            {
+                source_text += objects ? "{\"a\":" : "[";
+                path += objects ? "/a" : "/0";
+            }
+            target_text = source_text + "2";
+            equal_text = source_text + "1";
+            source_text += "1";
+            const std::string closing(depth, objects ? '}' : ']');
+            const auto source = json::parse(source_text + closing);
+
+            const auto patch = json::diff(source, json::parse(target_text + closing));
+            REQUIRE(patch.size() == 1);
+            CHECK(patch[0]["op"] == "replace");
+            CHECK(patch[0]["path"] == path);
+            CHECK(patch[0]["value"] == 2);
+
+            CHECK(json::diff(source, json::parse(equal_text + closing)).empty());
+        }
+    }
+}