Memory Overhead is Too High (10x or more)

[Meerkov]

Context:
I was parsing a 700MB json file, in Visual Studio 2019. The process took almost 9GB of memory! This at first looks like a 10x overhead.

We all know that std::strings have on the order of 2x overhead, so this doesn't seem to explain it.

All std::TreeNode<basic_string, basic_json> are showing up at taking exactly 140 bytes, which seems excessive. Since they are all the same size, it seems likely the keys are all being allocated much larger than necessary, perhaps. Keys in maps tend to be much shorter than the data that is being mapped to.

Possible solution is to move away from std::map ?

For reference, allocating strings looks like it's about 50% of memory, and treenodes are about 25% of memory in my instance.

[abrownsword]

We all know that std::strings have on the order of 2x overhead, so this doesn't seem to explain it.

This is highly dependent upon string length. If all your strings are 1 character, then overhead is typically 20-30x. If your strings are 1000 characters, then overhead is more like 2%. Most std::string implementations have an allocation-avoidance path below a certain length (depending upon the size of the std::string itself, which is usually about 24-32 bytes on 64-bit systems), whereby they store the string in the std::string structure. So the optimal string length is around 20-24 bytes.

All std::TreeNode<basic_string, basic_json> are showing up at taking exactly 140 bytes, which seems excessive

Yes, that does seem like too much. Worth investigating. My experience with std::map is that its overhead per entry is roughly 3 pointers worth, plus the memory allocator overhead and padding. On 64-bit systems, that is again 24-32 bytes. That plus the string for the key, means the nodes should be roughly 64 bytes plus the size of the basic_json. Is the basic_json around 70-80 bytes? That seems like more than it should be as, IIRC, it is a union of the possible types -- string, map, vector being the large ones. Plus a few bits for the type information. Best case scenario (on 64 bit systems) is going to be roughly 96 bytes though, so only about 40-45% smaller.

Doing better than the STL containers is actually quite a challenge unless you are willing to sacrifice generality and functionality fairly substantially. On 64-bit systems some cunning memory management could let you use smaller offsets in place of pointers, but this is a lot of effort (and bugs) for what is likely going to be no more than a 2-3x improvement. The win on 32-bit systems will be less. That's a pretty steep maintenance and generality cost, for relatively modest gains. Doesn't mean its not worth doing, but not to be undertaken lightly.

For large piles of json data you really should consider whether a DOM representation is even the right choice. It may simply be better to parse it SAX style and keep it around in its text form to reparse as-needed. Or keep pointers/offsets into the image and partially re-parse when needed.

[Meerkov]

For what it's worth, I think it's a universal truth that keys into the JSON will be smaller than 100 characters. The data could be very long, but optimizing the storage of the keys in the key-value pairs could be a fairly substantial gain across all use-cases.

In terms of the size of basic_json, the documentation claims:

Each JSON object has an overhead of one pointer (the maximal size of a union) and one enumeration element (1 byte).

I can confirm that it looks like the storage for the single json value is indeed just a pointer, defined here: https://github.com/nlohmann/json/blob/develop/single_include/nlohmann/json.hpp#L13671

That should be only 9 total bytes for basic_json (plus all the memory that it points to, of course). It's possible that Visual Studio is including the de-referenced string in the allocation calculation for the TreeNode, but that wouldn't explain why the value was a constant 140, instead of variable (I have some string data that are 1kb, though most are small).

[abrownsword]

BTW, how are you measuring the size of each TreeNode?

[Meerkov]

I'm using the Memory Profiler in Visual Studio, and running through the allocation output. That gives me the function call stack, and the size of the allocated object.

[nickaein]

Relevant: #1516

Have you tried building in Release mode? That sometimes can make a huge difference.

[Meerkov]

I would expect Release mode to improve speed, but not memory. At least not by 10x memory. That said, I switched over to RapidJson, so I don't have the code to run this again at this moment. The file I was using was this large file, if you'd like to reproduce the issue for analysis.

https://archive.scryfall.com/json/scryfall-all-cards.json

[nickaein]

Just as a reference, the memory consumption was ~2.79GB for the following code built by GCC 7.4 x86/64 with -O3 flag. I currently don't have access to msvc compiler to test it but I imagine it can be around the same number with a suitable configuration and compiler/build flags.

#include <string>
#include <iostream>
#include <fstream>
#include "json.hpp"

using namespace std;

int main() {

    ifstream fin("scryfall-all-cards.json");
    if(!fin)
    {
        cout << "Failed to open file." << endl;
        return 1;
    }
    
    auto json = nlohmann::json::parse(fin);

    cout << "json parsed. Press any key to exit" << endl;
    string dummy;
    cin >> dummy;
    
    return 0;
}

[gregmarr]

Release mode will definitely improve memory on Visual C++, as the debug versions of the containers include extra information to aid with iterator debugging issues. FYI, TreeNode is an internal implementation detail of std::map on VC++. You'd want to look at the size of basic_json, not the size of TreeNode<basic_string, basic_json> to see what memory size comes from the library itself. As a benchmark, you could compare it to the size of TreeNode<char, char> which should be the smallest possible TreeNode.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[igormcoelho]

Already posted in other issue, but reposting here...

I just got into this same issue, some 1.6GB json wouldn't load (even with 10GB memory).
I tried other libraries, but couldn't find any solution, so I coded my own solution here: https://github.com/igormcoelho/vastjson/
If this is useful to someone in the future, it just indexes top-level entries as strings (that consumes less space), and put into nlohmann::json when necessary. It worked and I could finally process my big json (with around 2.3GB RAM)!

[nlohmann]

Sounds great! Is it something that is compatible to this library? Would be great to have that feature.