The performance of DOM Parser and Schema-Based Parser.

[ZhaiMo15]

I've been testing the performance of Simdjson recently. The basic test is similar to default test, using twitter.json, as below:

@Benchmark
 public int recordSimdjson() {
 Set<String> defaultUsers = new HashSet<>();
 TwitterRecord twitter = simdJsonParser.parse(buffer, buffer.length, TwitterRecord.class);
 for (StatusRecord status : twitter.statuses()) {
 UserRecord user = status.user();
 if (user.default_profile()) {
 defaultUsers.add(user.screen_name());
 }
 }
 return defaultUsers.size();
 }

 @Benchmark
 public int JsonValueSimdjson() {
 JsonValue simdJsonValue = simdJsonParser.parse(buffer, buffer.length);
 Set<String> defaultUsers = new HashSet<>();
 Iterator<JsonValue> tweets = simdJsonValue.get("statuses").arrayIterator();
 while (tweets.hasNext()) {
 JsonValue tweet = tweets.next();
 JsonValue user = tweet.get("user");
 if (user.get("default_profile").asBoolean()) {
 defaultUsers.add(user.get("screen_name").asString());
 }
 }
 return defaultUsers.size();
 }

 @Benchmark
 public int recordJackson() throws IOException {
 Set<String> defaultUsers = new HashSet<>();
 TwitterRecord twitter = objectMapper.readValue(buffer, TwitterRecord.class);
 for (StatusRecord status : twitter.statuses()) {
 UserRecord user = status.user();
 if (user.default_profile()) {
 defaultUsers.add(user.screen_name());
 }
 }
 return defaultUsers.size();
 }

 record UserRecord(boolean default_profile, String screen_name) {
 }

 record StatusRecord(UserRecord user) {
 }

 record TwitterRecord(List<StatusRecord> statuses) {
 }

What's different is I shrunk the size of statuses, default is 101, I tested 101, 51, and 1 respectively, the result is below:
size 101:

size 51:

size 1:

What's more, I changed the depth of test, the default is 3 and I changed it to 2, as below:

@Benchmark
 public int recordSimdjson() {
 Set<Object> defaultUsers = new HashSet<>();
 TwitterRecord twitter = simdJsonParser.parse(buffer, buffer.length, TwitterRecord.class);
 for (StatusRecord status : twitter.statuses()) {
 long id = status.id();
 String text = status.text();
 defaultUsers.add(id);
 defaultUsers.add(text);
 }
 return defaultUsers.size();
 }

 @Benchmark
 public int JsonValueSimdjson() {
 JsonValue simdJsonValue = simdJsonParser.parse(buffer, buffer.length);
 Set<Object> defaultUsers = new HashSet<>();
 Iterator<JsonValue> tweets = simdJsonValue.get("statuses").arrayIterator();
 while (tweets.hasNext()) {
 JsonValue tweet = tweets.next();
 JsonValue id = tweet.get("id");
 JsonValue text = tweet.get("text");
 defaultUsers.add(id.asLong());
 defaultUsers.add(text.asString());
 }
 return defaultUsers.size();
 }

 @Benchmark
 public int recordJackson() throws IOException {
 Set<Object> defaultUsers = new HashSet<>();
 TwitterRecord twitter = objectMapper.readValue(buffer, TwitterRecord.class);
 for (StatusRecord status : twitter.statuses()) {
 long id = status.id();
 String text = status.text();
 defaultUsers.add(id);
 defaultUsers.add(text);
 }
 return defaultUsers.size();
 }

 record StatusRecord(long id, String text) {
 }

 record TwitterRecord(List<StatusRecord> statuses) {
 }

The results are:
size 101:

size 51:

size 1:

Here are my questions:

1. The performance of Simdjson is not always faster than jackson? The shorter the JSON, the worse of Simdjson? If my JSON is short, I'd better not use simdjson?
2. DOM Parser vs Schema-Based Parser, the performance also depends on size of JSON? My first thought is Schema-Based is faster.

[piotrrzysko]

Would you mind sharing the hardware and Java version you used to run these benchmarks? The output from lscpu and java -version should be sufficient. Also, it'd be great if you could share the benchmarks on a branch (or as a github repo). This would help me run them myself and profile the code.

[piotrrzysko]

Also, what do you mean by:

What's more, I changed the depth of test, the default is 3 and I changed it to 2, as below:
?

Both attached snippets look the same.

[ZhaiMo15]

Consider the JSON as a tree, the depth I meant is the depth of the node of the tree. For example,

// depth 1
{
 a: 1,
 b: 2
}

// depth 2
{
 a: {
 b: 1,
 c: 2
 }
}

The V2 benchmark visited the data of "statuses.user.screen_name"(depth 3) while V3 is "statuses.id"(depth 2).

Both attached snippets look the same.

I'm sorry, I paste the wrong snippets.

[ZhaiMo15]

And I re-run the benchmarks in a more stable environment, the results are similar.
My hardware:

And the java version:

The benchmarks I ran are in:
https://github.com/ZhaiMo15/simdjson-java/tree/performanceTest

[piotrrzysko]

Thanks for the update. I run your benchmarks on my desktop using two versions of Java (18 and 21). I got the following results:

JDK 21.0.1, OpenJDK 64-Bit Server VM, 21.0.1+12-LTS

Benchmark (fileName) Mode Cnt Score Error Units
TwitterBenchmarkV2.JsonValueSimdjson /twitter.json thrpt 5 1272.131 ± 77.097 ops/s
TwitterBenchmarkV2.recordJackson /twitter.json thrpt 5 577.435 ± 14.169 ops/s
TwitterBenchmarkV2.recordSimdjson /twitter.json thrpt 5 1963.116 ± 44.895 ops/s
TwitterBenchmarkV3.JsonValueSimdjson /twitter.json thrpt 5 1198.611 ± 63.058 ops/s
TwitterBenchmarkV3.recordJackson /twitter.json thrpt 5 748.944 ± 6.980 ops/s
TwitterBenchmarkV3.recordSimdjson /twitter.json thrpt 5 1976.876 ± 241.907 ops/s

TwitterBenchmarkV2.JsonValueSimdjson /twitter50.json thrpt 5 2435.258 ± 194.432 ops/s
TwitterBenchmarkV2.recordJackson /twitter50.json thrpt 5 1125.062 ± 2.477 ops/s
TwitterBenchmarkV2.recordSimdjson /twitter50.json thrpt 5 3787.566 ± 51.370 ops/s
TwitterBenchmarkV3.JsonValueSimdjson /twitter50.json thrpt 5 2429.744 ± 153.990 ops/s
TwitterBenchmarkV3.recordJackson /twitter50.json thrpt 5 1427.326 ± 6.044 ops/s
TwitterBenchmarkV3.recordSimdjson /twitter50.json thrpt 5 3883.897 ± 34.568 ops/s

TwitterBenchmarkV2.JsonValueSimdjson /twitter1.json thrpt 5 175359.845 ± 1522.490 ops/s
TwitterBenchmarkV2.recordJackson /twitter1.json thrpt 5 69225.339 ± 628.405 ops/s
TwitterBenchmarkV2.recordSimdjson /twitter1.json thrpt 5 83146.423 ± 654.256 ops/s
TwitterBenchmarkV3.JsonValueSimdjson /twitter1.json thrpt 5 181205.520 ± 269.705 ops/s
TwitterBenchmarkV3.recordJackson /twitter1.json thrpt 5 96834.366 ± 268.782 ops/s
TwitterBenchmarkV3.recordSimdjson /twitter1.json thrpt 5 102403.918 ± 625.203 ops/s

JDK 18.0.2.1, OpenJDK 64-Bit Server VM, 18.0.2.1+1

Benchmark (fileName) Mode Cnt Score Error Units
TwitterBenchmarkV2.JsonValueSimdjson /twitter.json thrpt 5 1152.353 ± 258.483 ops/s
TwitterBenchmarkV2.recordJackson /twitter.json thrpt 5 565.322 ± 17.139 ops/s
TwitterBenchmarkV2.recordSimdjson /twitter.json thrpt 5 1759.905 ± 41.637 ops/s
TwitterBenchmarkV3.JsonValueSimdjson /twitter.json thrpt 5 1122.889 ± 316.755 ops/s
TwitterBenchmarkV3.recordJackson /twitter.json thrpt 5 716.739 ± 6.083 ops/s
TwitterBenchmarkV3.recordSimdjson /twitter.json thrpt 5 1824.830 ± 19.503 ops/s

TwitterBenchmarkV2.JsonValueSimdjson /twitter50.json thrpt 5 2338.579 ± 58.298 ops/s
TwitterBenchmarkV2.recordJackson /twitter50.json thrpt 5 1094.865 ± 2.898 ops/s
TwitterBenchmarkV2.recordSimdjson /twitter50.json thrpt 5 3333.782 ± 55.180 ops/s
TwitterBenchmarkV3.JsonValueSimdjson /twitter50.json thrpt 5 2243.374 ± 9.085 ops/s
TwitterBenchmarkV3.recordJackson /twitter50.json thrpt 5 1419.183 ± 17.172 ops/s
TwitterBenchmarkV3.recordSimdjson /twitter50.json thrpt 5 3475.266 ± 132.370 ops/s

TwitterBenchmarkV2.JsonValueSimdjson /twitter1.json thrpt 5 164348.941 ± 9737.617 ops/s
TwitterBenchmarkV2.recordJackson /twitter1.json thrpt 5 68143.603 ± 257.766 ops/s
TwitterBenchmarkV2.recordSimdjson /twitter1.json thrpt 5 81290.062 ± 1121.079 ops/s
TwitterBenchmarkV3.JsonValueSimdjson /twitter1.json thrpt 5 170856.785 ± 1570.948 ops/s
TwitterBenchmarkV3.recordJackson /twitter1.json thrpt 5 94823.108 ± 256.773 ops/s
TwitterBenchmarkV3.recordSimdjson /twitter1.json thrpt 5 105020.635 ± 1673.175 ops/s

In general, Java 21 usually performs better, which is not surprising. However, the problem you've described is still valid. Let me go through your questions to make sure we are on the same page:

The performance of Simdjson is not always faster than jackson? The shorter the JSON, the worse of Simdjson? If my JSON is short, I'd better not use simdjson?

In this question, you are referring to the poor performance of the schema-based parser in comparison to Jackson for shorter JSONs. Overall, in all the above cases, some version of simdjson beats Jackson.

DOM Parser vs Schema-Based Parser, the performance also depends on size of JSON? My first thought is Schema-Based is faster.

This question is strictly related to the previous one, as the schema-based parser again performs unexpectedly poorly.

If my interpretation of the benchmark results and your concerns is correct, then we can narrow down the problem to the performance of the schema-based parser. I've profiled it while running the TwitterBenchmarkV2 for twitter1.json. This is what I got:

The flamegraph clearly shows that Java reflection is the culprit of the poor performance. Simdjson clears its internal cache of resolved classes every time the parse method is called. After commenting out the line in which the cache is cleared, I got the following results:

Benchmark (fileName) Mode Cnt Score Error Units
TwitterBenchmarkV2.JsonValueSimdjson /twitter1.json thrpt 5 175317.529 ± 2304.856 ops/s
TwitterBenchmarkV2.JsonValueSimdjson:·async /twitter1.json thrpt NaN ---
TwitterBenchmarkV2.recordJackson /twitter1.json thrpt 5 67562.644 ± 1220.035 ops/s
TwitterBenchmarkV2.recordJackson:·async /twitter1.json thrpt NaN ---
TwitterBenchmarkV2.recordSimdjson /twitter1.json thrpt 5 265694.392 ± 11438.461 ops/s
TwitterBenchmarkV2.recordSimdjson:·async /twitter1.json thrpt NaN ---

I suggest that you comment out this line and rerun the benchmarks in your environment. This is not the ultimate solution, of course. I just want to make sure that we are on the same page and that you don't see any other unexpected disparities between the parsers in terms of performance.

[ZhaiMo15]

I suggest that you comment out this line and rerun the benchmarks in your environment.

We are now on the same page! I rerun the benchmarks with commenting out the classResolver.reset(), the results are similar to you, as well as the flamegraph.

This is not the ultimate solution, of course.

BTW, is there a problem if commenting out the classResolver.reset() for good? Maybe clear cache in a different place?

[piotrrzysko]

If we comment it out without changing anything else, there can be a problem because the cache will grow infinitely. In some cases, this is acceptable because the cache can contain as many entries as there are classes in the application.

I'll need to think about it. Perhaps the cache needs to have a more sophisticated eviction policy (LRU?).

[ZhaiMo15]

Additionally, consider a real situation instead of benchmark, I believe the classResolver is different when parsing different JSON. So comment out the classResolver.reset() can increase the score of benchmark, but if each JSON would be parsed once, simdjson is still not good enough when JSON is short?

[ZhaiMo15]

Perhaps the cache needs to have a more sophisticated eviction policy (LRU?).

Perhaps the cache needs a fixed size(configurable)? Regardless of the eviction policy.

[piotrrzysko]

So comment out the classResolver.reset() can increase the score of benchmark, but if each JSON would be parsed once, simdjson is still not good enough when JSON is short?

This is the penalty for using reflection, so you would need to pay it regardless of which parser you use if the parser relies on it. I have an idea on how to replace reflection with an alternative approach, but it requires more research.

Also, I wonder how realistic this problem is. How many different schemas can you have in your system?

[ZhaiMo15]

I have an idea on how to replace reflection with an alternative approach, but it requires more research.

Great! Looking forward to it.

I wonder how realistic this problem is.

TBH, I've no idea. IDK the JSON size distribution in real world. In my case, I do have some small JSON.

[piotrrzysko]

TBH, I've no idea. IDK the JSON size distribution in real world. In my case, I do have some small JSON.

This is understandable, but I wasn't asking about the size of the JSON. Your question was:

if each JSON would be parsed once, simdjson is still not good enough when JSON is short?

So, I assumed that you have a situation where there are many different types of JSON schemas, and every time you parse a JSON, you use a different schema. In such a scenario, the cache is useless because the parser cannot reuse the classes that are already in the cache. However, I cannot think of a scenario where you have, say, a million different schemas and use a different one each time. Is this your case?

[ZhaiMo15]

I get your point. The number of different schemas should not be large.
However,

<T> T walkDocument(byte[] padded, int len, Class<T> expectedType) {
 jsonIterator.init(padded, len);
 classResolver.reset();

I think in current code, the cache would be cleared even the expectedType is always same?

if each JSON would be parsed once, simdjson is still not good enough when JSON is short?

What I mean JSON is JSON strings (i.e byte[] buffer). If I have lots of JSON string(let's say 100 different JSON strings), but few schemas(let's say only 1). When I parse 100 JSON strings in sequence, even though schema is same, the cache would still be cleared 100 times? And this would meet the performance penalty above.

[piotrrzysko]

Yes, exactly. This is why I mentioned that replacing this simple eviction policy with a more sophisticated one could be a good improvement. The new policy would keep already resolved schemas between parse method calls.

[ZhaiMo15]

Just for interest's sake,

The flamegraph clearly shows that Java reflection is the culprit of the poor performance.

Jackson also use reflection, right? Why doesn't it show the poor performance?