Draft replace rapidjson with simdjson by rok · Pull Request #47 · rok/arrow

rok · 2026-01-09T19:19:39Z

This is a discussion PR. It will likely be broken up in multiple PRs.

rok · 2026-01-28T18:56:51Z

@taepper this currently in a bit of a cmake limbo :).

taepper · 2026-01-28T18:58:51Z

@taepper this currently in a bit of a cmake limbo :).

Hehe :) it is quite the big change, I will also take a look tomorrow!

WillAyd

Cool nice work. Some suggestions to improve the meson integration, which hopefully can help while the cmake issues get figured out (?)

WillAyd · 2026-02-03T15:43:05Z

- 'RapidJSON',
- fallback: ['rapidjson', 'rapidjson_dep'],
- )
+ simdjson_dep = dependency('simdjson', fallback: ['simdjson', 'simdjson_dep'])


Suggested change
simdjson_dep = dependency('simdjson', fallback: ['simdjson', 'simdjson_dep'])
simdjson_dep = dependency('simdjson')
The fallback as written is not correct (the variable name in the simdjson module is actually still just simdjson), but I also don't think its required in this case

WillAyd · 2026-02-03T15:44:46Z

-source_hash = b9290a9a6d444c8e049bd589ab804e0ccf2b05dc5984a19ed5ae75d090064806
-source_fallback_url = https://apache.jfrog.io/artifactory/arrow/thirdparty/7.0.0/rapidjson-232389d4f1012dddec4ef84861face2d2ba85709.tar.gz
-patch_directory = rapidjson
+directory = simdjson-3.12.3


There's a wrap entry for 4.2.2 at this point - maybe worth using instead of 3.12? Or, if we wanted to stay in the 3 series, there's also 3.13

WillAyd · 2026-02-03T15:46:25Z

 # under the License.

-project('rapidjson', 'cpp')
+project('simdjson', 'cpp', version: '3.12.3', default_options: ['cpp_std=c++17'])


You should be able to remove this file altogether. subprojects/packagefiles contain user-specific overrides for packages (i.e. you want to create a meson.build file that is "better" than what you get upstream).
In the case of simdjson, we should probably trust the configuration provided by the wrapdb, unless we know better

rok · 2026-02-04T00:23:04Z

Cool nice work. Some suggestions to improve the meson integration, which hopefully can help while the cmake issues get figured out (?)

Thanks @WillAyd ! I've not had a chance to properly look into meson config yet and this helps!

taepper

A few comments – impressive how fast you went through the code-base! I would have thought that it would first only replace a few rapidjson usages.

Generally, we can choose whether to use the simdjson::dom or simdjson::ondemand frontend for arrow.

simdjson::ondemand can be faster and less memory-intensive. [1]

Its api is a little more difficult to use, but might be fine for our use-case, as we do not intend to parse multiple values in the document in parallel.

I see that you are using ondemand frontend for json/parser.cc, and dom for all other use-cases. It seems sensible to me at first glance, but it might make sense to document the reasoning for this decision

[1] https://arxiv.org/pdf/2312.17149

taepper · 2026-02-03T16:06:48Z

 py3-numpy-dev \
 python3-dev \
-rapidjson-dev \
+simdjson-dev \


Is this required when you bundle simdjson in all cases?

taepper · 2026-02-03T16:09:09Z

- add_arrow_example(rapidjson_row_converter EXTRA_LINK_LIBS RapidJSON)
- add_arrow_example(from_json_string_example EXTRA_LINK_LIBS RapidJSON)
-endif()
+add_arrow_example(from_json_string_example)


simdjson is not always enabled. Should this stay conditional?
if(ARROW_JSON OR ARROW_FLIGHT_SQL_ODBC) set(ARROW_WITH_SIMDJSON ON) endif()

taepper · 2026-02-03T16:12:59Z

+ )

- externalproject_add(rapidjson_ep
+ set(SIMDJSON_CMAKE_ARGS


We might want to set this option to OFF:
option(SIMDJSON_EXCEPTIONS "Enable simdjson's exception-throwing interface" ON)
https://github.com/simdjson/simdjson/blob/6fffc99dac4029a67dd5a794d2f59ec6ba84a1b1/cmake/exception-flags.cmake#L1
This disables any exception-throwing API-functions from simdjson

taepper · 2026-02-04T08:40:20Z

 python3-rados \
 rados-objclass-dev \
-rapidjson-dev \
+libsimdjson-dev \


taepper · 2026-02-04T08:41:13Z

 pkg-config \
 protobuf-compiler \
-rapidjson-dev \
+libsimdjson-dev \


simdjson is bundled again, right?

rok · 2026-02-04T10:39:47Z

Thanks for the review @taepper ! As a general comment, I thought I would first create a PR here and work on a POC "in secret" to see if it's feasible and then split it up into manageable PR against main.
I'm travelling today and will probably have time to work on this by end of week.

github-actionsBot added Component: FlightRPC Component: C++ Component: Parquet awaiting committer review Component: Python Component: Documentation CI: Extra: C++ labels Jan 9, 2026

rok force-pushed the simdjson branch 2 times, most recently from 5af4403 to bb71ad8Compare January 20, 2026 12:03

github-actionsBot added Component: R Component: Gandiva Component: MATLAB Component: Ruby Component: GLib labels Jan 20, 2026

rok force-pushed the simdjson branch from 53ee4e6 to 8ea924dCompare January 20, 2026 12:34

Repository owner deleted a comment from github-actionsBotJan 20, 2026

github-actionsBot removed Component: R Component: Gandiva Component: MATLAB Component: Ruby Component: GLib labels Jan 20, 2026

rok force-pushed the simdjson branch from aafaaeb to 5f1261aCompare January 21, 2026 16:25

github-actionsBot added Component: R Component: Ruby labels Jan 21, 2026

rok force-pushed the simdjson branch from 5f1261a to bfee058Compare January 21, 2026 16:27

rok mentioned this pull request Jan 21, 2026
[C++] can we use simdjson to replace rapidjson apache/arrow#35460
Open

github-actionsBot removed Component: R Component: Ruby labels Jan 21, 2026

rok force-pushed the simdjson branch from f54ab81 to 934c5cfCompare January 23, 2026 12:03

github-actionsBot added the Component: R label Jan 23, 2026

rok added 19 commits January 26, 2026 12:32

cmake
343dda0

cmake
7cf6df0

cmake
63d8f3e

cmake
28ae1db

cmake
fb52c90

lint
cf1f66c

fix simdjson version in conan
355a9f2

simplification
84af758

fix
bb6f94c

cmake
e696611

cmake
06cb2ce

cmake
bc16df5

cmake
734da24

cmake
d8f55d5

cmake
56f0185

cmake
142193c

cmake
017cdec

cmake
ae9b2e7

cmake and lint
76ee292

rok force-pushed the simdjson branch from 0420e36 to 76ee292Compare January 28, 2026 12:01

taepper mentioned this pull request Jan 28, 2026
GH-35460: Use simdjson instead of RapidJSON apache/arrow#49048
Closed

rok added 2 commits January 29, 2026 02:29

cmake
f4386b0

cmake
19f136b

WillAyd reviewed Feb 3, 2026
View reviewed changes

taepper reviewed Feb 4, 2026
View reviewed changes

Reranko05 mentioned this pull request Aug 21, 2026
GH-50944: [C++] Replace RapidJSON with simdjson in JSON chunker apache/arrow#50945
Open

	simdjson_dep = dependency('simdjson', fallback: ['simdjson', 'simdjson_dep'])
	simdjson_dep = dependency('simdjson')

Conversation

rok commented Jan 9, 2026

Uh oh!

rok commented Jan 28, 2026

Uh oh!

taepper commented Jan 28, 2026

Uh oh!

WillAyd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Feb 4, 2026

Uh oh!

taepper left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Feb 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants