Replication of MWDI ElasticSearch database needs to be optimized

[kmohr-soprasteria]

Problem
The replication shall not take more than 8 minutes, but it is much slower.

Notes:

• the replication shall be done periodically
• the replication shall be performant enough to create 10 replicas in parallel

[kmohr-soprasteria]

Last optimization results

The test produced the following results:
• Records processed: 8,000 of 35,790
• Elapsed time: 21.5 minutes
• Processing rate: approximately 6 records per second
• Estimated total duration at the current rate: approximately 96 minutes
Completing 35,790 records within 5–8 minutes would require approximately 75–119 records per second. This is around 12–19 times faster than the performance measured during our test.
For clarity, the relevant technical settings are explained below:
• requests_per_second=-1 removes the Elasticsearch software speed restriction and allows copying to proceed as quickly as the server can support. The test confirmed that no software throttling occurred.
• scrollSize, also called batch size, defines how many records Elasticsearch processes in one operation. We tested with a batch size of 500.
• Since the individual records are relatively large, a batch of 500 may contain approximately 165 MB of data. Increasing the batch size to 1,000 may produce approximately 330 MB per operation, which could explain why the test with a batch size of 1,000 failed.
• A shard is an independently processable partition of an Elasticsearch index. The source index currently has only one such partition.
The current environment contains one Elasticsearch server and one source-index partition. Therefore, the same server must read the source data, write the replica and perform Elasticsearch’s internal processing using the same CPU, memory and storage.
During the test, I observed:
• Server CPU usage: approximately 57%
• Elasticsearch JVM heap usage: approximately 18%
• Overall server memory usage: approximately 93%
• System load: 6.34
• Disk capacity usage: approximately 28%
The available disk capacity is sufficient. However, these measurements do not show the disk transfer speed or latency.
Elasticsearch reported no software throttling and no bulk or search retries. Therefore, the application-level speed restriction is no longer the limiting factor.
Based on these results, no application code change or Elasticsearch setting can guarantee completion within 5–8 minutes. Parallel reindexing alone will also not achieve the target because the source index has only one partition and the deployment contains only one Elasticsearch server.
To pursue the required performance, Elasticsearch infrastructure improvement is needed. Possible approaches include:
• Upgrading the existing server with faster local NVMe SSD storage and additional CPU capacity.
• Deploying multiple Elasticsearch data servers and redesigning the indices with multiple partitions, allowing the workload to be distributed across servers.
The 5–8 minute target must be validated through performance testing after the infrastructure changes. It cannot be guaranteed with the present single-server architecture.

[kmohr-soprasteria]

Suggestions from AI, which could be evaluated.

---

---

After providing the AI with the information that the replication shall take place again and again, in a fixed interval (8 min):

there is such a timestamp

[kmohr-soprasteria]

if replication shall be done in parallel for 10 applications: