Optimized the overall performance of IoTDB & Fixed the NPE in LimitOperatorTest

[Caideyipi]

Description

1. Removed the unnecessary copies in QueryDataSetUtils & PartialPath.
2. Cached the deletion ranges to speedup the mod checks
3. Removed some unnecessary copies for partial paths
4. Removed some boxings on hot paths
5. Optimized some data structures (List::contains -> Set::contains, LinkedList -> ArrayList)
6. Fixed the write-lock leak in TsFileManager.removeAll.
7. Fixed wildcard tree last-cache invalidation for device paths.

---

This PR has:

• been self-reviewed.
  • concurrent read
  • concurrent write
  • concurrent read and write
• added documentation for new or modified features or behaviors.
• added Javadocs for most classes and all non-trivial methods.
• added or updated version, license, or notice information
• added comments explaining the why and the intent of the code wherever would not be obvious
  for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold
  for code coverage.
• added integration tests.
• been tested in a test IoTDB cluster.

---

Key changed/added classes (or packages if there are too many classes) in this PR

[codecov]

Codecov Report

❌ Patch coverage is 52.36220% with 121 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.48%. Comparing base (402e399) to head (c0b4676).
⚠️ Report is 26 commits behind head on master.

Files with missing lines	Patch %	Lines
...read/filescan/impl/UnclosedFileScanHandleImpl.java	0.00%	38 Missing ⚠️
...in/java/org/apache/iotdb/rpc/IoTDBJDBCDataSet.java	0.00%	19 Missing ⚠️
...n/read/filescan/impl/ClosedFileScanHandleImpl.java	0.00%	18 Missing ⚠️
...java/org/apache/iotdb/isession/SessionDataSet.java	0.00%	12 Missing ⚠️
...rc/main/java/org/apache/iotdb/session/Session.java	85.91%	10 Missing ⚠️
.../apache/iotdb/rpc/stmt/PreparedParameterSerde.java	12.50%	7 Missing ⚠️
...a/org/apache/iotdb/db/utils/QueryDataSetUtils.java	73.07%	7 Missing ⚠️
...org/apache/iotdb/db/schemaengine/SchemaEngine.java	0.00%	4 Missing ⚠️
...g/apache/iotdb/db/storageengine/StorageEngine.java	33.33%	4 Missing ⚠️
...aregion/tsfile/timeindex/ArrayDeviceTimeIndex.java	0.00%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@ Coverage Diff @@## master #17664 +/- ##
============================================
+ Coverage 40.26% 40.48% +0.21% 
Complexity 2574 2574 ============================================
Files 5179 5179 Lines 349659 350152 +493 Branches 44688 44780 +92 ============================================
+ Hits 140798 141756 +958 + Misses 208861 208396 -465 

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[JackieTien97]

Thanks for the performance work — the changes are mostly solid and several are clear wins: the zero-copy wrapBuffer rewrite in QueryDataSetUtils, the bytesToHex lookup-table rewrite, factoring the device/time filter out of the per-measurement loop in ModificationUtils, the removeAll write-lock leak fix in TsFileManager, and the new matchFullPath(IDeviceID) / matchPrefixPath(IDeviceID) overloads in PartialPath (no more throwaway PartialPath allocations on every cache check).

A few items below are worth confirming before merge, grouped by priority:

Should confirm (potential correctness)

• SessionDataSet (Java + C++): the loop switched to 1-based index access (valueColumnStartIndex + 1 .. columnSize). Please confirm the index-based getters are 1-based relative to the column-name list — an off-by-one here silently reads adjacent columns with no error.
• Session.java: nullMap is now null when info logging is disabled. Every subsequent dereference in those methods must be guarded.
• UnclosedFileScanHandleImpl: the new lazy HashMap caches are not thread-safe; please confirm single-threaded access or switch to ConcurrentHashMap.
• TableDeviceSchemaCache.invalidateLastCache: the matchFullPath direction was swapped — the new direction looks correct (and likely fixes a latent wildcard bug), worth calling out and covering with a test.
• ModificationUtils: confirm the new overlap(...) helper has the same inclusive/exclusive boundary semantics as TimeRange.overlaps.

Minor / style / cleanup

• Inline fully-qualified java.util.Collection / java.util.HashSet in SchemaEngine and StorageEngine — please use imports.
• QueryDataSetUtils: rowCount and valueOccupation parameters are unused after the refactor — please drop them.
• IoTDBJDBCDataSet: getDeduplicatedColumnSize now returns max(index)+1 instead of distinct-count — equivalent for contiguous indices, just confirm no downstream code depends on the old size.

Details inline.

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
1.7% Duplication on New Code

See analysis details on SonarQube Cloud

[luoluoyuyu]

Review summary

Performance improvements across client, schema cache, storage scan, and TsFileManager. The removeAll finally block now calls writeUnlock instead of acquiring the write lock again, fixing a serious lock leak.

Please fix CI failures in this PR. See inline notes.

[luoluoyuyu]

nullMap is null when info logging is disabled; all uses in this method are guarded. Please verify every overload of filterNullValueAndMeasurement changed in this PR uses the same pattern.

[Caideyipi]

Verified. The two filterNullValueAndMeasurement overloads changed in this PR guard every nullMap access with nullMap != null, and the wrapper overloads only delegate to them without dereferencing the map.

[luoluoyuyu]

Critical fix: finally now calls writeUnlock() instead of writeLock again in removeAll, which previously leaked the write lock on every call.

[Caideyipi]

Confirmed, this fix is included and has been called out in the PR description as a correctness fix, not just a performance optimization.

[luoluoyuyu]

deviceToDeletionRanges caches merged deletion ranges per device via putIfAbsent, avoiding repeated sortAndMerge in isDeviceTimeDeleted. If this handle can be used from multiple threads, document single-thread use or use a concurrent map.

[Caideyipi]

Verified. ClosedFileScanHandleImpl now uses ConcurrentHashMap for deviceToDeletionRanges and deviceToModifications, publishes device-level ranges with putIfAbsent, and creates per-device time-series maps as ConcurrentHashMap instances as well.

[Caideyipi]

Thanks for the review. I rechecked the latest head and replied inline. The Windows and IT failures mentioned earlier are now green. The remaining C++ macOS failure was a Maven assembly OOM (Java heap space) while building iotdb-cli, so I reran the failed Multi-Language Client workflow; the macOS rerun is pending now.