Uh oh!
There was an error while loading. Please reload this page.
Add ThingsBoard latest-telemetry and attributes DAOs on IoTDB Table Mode - #113
Conversation
… Mode (GSOC-304 Wk 5) Implements the ThingsBoard AttributesDao SPI (v4.3.1.2, 14 methods) on IoTDB Table Mode against the entity_attributes table, as a Path-3 stretch artifact that stays inert by default. - 13/14 methods implemented; findNextBatch is a relational keyset-pagination migration helper with no IoTDB equivalent, so it throws UnsupportedOperationException. The three v4.3.1.2 additions (findAllKeysByEntityIdsAndScopeAsync, findLatestByEntityIdsAndScope and its async form) are implemented. - save is a tag-only delete-then-insert under an entity read lock plus a per-identity lock; find takes the same identity lock so the gap is invisible. - Inert by default: no @repository on the DAO; an independent database.attributes.type selector decoupled from the timeseries selectors; an AttributesDaoConflictGuard fail-fast; and a constructor cluster-mode acknowledgement gate (iotdb.attributes.cluster_mode). - Strategy F: eight compile-only ThingsBoard/commons stubs, excluded from the built jar. - Documented Phase-1 limitations: null version (IoTDB has no sequence), empty non-null-profile key lookup, best-effort bulk reads. - 114 unit tests and 11 Testcontainers integration tests against apache/iotdb:2.0.8-standalone. Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
…04 Wk 4) Adapts the Wk4 IoTDBTableLatestDao (single-table derived-latest, no shadow table) onto the merged PR-1 base so it coexists with the timeseries and attributes DAOs. No latest overlay yet — that is a follow-up. - LatestDao: findLatest (ORDER BY time DESC LIMIT 1), findAllLatest (LAST_BY(col,time) + MAX(time) GROUP BY key); saveLatest no-op and removeLatest half-open-window are documented derived-latest gaps. - Coherent opt-in activation via IoTDBTableLatestEnabledCondition (database.ts.type=iotdb-table AND database.ts_latest.type=iotdb-table + experimental flag), reusing the shared named session pool; a TimeseriesLatestDaoConflictGuard fails startup on a conflicting host DAO. - v4.3.1.2 TimeseriesLatestDao SPI stub (10 methods incl. findLatestByEntityIds /+Async) verified byte-for-byte against the real ThingsBoard source; pinned by StrategyFContractTest with a getDeclaredMethods length guard. - All three DAOs share one session pool via @ConditionalOnMissingBean(name). - 131 unit tests + 24 Testcontainers ITs (Latest 4 + Timeseries 9 + Attributes 11) against apache/iotdb:2.0.8-standalone. Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
Closes the saveLatest/removeLatest gaps the single-table derived-latest cannot satisfy (mentor-confirmed: add a minimal latest overlay rather than document the latest-only paths as unsupported). - New telemetry_latest overlay table (telemetry shape, TTL='INF'), created only under the latest selector via a second schema-bootstrap bean. - saveLatest does a tag-only delete-then-insert into the overlay under a per-identity lock; the async batch writer prevents distinguishing latest-only from full-save, so the overlay is written on every saveLatest (it is therefore a per-key latest store, written once per latest update). version stays null. - findLatest/findAllLatest merge the derived latest (telemetry) with the overlay by max timestamp per key (overlay wins ties); the merge is max-by-ts, never additive, so latest-only keys surface and stale overlay rows are shadowed. - removeLatest snapshots the merged latest under the per-identity lock, deletes the in-window overlay row, and on rewriteLatestIfDeleted resurrects the next-older telemetry value into the overlay. - Documented Phase-1 residuals (flagged for review): version always null (Cassandra parity), a telemetry-derived removeLatest race for full-save values, overlay TTL='INF' growth, and the same-ts overlay-wins tie-break. - 141 unit tests + 31 Testcontainers ITs (Latest 11) against apache/iotdb:2.0.8-standalone. Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
…304 PR-2) Make the non-null deviceProfileId degradation rationale precise in the DAO javadoc + the module README: it mirrors the official non-relational backend (CassandraBaseTimeseriesLatestDao.findAllKeysByDeviceProfileId also returns Collections.emptyList()), entity_attributes has no device_profile_id tag, and the sole upstream caller is DeviceProfileController GET /api/deviceProfile/devices/keys/attributes (TENANT_ADMIN, a config-time UI key enumeration) which tolerates an empty result. Failing loud would 500 under IoTDB while Cassandra returns empty -- an avoidable backend inconsistency. A real device->profile lookup remains a Phase-2 optional enhancement. Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
…raceful key discovery - saveLatest skips a backdated write so an out-of-order saveLatest never regresses an overlay-only latest below a newer stored value (max-ts-wins; an exact-ts tie still overwrites, preserving the documented overlay-wins-on-tie behaviour) - findLatest/findLatestOpt take the per-identity lock so a concurrent overlay delete-then-insert is never observed mid-window (findAllLatest stays unlocked) - findAllKeysByDeviceProfileId returns an empty list instead of throwing, so the TENANT_ADMIN config-UI key endpoint degrades gracefully instead of returning 500 (matching the non-relational backends; full tenant-wide discovery is deferred) - document the single-JVM overlay convergence limitation - add unit + Testcontainers regression tests for the out-of-order saveLatest case Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
- Key-discovery / batch SPI methods (findAllKeysByEntityIds / findLatestByEntityIds and their async forms) return an empty result instead of throwing UnsupportedOperationException, matching CassandraBaseTimeseriesLatestDao: they back reachable endpoints (the dashboard telemetry-key lookup and entity-delete housekeeping) where a throw would surface as an HTTP 500 / a failed cleanup task. - IoTDBTableAttributesDao.save returns a non-null version (the attribute lastUpdateTs) so BaseAttributesService#doSave never unboxes null; the write is a failure-safe insert-then-delete-older and point reads use ORDER BY time DESC LIMIT 1. - removeLatest deletes the overlay row only when the overlay row's own timestamp is inside the delete window, so an out-of-window overlay value survives as the next latest; the rewrite-resurrect path reads both telemetry and the overlay (max-ts-wins). - The saveLatest max-ts guard and the removeLatest snapshot read the derived and overlay latest B1-tolerantly (a same-timestamp two-type row is treated as absent so it never wedges a write or delete and self-heals on the next overwrite); the strict read paths still surface it. - Regression tests added for each; 153 unit + 37 Testcontainers IT green (cache off). Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…ter gate Addresses the five review comments on this PR: - Lift the bounded read/task executor (init/submit/shutdown-drain, bounded ArrayBlockingQueue + AbortPolicy, task tracking) out of the three DAOs into IoTDBTableBaseDao so a future shutdown-race fix lives in one place; the DAOs now delegate. No behavior change. - Lift the small per-DAO utilities into the base too: sqlString (the single SQL single-quote escape point), kvEntry type mapping, requireTelemetryKey / requireKey blank checks, and entityPredicate. - Give the attributes DAO its own executor config (iotdb.attributes.executor .threads / .queue-capacity, defaults equal to iotdb.ts.read) instead of borrowing the timeseries read-executor settings, so the attributes path can be tuned independently of the telemetry selector. - Document that the latest overlay write path (saveLatest / removeLatest) shares the read-configured executor/queue and enrich the queue-full message so a write-burst rejection of a concurrent findLatest is not mistaken for a read misconfiguration. - Add a symmetric cluster-mode acknowledgement (iotdb.ts_latest.cluster_mode) to the latest overlay, mirroring the existing attributes gate, so the in-JVM Striped-lock cluster caveat is opt-in the same way on both paths. 155 unit + 37 Testcontainers IT green (cache off); checkstyle, spotless, apache-rat, and enforcer clean. Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
PDGGK
commented
Jul 8, 2026
Thanks for the thorough review, @CritasWang — all five addressed in 511f9bf. 1. Duplicated bounded executor across the three DAOs ( 2. Duplicated small utilities ( 3. Attributes executor borrowed 4. Write path shares the read executor + "read"-named queue-full exception ( 5. Asymmetric cluster gate ( Re-verified with the build cache off: 155 unit + 37 Testcontainers IT (against |
CritasWang
left a comment
There was a problem hiding this comment.
README is missing documentation for the new required property iotdb.ts_latest.cluster_mode
This commit adds a fail-fast constructor gate on iotdb.ts_latest.cluster_mode (must be sticky-routing or disabled), but the README was not updated:
- The configuration table only documents iotdb.attributes.cluster_mode; there is no entry for iotdb.ts_latest.cluster_mode.
- The latest-telemetry section still describes activation as just setting database.ts_latest.type=iotdb-table, with no mention that the new property is now mandatory.
Impact: any deployment configured per the current README (with database.ts_latest.type=iotdb-table set) will fail to start after this change, and the operator has no documentation to consult — the only explanation lives in the exception message and javadoc.
Suggestion: add a row to the README configuration table for iotdb.ts_latest.cluster_mode (mirroring the existing iotdb.attributes.cluster_mode entry: required when the latest DAO is active, allowed values, fail-fast behavior, and why the acknowledgement exists), and note the requirement in the latest-telemetry activation section. While at it, the new tuning knobs iotdb.attributes.executor.threads / iotdb.attributes.executor.queue-capacity should also be added to the table.
…in README Add the README configuration-table rows the review flagged as missing after the executor / cluster-gate changes: iotdb.ts_latest.cluster_mode (required and fail-fast when the latest overlay DAO is active, mirroring iotdb.attributes.cluster_mode) and the iotdb.attributes.executor.threads / .queue-capacity tuning knobs. Also note the ts_latest.cluster_mode requirement in the latest-telemetry activation section, so an operator activating the overlay has documentation to consult rather than only the exception message and javadoc. Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
PDGGK
commented
Jul 8, 2026
Done in 9534959 — added the README configuration-table rows for |
…test The Strategy-F guard test pins the ThingsBoard SPI method signatures and the value-object getters the DAOs consume, but not the value-object constructors they invoke. A compile-only stub whose constructor arg order or shape drifts from ThingsBoard 4.3.1.2 would still compile against the stub, pass every stub-backed unit test and Docker integration test, and only surface as a NoSuchMethodError against the real ThingsBoard runtime. Add valueObjectConstructorsUsedByDaoExist to reflectively pin the exact constructor parameter types each DAO call site depends on -- BasicTsKvEntry, BaseAttributeKvEntry, the five *DataEntry types, the three TsKvLatestRemovingResult shapes and ReadTsKvQueryResult -- plus the TbPair.of and Pair.of factory signatures, and couple ReadTsKvQueryResult's int query-id parameter to TsKvQuery.getId(). Verified against ThingsBoard v4.3.1.2 (commit c37fb509). Signed-off-by: Zihan Dai <99155080+PDGGK@users.noreply.github.com>
This is the second PR of the GSoC series (follows #110, the timeseries
TimeseriesDao, now merged). It adds the latest-telemetry and attributes storage paths for running ThingsBoard on Apache IoTDB 2.x Table Mode.What this adds
1.
IoTDBTableLatestDao— the ThingsBoardTimeseriesLatestDaoSPI on Table Mode.telemetrytable (ORDER BY time DESC LIMIT 1for a single key;LAST_BY(col, time) + MAX(time) GROUP BY keyfor all keys), so the normal full-save path needs no separate latest table.telemetry_latest) that captures the latest-only write/delete cases the single-table derived read cannot satisfy (e.g. the EntityViewLATEST_AND_WScopy path callssaveLatestwith no pairedsave()).findLatest/findAllLatestmerge the derived latest with the overlay by max timestamp per key (overlay wins an exact-ts tie). The overlay table is created only under the latest selector.removeLatestsnapshots the derived and overlay latest separately under a per-identity lock and deletes the overlay row only when the overlay row's own timestamp is inside the half-open delete window — so an out-of-window overlay value is never wiped by a tag-only delete and survives as the next latest; the rewrite-resurrect path looks before the window across both telemetry and the overlay (max-ts-wins).saveLatestorremoveLatest, and a subsequent overlay overwrite self-heals it); the strict read paths (findLatest/findAllLatest) still surface such a row as a fault.findAllKeysByEntityIds(+Async),findLatestByEntityIds(+Async)) return an empty result rather than throwing — matching the officialCassandraBaseTimeseriesLatestDao— because they back reachable endpoints (the dashboard "available telemetry keys" lookup, entity-delete housekeeping) where a throw would surface as an HTTP 500 / a failed cleanup task. Full derived implementations are Wk9 / Wk10.2.
IoTDBTableAttributesDao— the ThingsBoardAttributesDaoSPI on Table Mode, against a dedicatedentity_attributestable.database.attributes.type=iotdb-tableis set — a selector that no current ThingsBoard release exposes — so a normal deployment never instantiates it and attributes keep flowing to the host entity DB. No@Repository; anAttributesDaoConflictGuardfails startup if a conflicting hostAttributesDaois present; a constructor gate requires an explicitiotdb.attributes.cluster_modeacknowledgement.saveis a failure-safe insert-then-delete-older (insert the current row atlastUpdateTs, then delete strictly-older rows for the identity), so anINSERTfailure leaves the prior value intact; point reads useORDER BY time DESC LIMIT 1. It returns a non-null version (the attribute'slastUpdateTs), whichBaseAttributesService#doSaveunboxes intoAttributeKv(..., long).Both follow the same activation isolation + compile-only ThingsBoard stub surface (excluded from the built jar) as #110, and reuse the module's single named session pool (deduped via
@ConditionalOnMissingBean(name=...)); each DAO can be activated independently.Testing
Verified with the build cache off: 153 unit tests + 37 Testcontainers integration tests (Latest 16 / Timeseries 9 / Attributes 12) against
apache/iotdb:2.0.8-standalone; apache-rat, checkstyle, spotless, dependency-analyze, and enforcer all clean. The compile-only SPI stubs are pinned to the real ThingsBoard v4.3.1.2 signatures byStrategyFContractTest.Points I'd appreciate review on
saveLatestcannot tell a latest-only write from a full-save's pairedsaveLatest, sotelemetry_latestis a per-key latest store (one row per key), written on everysaveLatestexcept when a strictly newer latest already exists (max-ts-wins; an exact-ts tie still overwrites, preserving the documented same-timestamp behaviour). The overlay write is a delete-then-insert so a same-timestamp two-type change resolves to a clean single-typed row. It is effectively a latest table (at most one extra overlay write per in-order latest update); I'd welcome confirmation the shape is acceptable, or a preference for a narrower variant.IoTDBTableLatestDaojavadoc):versionis alwaysnull(IoTDB has no per-series sequence — same as the Cassandra backend); for a purely telemetry-derived (full-save) latest,removeLatestand a concurrent historical delete are separate futures, so a re-read can transiently disagree until the historical delete lands;telemetry_latestisTTL='INF'with no entity-level cleanup; a telemetry-only write (saveWithoutLatest/ "skip latest persistence") surfaces via the derived read; and the per-identity overlay lock is in-JVM (Striped), so in a clustered deployment two nodes writing the same identity concurrently can transiently leave two overlay rows until the next write (the max-ts read merge still returns the newer value, and the B1-tolerant guard/remove reads prevent a two-type overlay row from wedging writes) — a symmetric cluster-mode acknowledgement for the latest overlay is a mentor-review option.AttributesDao.findAllKeysByDeviceProfileIdwith a non-nullDeviceProfileIdreturns an empty list — matching the official non-relational backend (CassandraBaseTimeseriesLatestDao.findAllKeysByDeviceProfileIdalso returnsCollections.emptyList()); the sole caller is a TENANT_ADMIN config-time UI key-enumeration endpoint (GET /api/deviceProfile/devices/keys/attributes) that tolerates empty. The rationale is documented in the DAO javadoc + README; a real device→profile lookup is a future optional enhancement.Happy to iterate.