Skip to content

README.md

Latest commit

History

History
1797 lines (1407 loc) · 50.9 KB

README.md

File metadata and controls

1797 lines (1407 loc) · 50.9 KB
Raw

Apache IoTDB Node.js Client

Licensenpm versionNode.js Version

A Node.js client for Apache IoTDB with support for SessionPool and TableSessionPool, providing efficient connection management and comprehensive query capabilities.

Table of Contents

Overview

The Apache IoTDB Node.js Client is a high-performance, feature-rich client library for interacting with Apache IoTDB, a time-series database designed for IoT data management. This client provides both tree model (timeseries) and table model (relational) APIs, enabling flexible data management strategies.

Features

  • Session Management: Single session with query, non-query, and insertTablet operations
  • SessionPool: Connection pooling for high-concurrency scenarios with automatic load balancing
    • ✨ Enhanced Metrics: Comprehensive pool monitoring (totalCount, idleCount, activeCount, waitingCount)
    • ✨ Wait Queue Tracking: Monitor requests waiting for connections
  • TableSessionPool: Specialized pool for table model operations with database context management
  • Multi-Node Support: Round-robin load balancing across multiple IoTDB nodes with failover
  • SSL/TLS Support: Secure connections with customizable SSL options and certificate validation
  • TypeScript Support: Full TypeScript definitions with strict type checking
  • Builder Pattern: Fluent API for elegant configuration management
  • Memory Efficient: SessionDataSet with lazy loading and pagination for large result sets
  • Comprehensive Testing: Unit tests, E2E tests, and benchmark tools included
  • Production Ready: Connection pooling, idle cleanup, health checks, and error handling

Installation

npm install @iotdb/client

Requirements

  • Node.js >= 14.0.0
  • Apache IoTDB >= 1.0.0

Quick Start

Basic Session Usage

import{Session}from'@iotdb/client';constsession=newSession({host: 'localhost',port: 6667,username: 'root',password: 'root',});awaitsession.open();// Execute non-query statementawaitsession.executeNonQueryStatement('CREATE DATABASE root.test');// Execute query with SessionDataSet (iterator pattern - memory efficient)constdataSet=awaitsession.executeQueryStatement('SELECT * FROM root.test.**');while(awaitdataSet.hasNext()){constrow=dataSet.next();console.log(row.getTimestamp(),row.getFields());}awaitdataSet.close();// Or use toArray() helper for small result sets (loads all into memory)constdataSet2=awaitsession.executeQueryStatement('SHOW DATABASES');constallRows=awaitdataSet2.toArray();// Returns [[timestamp, ...fields], ...]console.log('All rows:',allRows);// Execute query with custom timeout (30 seconds)constcustomDataSet=awaitsession.executeQueryStatement('SELECT * FROM root.test.**',30000);// ... iterate and close// Insert tablet data (supports both tree and table models)// Tree model example:awaitsession.insertTablet({deviceId: 'root.test.device1',measurements: ['temperature','humidity'],dataTypes: [3,3],// FLOATtimestamps: [Date.now(),Date.now()+1000],values: [[25.5,60.0],[26.0,61.5],],});awaitsession.close();

Using Builder Pattern (Recommended)

The Builder pattern provides a more elegant and fluent API for configuration:

import{Session,ConfigBuilder}from'@iotdb/client';// Build a session configurationconstsession=newSession(newConfigBuilder().host('localhost').port(6667).username('root').password('root').fetchSize(1024).timezone('UTC+8').build());awaitsession.open();// ... use sessionawaitsession.close();

Query Results with SessionDataSet

The executeQueryStatement() method returns a SessionDataSet for efficient iteration through query results:

import{Session,SessionDataSet,RowRecord}from'@iotdb/client';constsession=newSession({host: 'localhost',port: 6667,username: 'root',password: 'root',fetchSize: 1024,// Rows per fetch});awaitsession.open();// Execute query and get SessionDataSetconstdataSet: SessionDataSet=awaitsession.executeQueryStatement('SELECT temperature, humidity FROM root.test.device1');// Iterate through results efficientlywhile(awaitdataSet.hasNext()){constrow: RowRecord=dataSet.next();// Access by column nameconsttemp=row.getFloat('temperature');consthumidity=row.getFloat('humidity');// Access by indexconsttimestamp=row.getTimestamp();console.log(`${timestamp}: temp=${temp}, humidity=${humidity}`);}// Always close the datasetawaitdataSet.close();awaitsession.close();

Benefits of SessionDataSet:

  • ✅ Memory Efficient: Only keeps current batch in memory
  • ✅ Lazy Loading: Fetches data on-demand
  • ✅ Large Datasets: Can handle results larger than available RAM
  • ✅ Type Safety: Typed getters prevent errors
  • ✅ Resource Management: Proper cleanup with close()

See SessionDataSet Guide for complete documentation.

SessionPool Usage

import{SessionPool}from'@iotdb/client';constpool=newSessionPool('localhost',6667,{username: 'root',password: 'root',maxPoolSize: 10,minPoolSize: 2,maxIdleTime: 60000,waitTimeout: 60000,});awaitpool.init();// Execute queries using the poolconstresult=awaitpool.executeQueryStatement('SELECT * FROM root.test.**');// Execute non-query statementsawaitpool.executeNonQueryStatement('CREATE TIMESERIES root.test.device1.temperature WITH DATATYPE=FLOAT');// Insert dataawaitpool.insertTablet({deviceId: 'root.test.device1',measurements: ['temperature'],dataTypes: [3],// FLOATtimestamps: [Date.now()],values: [[25.5]],});// Get pool statisticsconsole.log('Pool size:',pool.getPoolSize());console.log('Available:',pool.getAvailableSize());// Enhanced metrics for better monitoringconsole.log('Waiting requests:',pool.waitingCount);conststats=pool.getPoolStats();console.log('Comprehensive stats:',stats);// { total, idle, active, waiting, endpoints, redirectCacheSize }awaitpool.close();

Enhanced Pool Metrics

The SessionPool provides comprehensive metrics for monitoring pool health:

constpool=newSessionPool({host: 'localhost',port: 6667,maxPoolSize: 10,minPoolSize: 2,waitTimeout: 60000,// Timeout for waiting requests (ms)maxIdleTime: 60000,// Idle connection timeout (ms)});// New metric getters (backward compatible with old methods)console.log('Total connections:',pool.totalCount);console.log('Idle connections:',pool.idleCount);console.log('Active connections:',pool.activeCount);console.log('Waiting requests:',pool.waitingCount);// Comprehensive stats objectconststats=pool.getPoolStats();// { total, idle, active, waiting, endpoints, redirectCacheSize }

Key Features:

  • ✅ Enhanced Metrics: Comprehensive pool health monitoring
  • ✅ Wait Queue Tracking: Monitor requests waiting for connections
  • ✅ Backward Compatible: All existing code works unchanged

await pool.close();


### Explicit Session Management
For more control, you can explicitly get and release sessions from the pool:
```typescript
import { SessionPool } from '@iotdb/client';
const pool = new SessionPool('localhost', 6667, {
username: 'root',
password: 'root',
maxPoolSize: 10,
});
await pool.init();
// Get a session from the pool
const session = await pool.getSession();
try {
// Use the session for multiple operations
await session.executeNonQueryStatement('CREATE DATABASE root.test');
const result = await session.executeQueryStatement('SHOW DATABASES');
await session.insertTablet({
deviceId: 'root.test.device1',
measurements: ['temperature'],
dataTypes: [3],
timestamps: [Date.now()],
values: [[25.5]],
});
} finally {
// Always release the session back to the pool
pool.releaseSession(session);
}
await pool.close();

Concurrent Operations (High Throughput)

For high-throughput scenarios, use the concurrent APIs optimized for Node.js:

Batch Insert - Single RPC Call

Insert multiple tablets in a single RPC call (most efficient for tree model):

import{Session,TreeTablet,TSDataType}from'@iotdb/client';constsession=newSession({host: 'localhost',port: 6667});awaitsession.open();// Create multiple tabletsconsttablets=[];for(leti=0;i<100;i++){tablets.push({deviceId: `root.test.device${i}`,measurements: ['temperature'],dataTypes: [TSDataType.FLOAT],timestamps: [Date.now()+i*1000],values: [[25.0+i]],});}// Insert all tablets in single RPC callawaitsession.insertTablets(tablets);awaitsession.close();

Concurrent Pool Insertion

Use pool-level concurrent insertion for maximum throughput:

import{SessionPool}from'@iotdb/client';constpool=newSessionPool({host: 'localhost',port: 6667,maxPoolSize: 20,minPoolSize: 5,});awaitpool.init();// Generate tabletsconsttablets=generateTablets(1000);// Insert concurrently with controlled parallelism// Pre-acquires sessions to avoid contentionawaitpool.insertTabletsParallel(tablets,{concurrency: 20});awaitpool.close();

Generic Parallel Execution

Execute any operations in parallel using the pool:

import{SessionPool}from'@iotdb/client';constpool=newSessionPool({host: 'localhost',port: 6667,maxPoolSize: 10});awaitpool.init();constdeviceNames=['d1','d2','d3','d4','d5'];// Execute operations in parallelconstresults=awaitpool.executeParallel(deviceNames,async(session,deviceName)=>{awaitsession.executeNonQueryStatement(`CREATE TIMESERIES root.test.${deviceName}.value WITH DATATYPE=FLOAT`);return`Created ${deviceName}`;},{concurrency: 5});console.log(results);// ['Created d1', 'Created d2', ...]awaitpool.close();

Utility Functions

Standalone utilities for concurrent execution:

import{executeConcurrent,chunkArray,createSemaphore}from'@iotdb/client';// Execute any async operations with controlled concurrencyconstresult=awaitexecuteConcurrent(items,async(item)=>awaitprocessItem(item),{concurrency: 10,logProgressEvery: 100});console.log(`Success: ${result.successCount}, Failed: ${result.failureCount}`);// Split arrays for batch processingconstchunks=chunkArray(largeArray,100);// Fine-grained concurrency controlconstsem=createSemaphore(5);awaitsem.acquire();try{awaitdoWork();}finally{sem.release();}

Multi-Node Support

Method 1: Using nodeUrls with String Format (Recommended)

When nodes have different host:port combinations, use the nodeUrls configuration with string array format:

import{SessionPool,PoolConfigBuilder}from'@iotdb/client';// Using config object with string array (RECOMMENDED)constpool1=newSessionPool({nodeUrls: ['node1.example.com:6667','node2.example.com:6668','node3.example.com:6669',],username: 'root',password: 'root',maxPoolSize: 15,minPoolSize: 3,});// Or using Builder pattern with string arrayconstpool2=newSessionPool(newPoolConfigBuilder().nodeUrls(['node1.example.com:6667','node2.example.com:6668','node3.example.com:6669',]).username('root').password('root').maxPoolSize(15).minPoolSize(3).build());awaitpool1.init();// Connections will be distributed across all nodes using round-robin

Method 2: Using nodeUrls with Object Format (Also Supported)

You can also use the object format for nodeUrls:

constpool=newSessionPool({nodeUrls: [{host: 'node1.example.com',port: 6667},{host: 'node2.example.com',port: 6668},{host: 'node3.example.com',port: 6669},],username: 'root',password: 'root',maxPoolSize: 15,});

Method 3: Traditional API (For Same Port)

When all nodes share the same port:

import{SessionPool}from'@iotdb/client';constpool=newSessionPool(['node1.example.com','node2.example.com','node3.example.com'],6667,{username: 'root',password: 'root',maxPoolSize: 15,});awaitpool.init();// Connections will be distributed across all nodes using round-robin

SSL/TLS Support

import{Session}from'@iotdb/client';import*asfsfrom'fs';constsession=newSession({host: 'localhost',port: 6667,username: 'root',password: 'root',enableSSL: true,sslOptions: {ca: fs.readFileSync('/path/to/ca.crt'),cert: fs.readFileSync('/path/to/client.crt'),key: fs.readFileSync('/path/to/client.key'),rejectUnauthorized: true,},});awaitsession.open();

TableSessionPool Usage

import{TableSessionPool}from'@iotdb/client';consttablePool=newTableSessionPool('localhost',6667,{username: 'root',password: 'root',database: 'my_database',// Set default database for table modelmaxPoolSize: 10,minPoolSize: 2,});awaittablePool.init();// Execute queries in table modeconstresult=awaittablePool.executeQueryStatement('SHOW DATABASES');awaittablePool.close();

Redirection Support ✅ (Implemented)

Status: ✅ Fully Implemented

The client now supports automatic write redirection for multi-node IoTDB clusters. When a write operation is sent to a node that doesn't own the device's data, the server responds with a redirect recommendation (status code 400). The client automatically:

  1. Caches the device→endpoint mapping
  2. Creates/reuses a connection to the optimal endpoint
  3. Retries the operation on the correct node
  4. Uses the cached mapping for future writes to the same device

Benefits:

  • 30-50% throughput improvement by avoiding cross-node data forwarding
  • Reduced network latency
  • Better resource utilization

Configuration:

import{SessionPool,TableSessionPool}from'@iotdb/client';// Tree model pool with redirectionconsttreePool=newSessionPool({nodeUrls: ['192.168.1.100:6667','192.168.1.101:6667','192.168.1.102:6667'],username: 'root',password: 'root',maxPoolSize: 10,enableRedirection: true,// Enable redirection (default: true)redirectCacheTTL: 300000,// Cache TTL in ms (default: 5 minutes)});// Table model pool with redirectionconsttablePool=newTableSessionPool({nodeUrls: ['192.168.1.100:6667','192.168.1.101:6667','192.168.1.102:6667'],enableRedirection: true,});

How It Works:

// First write to a device - server returns redirect recommendationconsttablet={deviceId: 'root.sg.device1',measurements: ['temperature'],dataTypes: [TSDataType.FLOAT],timestamps: [Date.now()],values: [[25.5]],};awaitpool.insertTablet(tablet);// → Writes to Node A (via round-robin)// → Write succeeds!// → Server responds with code 400: "Recommend using Node B for this device in the future"// → Client caches: device1 → Node B (for next write)// Second write to same device - uses cached endpointawaitpool.insertTablet({deviceId: 'root.sg.device1',measurements: ['temperature'],dataTypes: [TSDataType.FLOAT],timestamps: [Date.now()+1000],values: [[26.0]],});// → Client checks cache: device1 → Node B// → Writes directly to Node B// → No redirect needed!

Testing:

Redirection support has been tested with the 1C3D (1 ConfigNode, 3 DataNodes) cluster configuration. Run E2E tests:

# Start 1C3D cluster
docker-compose -f docker-compose-1c3d.yml up -d
# Run redirection tests
MULTI_NODE=true npm run test:e2e

Implementation Details:

See docs/redirection-design.md for detailed design documentation.

API Reference

Configuration Builders

ConfigBuilder

Fluent API for building Session configurations:

import{ConfigBuilder}from'@iotdb/client';constconfig=newConfigBuilder().host('localhost').port(6667).username('root').password('root').database('mydb').timezone('UTC+8').fetchSize(2048).enableSSL(false).build();

Methods:

  • host(host: string): this - Set the host
  • port(port: number): this - Set the port
  • nodeUrls(nodeUrls: EndPoint[]): this - Set multiple node URLs
  • username(username: string): this - Set the username
  • password(password: string): this - Set the password
  • database(database: string): this - Set the database
  • timezone(timezone: string): this - Set the timezone
  • fetchSize(fetchSize: number): this - Set the fetch size
  • enableSSL(enable: boolean): this - Enable or disable SSL
  • sslOptions(sslOptions: SSLOptions): this - Set SSL options
  • build(): Config - Build and return the configuration

PoolConfigBuilder

Fluent API for building SessionPool configurations (extends ConfigBuilder):

import{PoolConfigBuilder}from'@iotdb/client';constconfig=newPoolConfigBuilder().host('localhost').port(6667).username('root').password('root').maxPoolSize(20).minPoolSize(5).maxIdleTime(30000).waitTimeout(45000).build();

Additional Methods:

  • maxPoolSize(size: number): this - Set maximum pool size
  • minPoolSize(size: number): this - Set minimum pool size
  • maxIdleTime(time: number): this - Set maximum idle time (ms)
  • waitTimeout(timeout: number): this - Set wait timeout (ms)
  • build(): PoolConfig - Build and return the pool configuration

Data Types

IoTDB Node.js client supports all IoTDB data types including BOOLEAN, INT32, INT64, FLOAT, DOUBLE, TEXT, BLOB, STRING, DATE, and TIMESTAMP. See Data Types Reference for comprehensive documentation on:

  • Type mappings between JavaScript and IoTDB
  • Usage examples for each data type
  • Best practices and encoding options
  • Null value handling

Session

Constructor

Option 1: Using config object

newSession(config: Config)

Option 2: Using Builder pattern (Recommended)

newSession(newConfigBuilder()...build())

The config must include either:

  • host and port for single node connection
  • nodeUrls for multi-node connection (uses first node)

Methods

  • async open(): Promise<void> - Open the session
  • async close(): Promise<void> - Close the session
  • async executeQueryStatement(sql: string, timeoutMs?: number): Promise<QueryResult> - Execute a query with optional timeout (default: 60000ms)
  • async executeNonQueryStatement(sql: string): Promise<void> - Execute a non-query statement
  • async insertTablet(tablet: Tablet): Promise<void> - Insert tablet data
  • isOpen(): boolean - Check if session is open

SessionPool

Constructor

Option 1: Traditional API (Backward compatible)

newSessionPool(hosts: string|string[],port: number,config?: Partial<PoolConfig>)

Option 2: Using config object with nodeUrls

newSessionPool(config: PoolConfig)

Option 3: Using Builder pattern (Recommended)

newSessionPool(newPoolConfigBuilder()...build())

Methods

Connection Management:

  • async init(): Promise<void> - Initialize the pool
  • async close(): Promise<void> - Close all connections

Automatic Session Management:

  • async executeQueryStatement(sql: string, timeoutMs?: number): Promise<QueryResult> - Execute a query with optional timeout (default: 60000ms)
  • async executeNonQueryStatement(sql: string): Promise<void> - Execute a non-query statement
  • async insertTablet(tablet: Tablet): Promise<void> - Insert tablet data

Explicit Session Management:

  • async getSession(): Promise<Session> - Get a session from the pool (must be released)
  • releaseSession(session: Session): void - Release a session back to the pool

Pool Statistics:

  • getPoolSize(): number - Get current pool size
  • getAvailableSize(): number - Get available connections
  • getInUseSize(): number - Get number of sessions currently in use

TableSessionPool

Same as SessionPool but optimized for table model operations. Automatically executes USE DATABASE when configured with a database. All query methods support the same timeout parameter (default: 60000ms).

Constructor

Same constructor options as SessionPool.

Types

Config

interfaceConfig{host?: string;port?: number;nodeUrls?: string[]|EndPoint[];// String array format: ["host1:6667", "host2:6668"]username?: string;password?: string;database?: string;timezone?: string;fetchSize?: number;enableSSL?: boolean;sslOptions?: SSLOptions;}

Note: Either host/port OR nodeUrls must be provided.

  • Use nodeUrls in string array format (e.g., ["host1:6667", "host2:6668"]) for nodes with different ports (RECOMMENDED)
  • Object format [{ host, port }] is also supported for backward compatibility

EndPoint

interfaceEndPoint{host: string;port: number;}

PoolConfig

interfacePoolConfigextendsConfig{maxPoolSize?: number;minPoolSize?: number;maxIdleTime?: number;waitTimeout?: number;}

Tablet

#### TreeTablet(TreeModel/TimeseriesModel)**Interface(forplainobjects):**```typescriptinterfaceITreeTablet{deviceId: string;// Full path like "root.test.device1"measurements: string[];// Sensor namesdataTypes: number[];// TSDataType for each measurementtimestamps: number[];// Array of timestampsvalues: any[][];// 2D array: [rows][columns]}

Class (with helper methods):

import{TreeTablet,TSDataType}from'@iotdb/client';// Create a tabletconsttablet=newTreeTablet('root.test.device1',['temperature','humidity'],[TSDataType.FLOAT,TSDataType.DOUBLE]);// Add rows one at a time using addRow methodtablet.addRow(Date.now(),[25.5,60.0]);tablet.addRow(Date.now()+1000,[26.0,61.5]);tablet.addRow(Date.now()+2000,[26.5,62.0]);// Insert the tabletawaitsession.insertTablet(tablet);

Alternative: Plain object approach (still supported)

awaitsession.insertTablet({deviceId: 'root.test.device1',measurements: ['temperature','humidity'],dataTypes: [TSDataType.FLOAT,TSDataType.DOUBLE],timestamps: [Date.now(),Date.now()+1000],values: [[25.5,60.0],[26.0,61.5]],});

TableTablet (Table Model / Relational Model)

Interface (for plain objects):

interfaceITableTablet{tableName: string;// Table namecolumnNames: string[];// Column names (excluding timestamp)columnTypes: number[];// TSDataType for each columncolumnCategories: ColumnCategory[];// Category for each columntimestamps: number[];// Array of timestampsvalues: any[][];// 2D array: [rows][columns]}enumColumnCategory{TAG=0,// Tag column - indexed for WHERE clause filteringFIELD=1,// Field column - measurement valuesATTRIBUTE=2,// Attribute column - metadata not indexedTIME=3,// Time column (reserved for internal use, do not use in columnCategories)}

Class (with helper methods):

import{TableTablet,ColumnCategory,TSDataType}from'@iotdb/client';// Create a tabletconsttablet=newTableTablet('sensor_data',['device_id','temperature','humidity'],[TSDataType.TEXT,TSDataType.FLOAT,TSDataType.DOUBLE],[ColumnCategory.TAG,ColumnCategory.FIELD,ColumnCategory.FIELD]);// Add rows one at a time using addRow methodtablet.addRow(Date.now(),['device_001',25.5,60.0]);tablet.addRow(Date.now()+1000,['device_002',26.0,61.5]);tablet.addRow(Date.now()+2000,['device_003',26.5,62.0]);// Insert the tabletawaitpool.insertTablet(tablet);

Alternative: Plain object approach (still supported)

awaitpool.insertTablet({tableName: 'sensor_data',columnNames: ['device_id','temperature','humidity'],columnTypes: [TSDataType.TEXT,TSDataType.FLOAT,TSDataType.DOUBLE],columnCategories: [ColumnCategory.TAG,ColumnCategory.FIELD,ColumnCategory.FIELD],timestamps: [Date.now(),Date.now()+1000],values: [['device_001',25.5,60.0],['device_002',26.0,61.5]],});

Note: TIME is reserved for internal use. When specifying columnCategories in TableTablet, only use TAG, FIELD, and ATTRIBUTE. Timestamps are handled separately via the timestamps array.

Deprecated Tablet (for backward compatibility)

interfaceTablet{deviceId: string;measurements: string[];dataTypes: number[];// 0=BOOLEAN, 1=INT32, 2=INT64, 3=FLOAT, 4=DOUBLE, 5=TEXTtimestamps: number[];values: any[][];}// Note: Use TreeTablet instead

#### QueryResult
```typescript
interface QueryResult {
columns: string[];
dataTypes: string[];
rows: any[][];
queryId?: number;
}

Data Types

When using insertTablet, specify data types using these constants:

  • 0 - BOOLEAN
  • 1 - INT32
  • 2 - INT64
  • 3 - FLOAT
  • 4 - DOUBLE
  • 5 - TEXT

Migration Guide

Upgrading to the New API

The new version maintains full backward compatibility while adding new features. No changes are required for existing code, but you may want to adopt the new features:

Multi-Node with Different Ports

Old way (still works, but limited to same port):

constpool=newSessionPool(['node1','node2','node3'],6667,{username: 'root',password: 'root'});

New way (supports different ports per node with string format - RECOMMENDED):

constpool=newSessionPool({nodeUrls: ['node1:6667','node2:6668','node3:6669',],username: 'root',password: 'root',});

Alternative (object format also supported):

constpool=newSessionPool({nodeUrls: [{host: 'node1',port: 6667},{host: 'node2',port: 6668},{host: 'node3',port: 6669},],username: 'root',password: 'root',});

Using Builder Pattern

Old way (still works):

constsession=newSession({host: 'localhost',port: 6667,username: 'root',password: 'root',fetchSize: 2048,});

New way (more fluent):

import{ConfigBuilder}from'@iotdb/client';constsession=newSession(newConfigBuilder().host('localhost').port(6667).username('root').password('root').fetchSize(2048).build());

Explicit Session Management

Old way (still works):

// Pool automatically manages sessionsconstresult=awaitpool.executeQueryStatement('SELECT ...');

New way (more control):

// Explicitly get and release sessionsconstsession=awaitpool.getSession();try{constresult1=awaitsession.executeQueryStatement('SELECT ...');constresult2=awaitsession.executeQueryStatement('SELECT ...');// ... multiple operations with same session}finally{pool.releaseSession(session);}

Technical Architecture

Overview

The IoTDB Node.js client follows a three-layer architecture design, optimized for both single-session and high-concurrency scenarios:

┌─────────────────────────────────────────────────────┐
│ Application Layer (Your Code) │
└─────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────┐
│ Pool Layer │
│ ┌──────────────────┐ ┌──────────────────────┐ │
│ │ SessionPool │ │ TableSessionPool │ │
│ │ - Load Balance │ │ - Database Context │ │
│ │ - Pool Mgmt │ │ - Pool Mgmt │ │
│ └──────────────────┘ └──────────────────────┘ │
└─────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────┐
│ Session Layer │
│ ┌──────────────────────────────────────────────┐ │
│ │ Session │ │
│ │ - Query / Non-Query │ │
│ │ - InsertTablet │ │
│ │ - Result Parsing │ │
│ └──────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────┐
│ Connection Layer │
│ ┌──────────────────────────────────────────────┐ │
│ │ Connection (Thrift) │ │
│ │ - TCP/SSL Transport │ │
│ │ - Session Lifecycle │ │
│ │ - Low-level Protocol │ │
│ └──────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘
↓
Apache IoTDB

Core Components

1. Connection Layer (src/connection/Connection.ts)

  • Manages low-level Thrift connections over TCP or SSL/TLS
  • Handles session lifecycle (open/close with sessionId/statementId)
  • Implements TFramedTransport and TBinaryProtocol
  • Supports single endpoint connections
  • Key Pattern: One connection = one IoTDB node endpoint

2. Session Layer (src/client/Session.ts)

  • High-level API for database operations
  • Methods: executeQueryStatement(), executeNonQueryStatement(), insertTablet()
  • Handles query result parsing with SessionDataSet
  • Supports pagination with configurable fetchSize
  • Key Pattern: Uses first node from nodeUrls for single-session scenarios

3. Pool Layer (src/client/BaseSessionPool.ts, SessionPool.ts, TableSessionPool.ts)

  • Connection pooling with configurable min/max sizes
  • Round-robin load balancing across multiple endpoints
  • Automatic idle connection cleanup (maxIdleTime)
  • Wait queue when pool exhausted (waitTimeout)
  • Health checks and connection recycling
  • Key Pattern: Distributes connections across all nodes in nodeUrls

4. Configuration System (src/utils/Config.ts)

  • Builder pattern for fluent configuration
  • Support for both old (host/port) and new (nodeUrls) formats

5. Tablet Type System

The client provides distinct tablet types for tree and table models:

TreeTablet (Timeseries Model):

  • deviceId: Full path (e.g., "root.sg.device")
  • measurements: Sensor names
  • dataTypes: Type for each measurement
  • timestamps and values: Time-series data

TableTablet (Relational Model):

  • tableName: Table name (not a path)
  • columnNames: All columns including tags, time, fields, attributes
  • columnTypes: Type for each column
  • columnCategories: Category (TAG, TIME, FIELD, ATTRIBUTE) for each column
  • timestamps and values: Includes tag values in data

Polymorphic insertTablet(): Both Session and TableSession use the same method name with runtime type dispatch:

// Works with TreeTabletawaitsession.insertTablet({deviceId: '...',measurements: [...], ...});// Works with TableTabletawaittableSession.insertTablet({tableName: '...',columnNames: [...], ...});
  • Type-safe with TypeScript interfaces
  • Validation and default values

Data Flow

Query Execution Flow

1. Application calls pool.executeQueryStatement()
2. Pool acquires available Session (round-robin)
3. Session sends query via Connection to IoTDB
4. IoTDB returns SessionDataSet with queryId
5. SessionDataSet fetches data in batches (fetchSize)
6. Application iterates results with hasNext()/next()
7. Session released back to pool
8. SessionDataSet.close() releases server resources

Insert Flow

1. Application calls pool.insertTablet()
2. Pool acquires available Session (round-robin)
3. Session serializes Tablet data by column
4. Data sent via Connection to IoTDB
5. IoTDB acknowledges write
6. Session released back to pool

Thread Safety & Concurrency

  • Sessions: Not thread-safe; use SessionPool for concurrency
  • SessionPool: Thread-safe; internal locking for session management
  • Connection Lifecycle: Managed automatically by pool
  • Load Balancing: Round-robin assignment on session acquisition
  • Idle Cleanup: Background task removes idle connections

Thrift Integration

The client uses Apache Thrift for RPC communication:

  • Generated Code: src/thrift/generated/ from IoTDB's .thrift files
  • Protocol: TBinaryProtocol (compact, efficient)
  • Transport: TFramedTransport (message boundaries)
  • SSL Support: Configurable TLS transport layer
  • Version: Compatible with Apache IoTDB 1.0+

Memory Management

  • SessionDataSet: Lazy loading with pagination (default: 1024 rows/fetch)
  • Connection Pool: Bounded size prevents resource exhaustion
  • Idle Cleanup: Automatic connection cleanup after maxIdleTime
  • Result Sets: Must call close() to release server resources

Error Handling

  • Connection Errors: Automatic retry with next node in pool
  • Timeout Handling: Configurable query timeouts (default: 60s)
  • Pool Exhaustion: Wait queue with timeout
  • Thrift Errors: Wrapped in JavaScript errors with stack traces

Configuration Patterns

Constructor Overloading

SessionPool/TableSessionPool support two constructor signatures:

// New format (recommended):newSessionPool({nodeUrls: ["host1:6667","host2:6667"]});// Old format (backward compatible):newSessionPool(["host1","host2"],6667,{/* options */});

Builder Pattern

constsession=newSession(newConfigBuilder().host('localhost').port(6667).fetchSize(2048).build());

Development

Prerequisites

  • Node.js >= 14.0.0
  • npm >= 6.0.0
  • Apache Thrift compiler (optional, for regenerating Thrift files)
  • Git

Development Setup

  1. Clone the repository:
git clone https://github.com/CritasWang/@iotdb/client.git
cd @iotdb/client
  1. Install dependencies:
npm install
  1. Build the project:
npm run build

Build System

The project uses a two-step build process:

  1. esbuild: Fast TypeScript compilation

    • Configured in esbuild.config.js
    • Compiles src/ to dist/ directory
    • Excludes type declaration files

  2. tsc: Type declaration generation

    • Generates .d.ts files for TypeScript support
    • Run with --emitDeclarationOnly flag
    • Ensures type safety for consumers

  3. copy:thrift: Copy generated Thrift files

    • Copies .js files from src/thrift/generated/ to dist/thrift/generated/
    • Required because Thrift code uses require() statements

Build commands:

npm run build # Complete build (esbuild + tsc + copy)
npm run build:esbuild # Only esbuild compilation
npm run build:types # Only type declarations

Development Workflow

  1. Make changes in src/ directory
  2. Build with npm run build
  3. Test with npm test
  4. Lint with npm run lint
  5. Format with npm run format

Code Style

  • Use TypeScript strict mode
  • Follow existing code formatting (Prettier)
  • Add JSDoc comments for public APIs
  • Keep functions focused and concise
  • Use async/await instead of callbacks
  • Handle errors appropriately
  • Prefer explicit types over any

Regenerating Thrift Files

If you need to update to a newer version of IoTDB's Thrift definitions:

  1. Download the latest Thrift files from Apache IoTDB:
git clone --depth 1 https://github.com/apache/iotdb.git /tmp/iotdb
  1. Copy the Thrift files:
cp /tmp/iotdb/iotdb-protocol/thrift-datanode/src/main/thrift/client.thrift thrift/
cp /tmp/iotdb/iotdb-protocol/thrift-commons/src/main/thrift/common.thrift thrift/
  1. Regenerate the Node.js client:
npm run generate:thrift
  1. Test thoroughly to ensure compatibility

Testing

Test Structure

tests/
├── unit/ # Unit tests (fast, no external dependencies)
│ ├── Config.test.ts
│ ├── Logger.test.ts
│ └── ...
└── e2e/ # End-to-end tests (require IoTDB)
├── Session.test.ts
├── SessionPool.test.ts
├── TableSessionPool.test.ts
└── ...

Running Tests

Run all tests:

npm test

Run only unit tests:

npm run test:unit

Run only E2E tests (requires IoTDB instance):

export IOTDB_HOST=localhost
export IOTDB_PORT=6667
export IOTDB_USER=root
export IOTDB_PASSWORD=root
npm run test:e2e

E2E Test Setup

E2E tests require a running IoTDB instance. You can use Docker Compose:

Single Node (1c1d):

docker-compose -f docker-compose-1c1d.yml up -d

3-Node Cluster (3c3d):

docker-compose -f docker-compose-3c3d.yml up -d

Stop containers:

docker-compose -f docker-compose-1c1d.yml down

Test Patterns

Unit Test Example

describe('ConfigBuilder',()=>{test('should build config with all options',()=>{constconfig=newConfigBuilder().host('localhost').port(6667).username('root').password('root').build();expect(config.host).toBe('localhost');expect(config.port).toBe(6667);});});

E2E Test Example

describe('Session E2E Tests',()=>{letsession: Session;beforeAll(async()=>{session=newSession({host: process.env.IOTDB_HOST||'localhost',port: parseInt(process.env.IOTDB_PORT||'6667'),username: process.env.IOTDB_USER||'root',password: process.env.IOTDB_PASSWORD||'root',});awaitsession.open();},60000);// 60s timeout for connectionafterAll(async()=>{if(session?.isOpen()){awaitsession.close();}});test('should execute query',async()=>{if(!session.isOpen())return;// Skip if no connectionconstdataSet=awaitsession.executeQueryStatement('SHOW DATABASES');constrows=awaitdataSet.toArray();expect(Array.isArray(rows)).toBe(true);awaitdataSet.close();});});

Test Coverage

Current test coverage:

  • Unit tests: Core utilities (Config, Logger, data serialization)
  • E2E tests: Session, SessionPool, TableSessionPool
  • All data types tested simultaneously
  • Multi-node scenarios tested
  • Pool behavior tested (size limits, timeouts, cleanup)

Debugging Tests

Debug single test:

npm run test:debug

Debug E2E tests:

npm run test:e2e:debug

Check for open handles:

npm run test:e2e:check-handles

Performance Testing

Comprehensive benchmark tools are available in the benchmark/ directory for performance testing and optimization.

Overview

  • Tree Model Benchmark: Tests timeseries data model using insertTablet API
  • Table Model Benchmark: Tests relational data model using insertTablet API
  • Pre-generated Data: Eliminates data generation overhead during testing
  • Concurrent Clients: Simulates real-world high-concurrency scenarios
  • Detailed Metrics: Throughput, latency, percentiles (P50, P90, P95, P99)

Quick Start

Test benchmark infrastructure (no IoTDB required):

node benchmark/test-benchmark.js

Run tree model benchmark:

CLIENT_NUMBER=10 DEVICE_NUMBER=100 node benchmark/benchmark-tree.js

Run table model benchmark:

CLIENT_NUMBER=10 DEVICE_NUMBER=100 node benchmark/benchmark-table.js

Key Configuration Parameters

ParameterDefaultDescription
CLIENT_NUMBER10Number of concurrent clients
DEVICE_NUMBER100Number of devices to simulate
SENSOR_NUMBER10Number of sensors per device
BATCH_SIZE_PER_WRITE100Data rows per write operation
TOTAL_DATA_POINTS100000Total data points to generate
POOL_MAX_SIZE20Maximum connections in pool

Example: High Concurrency Test

CLIENT_NUMBER=50 \
DEVICE_NUMBER=1000 \
SENSOR_NUMBER=10 \
BATCH_SIZE_PER_WRITE=1000 \
TOTAL_DATA_POINTS=1000000 \
node benchmark/benchmark-tree.js

Performance Metrics

The benchmark reports:

  • Execution Time: Total test duration
  • Operations: Total, successful, failed, success rate
  • Data Points: Total points written
  • Throughput: Operations/sec, Points/sec
  • Latency: Min, Max, Average, P50, P90, P95, P99

Sample Output

================================================================================
BENCHMARK RESULTS
================================================================================
[Execution Time]
Duration: 45.23s (45234ms)
[Operations]
Total Operations: 1000
Successful: 998
Failed: 2
Success Rate: 99.80%
[Data Points]
Total Points Written: 100,000
[Throughput]
Operations/sec: 22.11
Points/sec: 2,210
[Latency (ms)]
Min: 15.23ms
Max: 1250.45ms
Average: 45.23ms
P50 (Median): 42.15ms
P90: 78.45ms
P95: 95.23ms
P99: 125.67ms
================================================================================

Performance Tuning Tips

  1. Optimize Batch Size: Test different values (100-1000 rows)
  2. Adjust Concurrency: Start with 10-20 clients, adjust based on results
  3. Use Connection Pooling: Set appropriate POOL_MIN_SIZE and POOL_MAX_SIZE
  4. Pre-generate Data: Use cached data for accurate results
  5. Monitor Resources: Watch CPU, memory, disk I/O, and network

For complete documentation, see benchmark/README.md.

Examples

See the examples/ directory for more usage examples:

  • examples/basic-session.ts - Basic session usage
  • examples/session-pool.ts - SessionPool usage
  • examples/table-session-pool.ts - TableSessionPool usage
  • examples/multi-node.ts - Multi-node configuration
  • examples/ssl-connection.ts - SSL/TLS connection

Documentation

Comprehensive documentation is available in the docs/ directory:

User Guides

Performance Documentation

Technical Documentation

For Contributors

Additional Resources

Contributing

We welcome contributions from the community! Whether you're fixing bugs, adding features, improving documentation, or reporting issues, your help is appreciated.

How to Contribute

  1. Fork the repository on GitHub
  2. Create a feature branch from main
  3. Make your changes following our code style guidelines
  4. Add tests for new functionality
  5. Update documentation as needed
  6. Submit a pull request with a clear description

Development Guidelines

  • Follow existing code style and conventions
  • Write clear, concise commit messages
  • Add unit tests for new features
  • Ensure all tests pass before submitting PR
  • Update CHANGELOG.md for notable changes
  • Keep PRs focused on a single feature or fix

Code Review Process

All submissions require review before merging:

  1. Automated tests must pass (CI/CD)
  2. Code review by maintainers
  3. Documentation review (if applicable)
  4. Final approval and merge

Reporting Issues

When reporting bugs, please include:

  • Node.js version
  • IoTDB version
  • Operating system
  • Steps to reproduce
  • Expected vs actual behavior
  • Error messages and stack traces

For detailed contribution guidelines, see CONTRIBUTING.md.

Release Process

This project follows semantic versioning (SemVer) and maintains a regular release cycle.

Version Numbering

Given a version number MAJOR.MINOR.PATCH:

  • MAJOR: Breaking API changes
  • MINOR: New features, backward compatible
  • PATCH: Bug fixes, backward compatible

Release Workflow

1. Pre-release Preparation

Update version and changelog:

# Update version in package.json
npm version [major|minor|patch] --no-git-tag-version
# Update CHANGELOG.md with release notes# - New features# - Bug fixes# - Breaking changes# - Deprecations

2. Testing

Run comprehensive tests:

# Unit tests
npm run test:unit
# E2E tests (requires IoTDB)export IOTDB_HOST=localhost
export IOTDB_PORT=6667
npm run test:e2e
# Linting
npm run lint
# Build verification
npm run build

3. Version Tagging

Create and push version tag:

# Commit version bump
git add package.json CHANGELOG.md
git commit -m "chore: bump version to X.Y.Z"# Create tag
git tag -a vX.Y.Z -m "Release vX.Y.Z"# Push to remote
git push origin main
git push origin vX.Y.Z

4. Publishing to npm

Build and publish:

# Build production assets
npm run build
# Publish to npm (requires npm account)
npm publish
# For beta/RC releases
npm publish --tag beta

5. GitHub Release

Create GitHub release:

  1. Go to GitHub Releases page
  2. Click "Create a new release"
  3. Select the version tag
  4. Add release title: v X.Y.Z - Release Name
  5. Copy changelog entries to release notes
  6. Attach build artifacts (if applicable)
  7. Publish release

Release Checklist

  • All tests passing
  • CHANGELOG.md updated
  • Version bumped in package.json
  • Documentation updated
  • Breaking changes documented
  • Migration guide (for major versions)
  • Git tag created
  • npm package published
  • GitHub release created
  • Release announcement (if major)

Release Schedule

  • Patch releases: As needed for critical bugs
  • Minor releases: Monthly or when features are ready
  • Major releases: When breaking changes are necessary

Beta/RC Releases

For testing before stable release:

# Create beta version
npm version 1.2.0-beta.1 --no-git-tag-version
# Publish with beta tag
npm publish --tag beta
# Install beta version
npm install @iotdb/client@beta

Hotfix Process

For critical production issues:

  1. Create hotfix branch from release tag
  2. Fix the issue
  3. Bump patch version
  4. Tag and publish immediately
  5. Merge back to main

Post-release Tasks

  • Update documentation site (if applicable)
  • Announce on project channels
  • Monitor for issues and feedback
  • Prepare next release milestone

License

Apache License 2.0

Copyright © 2024 Apache IoTDB

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

References