Skip to content

sessiondataset-guide.md

Latest commit

History

History
437 lines (320 loc) · 10.5 KB

sessiondataset-guide.md

File metadata and controls

437 lines (320 loc) · 10.5 KB
Raw

SessionDataSet Iterator Pattern Guide

This guide explains how to use the SessionDataSet iterator pattern for efficiently querying large datasets from Apache IoTDB.

Overview

The SessionDataSet provides an iterator-based approach to reading query results, similar to JDBC ResultSet or database cursors. This approach offers several advantages over loading all data into memory at once:

  • Memory Efficient: Only keeps the current batch in memory
  • Lazy Loading: Fetches data on-demand as you iterate
  • Large Datasets: Can handle result sets larger than available RAM
  • Resource Management: Proper cleanup of server-side resources

Basic Usage

Simple Iteration

import{Session}from'@iotdb/client';constsession=newSession({host: 'localhost',port: 6667,username: 'root',password: 'root',});awaitsession.open();// Execute query and get SessionDataSetconstdataSet=awaitsession.executeQueryStatement('SELECT * FROM root.test.d1');// Iterate through resultswhile(awaitdataSet.hasNext()){constrow=dataSet.next();// Access timestampconsttimestamp=row.getTimestamp();// Access values by column nameconsttemperature=row.getFloat('temperature');consthumidity=row.getDouble('humidity');conststatus=row.getString('status');console.log(`${timestamp}: temp=${temperature}, humidity=${humidity}, status=${status}`);}// Always close the dataset when doneawaitdataSet.close();awaitsession.close();

API Reference

SessionDataSet

Methods

hasNext(): Promise<boolean>

Checks if there are more rows available. This may trigger fetching the next batch from the server.

while(awaitdataSet.hasNext()){// Process next row}
next(): RowRecord

Returns the next row. Must call hasNext() first to ensure a row is available.

if(awaitdataSet.hasNext()){constrow=dataSet.next();}
close(): Promise<void>

Closes the dataset and releases server-side resources. Always call this when done.

awaitdataSet.close();
getColumnNames(): string[]

Returns array of column names.

constcolumns=dataSet.getColumnNames();console.log('Columns:',columns);// ['temperature', 'humidity', 'status']
getColumnTypes(): string[]

Returns array of column data types.

consttypes=dataSet.getColumnTypes();console.log('Types:',types);// ['FLOAT', 'DOUBLE', 'TEXT']
findColumn(columnName: string): number

Returns the zero-based index of a column by name.

consttempIndex=dataSet.findColumn('temperature');// Returns 0
toArray(): Promise<any[][]> (Deprecated)

Loads all remaining rows into memory as an array. Use only for small result sets.

// ⚠️ Not recommended for large datasetsconstallRows=awaitdataSet.toArray();

RowRecord

Represents a single row of data from the query result.

Methods

Timestamp Access
consttimestamp=row.getTimestamp();// Returns number (milliseconds)
Access by Column Name
// Typed gettersconststringValue=row.getString('name');constintValue=row.getInt('count');constlongValue=row.getLong('id');constfloatValue=row.getFloat('temperature');constdoubleValue=row.getDouble('humidity');constboolValue=row.getBoolean('status');// Generic getter (returns any)constvalue=row.getValue('columnName');
Access by Column Index
conststringValue=row.getStringByIndex(0);constintValue=row.getIntByIndex(1);constfloatValue=row.getFloatByIndex(2);constvalue=row.getValueByIndex(3);
Null Checking
if(row.isNull('optionalColumn')){console.log('Column is null');}else{constvalue=row.getString('optionalColumn');}// By indexif(row.isNullByIndex(0)){console.log('First column is null');}
Get All Data
constfields=row.getFields();// Returns array of field valuesconstarray=row.toArray();// Returns [timestamp, ...fields]

Advanced Usage

Configuring Fetch Size

Control how many rows are fetched in each batch:

constsession=newSession({host: 'localhost',port: 6667,fetchSize: 1024,// Fetch 1024 rows at a time});constdataSet=awaitsession.executeQueryStatement('SELECT * FROM root.test.d1');// Will automatically fetch in batches of 1024 rows

Processing Large Result Sets

For very large result sets, use iterator pattern to avoid memory issues:

constdataSet=awaitsession.executeQueryStatement('SELECT * FROM root.large_dataset');letcount=0;letsum=0;while(awaitdataSet.hasNext()){constrow=dataSet.next();sum+=row.getDouble('value');count++;// Log progress every 10000 rowsif(count%10000===0){console.log(`Processed ${count} rows...`);}}console.log(`Total rows: ${count}, Average: ${sum/count}`);awaitdataSet.close();

Error Handling

Always use try-finally to ensure cleanup:

letdataSet;try{dataSet=awaitsession.executeQueryStatement('SELECT * FROM root.test.d1');while(awaitdataSet.hasNext()){constrow=dataSet.next();// Process row}}catch(error){console.error('Query error:',error);throwerror;}finally{if(dataSet){awaitdataSet.close();}}

Working with Multiple Columns

constdataSet=awaitsession.executeQueryStatement(` SELECT temperature, humidity, pressure, status FROM root.weather.station1 WHERE time > now() - 1h`);while(awaitdataSet.hasNext()){constrow=dataSet.next();constreading={timestamp: newDate(row.getTimestamp()),temperature: row.getFloat('temperature'),humidity: row.getFloat('humidity'),pressure: row.getFloat('pressure'),status: row.getString('status'),};// Process readingif(!row.isNull('status')&&reading.temperature>30){console.log('High temperature alert:',reading);}}awaitdataSet.close();

Aggregation Queries

constdataSet=awaitsession.executeQueryStatement(` SELECT AVG(temperature), MAX(temperature), MIN(temperature) FROM root.test.d1 GROUP BY ([2024-01-01, 2024-02-01), 1d)`);while(awaitdataSet.hasNext()){constrow=dataSet.next();console.log({timestamp: newDate(row.getTimestamp()),avg: row.getDouble('AVG(temperature)'),max: row.getDouble('MAX(temperature)'),min: row.getDouble('MIN(temperature)'),});}awaitdataSet.close();

Migration from Old Pattern

The old pattern loaded all data into memory at once. The new pattern uses lazy loading with iterators.

Old Approach (No Longer Supported)

// ❌ This no longer works - executeQueryStatement now returns SessionDataSet// const result = await session.executeQueryStatement('SELECT * FROM root.test.d1');// for (const row of result.rows) {// const timestamp = row[0];// const value1 = row[1];// const value2 = row[2];// console.log(timestamp, value1, value2);// }

New Approach (Current Implementation)

// ✅ New way - lazy loading with iteratorconstdataSet=awaitsession.executeQueryStatement('SELECT * FROM root.test.d1');while(awaitdataSet.hasNext()){constrow=dataSet.next();consttimestamp=row.getTimestamp();constvalue1=row.getValueByIndex(0);constvalue2=row.getValueByIndex(1);console.log(timestamp,value1,value2);}awaitdataSet.close();

Migration Steps

  1. Change method call: executeQueryStatement() now returns SessionDataSet (not QueryResult)
  2. Replace array access: Use iterator pattern instead of result.rows
  3. Update row access: Use RowRecord methods instead of array indices
  4. Add cleanup: Always call await dataSet.close()

For Small Datasets

If you need all data at once for small result sets, use toArray():

constdataSet=awaitsession.executeQueryStatement('SELECT * FROM root.test.d1');constallRows=awaitdataSet.toArray();// Loads everything into memory// allRows is [[timestamp, value1, value2], ...]

⚠️Warning: toArray() loads all data into memory. Only use for small result sets.

Best Practices

  1. Always Close: Use try-finally to ensure dataSet.close() is called
  2. Check hasNext(): Always call hasNext() before calling next()
  3. Use Fetch Size: Set appropriate fetch size based on memory and network
  4. Handle Nulls: Check isNull() before accessing nullable columns
  5. Typed Access: Use typed getters (getInt, getString) for type safety
  6. Avoid toArray(): Don't use toArray() for large datasets

Performance Tips

  1. Fetch Size: Larger fetch size = fewer network calls, more memory used

    • Small datasets: 100-1000
    • Large datasets: 1000-10000
    • Very large: 10000-50000

  2. Batch Processing: Process rows in batches for better performance

constdataSet=awaitsession.executeQueryStatement('SELECT * FROM root.test.d1');constbatch=[];while(awaitdataSet.hasNext()){batch.push(dataSet.next());if(batch.length>=1000){awaitprocessBatch(batch);batch.length=0;}}if(batch.length>0){awaitprocessBatch(batch);}awaitdataSet.close();
  1. Column Access: Access by index is slightly faster than by name
// Fasterconstvalue=row.getIntByIndex(0);// Slightly slower (name lookup)constvalue=row.getInt('temperature');

Troubleshooting

"No more rows available" Error

// ❌ Wrong - calling next() without checking hasNext()constrow=dataSet.next();// May throw error// ✅ Correctif(awaitdataSet.hasNext()){constrow=dataSet.next();}

"Column not found" Error

// Make sure column names match exactly (case-sensitive)constcolumns=dataSet.getColumnNames();console.log('Available columns:',columns);// Use correct column nameconstvalue=row.getString('temperature');// Not 'Temperature' or 'temp'

Memory Issues

// ❌ Don't do this with large datasetsconstallRows=awaitdataSet.toArray();// Loads everything into memory// ✅ Process iteratively insteadwhile(awaitdataSet.hasNext()){constrow=dataSet.next();// Process and discard each row}

See Also