Skip to content

Add pluggable page-level AEAD encryption support - #866

Draft
HTHou wants to merge 1 commit into
developfrom
codex/tsfile-tde-page-aead
Draft

Add pluggable page-level AEAD encryption support#866
HTHou wants to merge 1 commit into
developfrom
codex/tsfile-tde-page-aead

Conversation

@HTHou

Copy link
Copy Markdown
Contributor

Summary

  • add an optional file encryption header carrying provider, profile, key, wrapped data key, and file identity metadata
  • introduce a provider-neutral AEAD SPI and registry without embedding a concrete cryptographic implementation
  • encrypt compressed page bodies for non-aligned, aligned, and table-model write paths, and decrypt them on the corresponding read paths
  • preserve existing unencrypted TsFile behavior and propagate encryption metadata through append, recovery, lazy loading, and sketch tooling
  • add compatibility, round-trip, tamper-detection, append, and recovery tests

Motivation

TsFile currently has legacy encryption interfaces but no self-describing file-level context for pluggable page-level authenticated encryption. This change establishes the format and I/O integration points while leaving algorithm and key-management implementations to external providers.

Compatibility

  • Existing unencrypted TsFiles keep their current layout and remain readable.
  • Encrypted files require a registered provider matching the identifiers stored in the encryption header.
  • Page metadata remains readable while compressed page bodies are protected with AEAD.
  • The encrypted-file format and public API are still under discussion and are not intended as a compatibility commitment in this draft.

Validation

  • ./mvnw spotless:check -P with-java -pl java/tsfile -am
  • ./mvnw test -P with-java -pl java/tsfile -am -Dtest='PageCryptoContextTest,FileEncryptionHeaderTest,TDEPageAeadTsFileTest,UnClosedTsFileReaderTest,ForceAppendTsFileWriterTest,RestorableTsFileIOWriterTest,TimePageWriterTest' -Dsurefire.failIfNoSpecifiedTests=false
  • 30 targeted tests passed with no failures or errors

Draft discussion points

  • finalize the encryption-header schema and format-version semantics
  • bind page authentication data to an unambiguous chunk/page identity
  • refine page-index ownership and the separation between legacy encryption parameters and file AEAD context
  • confirm append/recovery behavior and backward-compatibility expectations before stabilizing the format

@codecov-commenter

codecov-commenter commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 74.37107% with 163 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.71%. Comparing base (86252e5) to head (76754e0).
⚠️ Report is 29 commits behind head on develop.

Files with missing linesPatch %Lines
...pache/tsfile/file/header/FileEncryptionHeader.java73.56%23 Missing ⚠️
...a/org/apache/tsfile/read/TsFileSequenceReader.java63.46%19 Missing ⚠️
...sfile/write/v4/AbstractTableModelTsFileWriter.java37.03%17 Missing ⚠️
...a/org/apache/tsfile/encrypt/PageCryptoContext.java74.57%15 Missing ⚠️
...g/apache/tsfile/read/reader/chunk/ChunkReader.java60.00%14 Missing ⚠️
...ain/java/org/apache/tsfile/write/TsFileWriter.java59.37%13 Missing ⚠️
...e/tsfile/write/writer/ForceAppendTsFileWriter.java54.16%11 Missing ⚠️
...org/apache/tsfile/write/writer/TsFileIOWriter.java82.60%8 Missing ⚠️
...che/tsfile/encrypt/EncryptionProviderRegistry.java75.00%7 Missing ⚠️
...ache/tsfile/read/reader/page/LazyLoadPageData.java70.83%7 Missing ⚠️
... and 10 more
Additional details and impacted files
@@ Coverage Diff @@## develop #866 +/- ##
===========================================
+ Coverage 60.52% 60.71% +0.19% 
===========================================
Files 735 740 +5 Lines 48439 48930 +491 Branches 7679 7746 +67 ===========================================
+ Hits 29319 29710 +391 - Misses 17718 17818 +100 
Partials 1402 1402 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@HTHou@codecov-commenter