Skip to content

Repository files navigation

Lite Transaction File (LTX)

The LTX file format provides a way to store SQLite transactional data in a way that can be encrypted and compacted and is optimized for performance.

File Format

This document describes format version 3. LTX files carry no on-disk version field, and versions 2 and 3 both use the LTX1 magic number. Version 2 page frames used a four-byte page header with no flags or compressed-size prefix. Version 3 uses a six-byte page header and, in the current encoding, a four-byte compressed-size prefix. A reader cannot determine the format version from the file alone and must know it out of band.

An LTX file is composed of four sections:

  1. Header
  2. Page block
  3. Page index
  4. Trailer

The header contains metadata about the file, the page block contains page frames, the page index enables random access to frames, and the trailer contains checksums for the file and the database end state. Unless otherwise specified, all fixed-width integer fields use big-endian byte order.

Header

The 100-byte header describes the database state and transaction ID (TXID) range represented by the file.

OffsetSizeFieldDescription
04MagicAlways LTX1.
44FlagsHeader flags.
84PageSizeDatabase page size, in bytes.
124CommitDatabase size after applying the file, in pages.
168MinTXIDMinimum transaction ID.
248MaxTXIDMaximum transaction ID.
328TimestampMilliseconds since the Unix epoch.
408PreApplyChecksumDatabase checksum before applying the file.
488WALOffsetOffset in the original WAL; zero for a journal.
568WALSizeWAL segment size; zero for a journal.
644WALSalt1First WAL salt; zero for a journal or compaction.
684WALSalt2Second WAL salt; zero for journal or compaction.
728NodeIDCreator node ID; zero if unset.
8020ReservedWritten as zero by the current encoder.
Header flags
FlagNameDescription
0x00000002HeaderFlagNoChecksumDisable database checksum tracking.

HeaderFlagNoChecksum is bit 1 (1 << 1). When set, the pre-apply and post-apply database checksums are zero. All other header flag bits are currently invalid. The file checksum is still required when database checksum tracking is disabled.

Page block

The page block stores page frames in ascending page-number order. Each frame written by the current encoder has this layout:

OffsetSizeFieldDescription
04PgnoOne-based database page number.
42FlagsPage header flags.
64CompressedSizeSize of the compressed payload, in bytes.
10NDataLZ4 block-compressed database page data.
Page header flags
FlagNameDescription
0x0001PageHeaderFlagSizeA four-byte compressed-size field follows.

PageHeaderFlagSize is bit 0 (1 << 0). The payload must decompress to Header.PageSize bytes. The current encoder always sets this flag. Within version 3, the decoder uses it as a per-frame encoding heuristic and supports legacy frames without the flag; those store page data as an LZ4 frame without a size prefix. The flag is not a format version field. All other page header flag bits are invalid.

A six-byte zero page header terminates the page block and has no size prefix or page data.

Page index

The page index follows the zero page header. It contains one entry per page, sorted by page number. Each entry is encoded as three consecutive unsigned varints:

  1. Page number
  2. Absolute byte offset of the page frame from the start of the file
  3. Encoded frame size, including its header, optional size prefix, and payload

A zero page-number varint terminates the entries. An eight-byte big-endian unsigned integer follows and contains the total byte size of the varint entries, including the zero terminator but excluding the size field itself.

Trailer

The 16-byte trailer contains the database checksum after applying the file and the file checksum.

OffsetSizeFieldDescription
08PostApplyChecksumDatabase checksum after applying the file.
88FileChecksumCRC-ISO-64 checksum described below.

File checksum

The file checksum is a CRC-ISO-64 over the following input, in order:

  1. The header bytes.
  2. For every page, the page header and compressed-size prefix, when present, as stored, followed by the decompressed page data instead of the compressed payload bytes.
  3. The zero page header that terminates the page block.
  4. All page index bytes, including its zero terminator and size field.
  5. The trailer's post-apply checksum field.

The final eight-byte file checksum field is not included. The stored value is the calculated CRC with ChecksumFlag (1 << 63) set.

This checksum provides logical-content integrity plus integrity for selected structural metadata; it is not a byte-for-byte checksum of the file on disk. In particular, different valid LZ4 payload bytes produce the same checksum when they decompress to the same page data and do not change the hashed size or index values.

About

Go library for the LTX file format

Resources

Stars

308 stars

Watchers

5 watching

Forks

Releases

Packages

Used by

Contributors

Languages