[#144] Read storage block offsets from the archive BlocksInfo (format version 9)

[SkowronskiAndrew]

Summary

Proactive support for an upcoming Unity Archive format change, so we find out whether it
causes UnityDataTools any trouble before the format change lands. The archive work is on
unity/unity#119906.

The design of that change was revised, and this PR now reflects the revision. The original
approach bumped the format to version 9 and added a header flag,
kArchiveBlockPaddingBetweenChunks, telling readers to round each block's start offset up to
the next 16-byte boundary. The current approach instead has each StorageBlock record its own
offset, and the flag is gone.

That is the better trade for a reader. Block positions are no longer reconstructed from a
padding rule that the reader has to know and the writer has to keep, so the writer can change
its alignment policy later (4 KB for DirectStorage, none at all) with no further format bump,
and the "streamed blocks are not padded, chunk blocks are" special case disappears.

The offset is relative to the start of the data section, not the file. It has to be: in the
BlocksInfo-at-the-start layout that AssetBundle builds use, the absolute data offset depends on
compressedBlocksInfoSize, which is not known until every block has been written — and putting
absolute offsets in the blocks would change the bytes that get compressed, which changes that
size, which changes the offsets. Readers add GetDataOffset(header).

Changes

Parsing — UnityBinaryFormat/ArchiveDetector.cs

• StorageBlock.Offset (UInt64, big-endian) is parsed as the fourth field of each block
  record, after Flags. Note the serialized order differs from the C++ struct layout, where
  offset is declared first.
• Gated on Signature == "UnityFS" && Version >= 9, mirroring Header::HasStorageBlockOffsets().
  Version 8 and earlier store blocks contiguously, so those offsets are still accumulated from
  the compressed sizes — the pre-existing behaviour, now the explicit legacy path.
• A v9 archive whose blocks are out of order or overlapping is rejected, matching the kError
  in ReadBlocksInfo. These offsets are corruption-controlled input rather than derived from
  sizes, so this is the one new failure mode the format change introduces.
• Removed HasBlockPaddingBetweenChunks and the 0x400 flag constant; added
  HasStorageBlockOffsets.

Reporting — Archive/ArchiveTool.cs

• BlockPaddingBetweenChunks removed from the flag name table, since the bit is no longer part
  of the format.
• archive info keeps the Block Padding Size / blockPaddingSize field but now derives it
  from the stored offsets rather than an assumed alignment rule. Without it, Data Size silently
  stops accounting for the whole data section.

Test data

• UnityProjects/LeadingEdge/Assets/Editor/BuildAssetBundles.cs has a BuildLz4 entry point
  (ChunkBasedCompression) writing the same bundle layout to a separate folder. The existing
  LZMA reference bundles are deliberately untouched — both the v8 contiguous layout and the v9
  offset layout are wanted as reference data.
• TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/ regenerated in the new format. The
  scenes bundle is the interesting one: 11 blocks, 80 bytes of padding, with
  BuildPlayer-Scene1.sharedAssets spanning the first eight.
• The two mp3 .meta files gain explicit AudioImporter settings (GUIDs unchanged), which the
  LeadingEdge project needs in order to build with a 6.7 editor at all.

Testing

dotnet test — 790 passed, 10 skipped, 0 failed.

New coverage in UnityDataTool.Tests/ArchiveTests.cs:

• ArchiveBlocks_Version9_OffsetsComeFromTheBlockList — checks the parsed offsets against the
  actual file: blocks in order, every byte of every gap between them zero, and a non-zero total
  gap. That last assertion matters: without it the test would still pass on a contiguous archive
  even if the stored offsets were being ignored entirely.
• ArchiveBlocks_LegacyVersion8_BlocksAreContiguous — the accumulate fallback, against
  PlayerDataCompressed/data.unity3d, a real v8 8-block Lz4 archive.
• ArchiveHeader_Version9_AllFlagsRecognized — version 9, and no flag bit reported as raw hex
  (which would mean a flag we don't know about).
• ArchiveInfo_Version9_ReportsPaddingSize, ArchiveExtract_Version9_FilesExtractedSuccessfully.

Validating that the writer is correct

This branch is also meant to be evidence that the v9 writer on the Unity side is right, so the
checks were aimed at the produced bytes rather than at self-consistency. Against a debug build
of buildpipeline/archive_alignment at 44a19c46:

Check	Result
Offset field is really serialized, and sized right	Uncompressed metadata is 404 bytes = 16 + 4 + 11×18 + 186. At the old 10-byte record it was 316. GetBlocksInfoSize therefore agrees with the bytes written, which the writer asserts.
Offsets point at the real block positions	An independent C# parse and the native reader agree on all 11 offsets, and each block decompresses from its recorded position.
Padding is excluded from compressedSize	Every byte in every inter-block gap is zero, across all 8 bundles.
Blocks ordered and non-overlapping	The invariants hold on every bundle; the parser validates and accepts them.
Flag removal is complete in the writer	BlockPaddingBetweenChunks absent from every regenerated header.
Content survives the round trip	Extraction is byte-identical to the source files, including a file spanning 8 non-contiguous blocks.
StoreStream path (LZMA / uncompressed), not just StoreChunk	Both produce a single block at offset 0 with no padding, and round-trip byte-identically. Driven through UFS_CreateArchive.
v8 archives unaffected	The v8 bundles in the test data still parse and extract; the legacy test asserts contiguity.

Two writer paths were exercised: the Editor's AssetBundle build, and UFS_CreateArchive via the
native library. Both use the BlocksInfo-at-the-start layout.

Residual gaps, neither reachable from here: multiple streamed blocks in one archive (needs

4 GB to trigger the 32-bit size overflow) and a mixed streamed + chunk archive. Both are covered
by the native ArchiveStorage tests on the Unity side.

[Copilot]

Pull request overview

This PR adds proactive support for Unity Archive format v9’s BlockPaddingBetweenChunks flag by updating the C# archive parser to correctly skip 16-byte alignment padding between non-streamed blocks, and extends CLI reporting + test data/tests to validate the new layout.

Changes:

• Update archive block offset accumulation to align after each non-streamed block when BlockPaddingBetweenChunks is set, and consolidate header flag constants into a shared ArchiveFlags definition.
• Improve reporting: recognize the new flag in archive header output and report total block padding size in archive info.
• Add LeadingEdge LZ4 (chunk-based) AssetBundle build output and new tests that validate padding bytes and alignment behavior.

Reviewed changes

Copilot reviewed 14 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
UnityBinaryFormat/ArchiveDetector.cs	Adds v9 padding-aware block offset accumulation and introduces shared ArchiveFlags.
Archive/ArchiveTool.cs	Reports BlockPaddingBetweenChunks as a known flag and adds blockPaddingSize reporting in archive info.
UnityDataTool.Tests/ArchiveTests.cs	Adds tests for v9 flag reporting, block alignment/padding verification, and native extraction on padded archives.
UnityProjects/LeadingEdge/Assets/Editor/BuildAssetBundles.cs	Adds a second build entry point to generate LZ4 chunk-based bundles into a separate folder.
UnityProjects/LeadingEdge/AGENTS.md	Documents the new LZ4 build entry point and its purpose.
TestCommon/Data/LeadingEdgeBuilds/AGENTS.md	Documents the new AssetBundlesLz4/ test-data folder and why it exists.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/AssetBundlesLz4.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/a.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/6.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/assetbundleroot.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/directaudioclipreference.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/serializationdemo.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/scenes.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/singleaudioclipdirectreference.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[SkowronskiAndrew]

The new format is going to be part of in 6.7 b2 and it should already be in the 7.0 alpha release.
I will get a new version of UnityFileSystemApi to unblock this task once the official 6.7 b2 builds are released.

[SkowronskiAndrew]

This will land in unity6.7support branch where we are accumulating fixes.
Tests are still expected to fail until update the UnityFileSystemApi dll in that branch to a version from 6.7