Add support for everything Firmware_extractor supported by akhilnarang · Pull Request #146 · sebaubuntu-python/dumpyara

akhilnarang · 2026-06-28T09:36:17Z

Summary

This change adds a Rust parser module for Android firmware files. PyO3 provides the parser functions to Python.

Firmware formats

The module adds these parsers:

Android sparse images and sparse image chunks
Amlogic firmware
KDDI firmware
MediaTek signed images
Nokia NB0 files
Oppo OZIP files
Spreadtrum PAC files
Qualcomm QFIL files
Rockchip RKFW and AFP files
Sony SIN and FTF files
ZTE firmware

Extraction changes

Detect a known firmware format before generic archive extraction.
Detect nested firmware files after archive extraction.
Use ZIP extraction for archives that have a nonstandard file extension.
Remove a common vendor prefix from extracted file names.
Use the Rust sparse-image parser when the module is available.
Use simg2img when the Rust module is not available.
Search nested ZIP files for partition markers.
Keep a canonical partition when an alias also exists.
Accept partition files that use the .bin suffix.
Write all_files.txt with UTF-8 encoding.

Compatibility and checks

Use PyO3 0.29.
Support Python 3.14.
Add type information for the optional parser module.
Define the Ruff lint rules for stable CI results.
Fix the issues that Codacy reported.
Apply the standard Python and Rust formats.

The following checks pass:

Origin ruff-check
Origin ruff-format
Origin pyright
Upstream Codacy analysis
python -m compileall -q dumpyara
cargo fmt --manifest-path firmware-parsers/Cargo.toml -- --check
cargo check --locked --manifest-path firmware-parsers/Cargo.toml
cargo test --locked --manifest-path firmware-parsers/Cargo.toml

codacy-production · 2026-06-28T09:38:07Z

Not up to standards ⛔

🔴 Issues1 minor

Alerts:
⚠ 1 issue (≤ 0 issues of at least minor severity)
Results:
1 new issue
Category Results
Security 1 minor
View in Codacy

🟢 Metrics629 complexity · 70 duplication

Metric Results
Complexity 629
Duplication 70
View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer}
_{TIP This summary will be updated as you push new changes.}

Add a new Rust + PyO3 native extension (firmware-parsers/) that provides Android sparse image conversion, replacing the external simg2img binary dependency. The crate exposes sparse_to_raw, sparse_chunks_to_raw, and is_sparse functions to Python via the firmware_parsers module. All simg2img subprocess calls in multipartitions.py, raw_image.py, and sparsed_images.py now use the native Rust implementation when available, with graceful fallback to the simg2img binary when firmware_parsers is not installed. The simg2img tool requirement is also conditionally removed from REQUIRED_TOOLS in dumpyara.py.

Add Rust extractors for NB0, PAC, and MTK signed image formats and expose them through the PyO3 module. Update format detection to recognize PAC and NB0 files directly and inspect zip archives for MTK signed image payloads.

Implement Phases 3-6 of the firmware-parsers plan: - Phase 3: OPPO ozip (AES-128-ECB decrypt, 35 keys, mode-1 direct and mode-2 zip-wrapped with inner .ozip entry decryption) and Sony SIN/FTF (v3/v4/v5 + legacy SSSS/BFBF + Sony sparse chunks 0xCAC1-0xCAC5) - Phase 4: Amlogic USB burning tool and Rockchip RKFW/AFP containers - Phase 5: QFIL rawprogram XML, ZTE update.zip, KDDI .bin extractors - Phase 6: Wire firmware_parsers.detect() into extract_archive.py with graceful fallback to shutil.unpack_archive on failure Also fix issues found in review of Phase 1-2 code: - Remove "sparse" from detect() returns (no matching Python function) - Strengthen NB0 probe with printable-ASCII filename validation - Validate BFBF/MTK header magic before 0x4040 byte strip - Sanitize output filenames in nb0/pac/amlogic/rockchip/qfil (path traversal) - Fix SIN tar entry processing (per-entry instead of concatenation) - Use static OnceLock<Regex> in ZTE/KDDI instead of per-call compilation - Expand partition name whitelist for ZTE/KDDI bin detection Format-specific fixes: - OZIP: handle PK-wrapped mode-2 inputs, decrypt inner .ozip entries, strip .ozip extension so downstream recognizes decrypted payloads; preserve decrypted filename for non-zip payloads (e.g. system.new.dat.br.ozip -> system.new.dat.br, not .br.img); use temp work directory for zip extraction to avoid leaked temp files; propagate decryption failure instead of silently renaming ciphertext - QFIL: group <program> entries by label to merge sparsechunks instead of overwriting; resolve filenames by basename for flattened layouts; sanitize XML labels against path traversal; honor file_sector_offset when rebuilding partitions from shared backing files; branch on all_same_file without requiring non-zero first offset - Amlogic: detect tar.bz2 wrappers by content (BZh magic) instead of filename extension; verify Amlogic magic before accepting tar entries to avoid selecting non-Amlogic .img files that precede the payload - SIN: stream-scan for gzip/tar offset instead of reading entire file into memory (avoids OOM on multi-GB files); strip .sin extension case-insensitively to match case-insensitive FTF detection - detect.rs: scan all tar.bz2 members for Amlogic magic (not just the first entry, since packages may have readme/manifest first); tighten bzip2 check from 2-byte "BZ" to 3-byte "BZh" Housekeeping: - Remove unused Cargo dependencies (thiserror, memmap2, bytemuck, byteorder)

Rockchip RKFW header uses packed/unaligned fields: - AFP container offset at 0x21 (not 0x1C) - AFP container size at 0x25 (not 0x20) AFP container structure corrected: - AFP header is 0x8C bytes (not 0x4C) - entry_count at offset 0x88 (not 0x44) - Each entry is 0x70 / 112 bytes (not 0x48 / 72) - Entry offset at 0x60, size at 0x6C (not 0x20/0x28)

When a firmware ships both a real partition and its alias (e.g. modem.img alongside NON-HLOS.img on recent Oppo/ColorOS builds), fix_aliases logged "Ignoring <alias> (<name> already extracted)" and unlinked the alias, but then fell through and still attempted move(alias, partition). Since the alias was just deleted, this raised FileNotFoundError and aborted the dump. Add the missing continue so the redundant alias is dropped and the real partition is left untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Some firmware (e.g. aftermarket car head units) wrap the real dumpable ROM one zip deeper -- the outer archive holds vendor blobs and APK dirs, while the actual block-based OTA (system.new.dat.br + *.transfer.list, payload.bin, super*.img, *.tar.md5) lives inside an inner .zip such as update_car.zip. dumpyara only scanned the outer archive's top level, found no recognized partition container, and aborted with "System folder doesn't exist". Extend nested-archive handling: for each top-level .zip not already covered by NESTED_ARCHIVES, peek its central directory (zipfile namelist, no extraction) and recurse only when it contains a partition marker. The marker patterns are anchored to a path boundary and the *.new.dat.br / *.transfer.list markers are restricted to known partition names, so APK/config zips (which carry none of these) are never exploded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Signed-off-by: Muhammad Asif <thevancedgamer@mentallysanemainliners.org>

Fixes dumping OPPO and/or OnePlus stock firmwares that use Chinese characters for app paths Signed-off-by: Muhammad Asif <thevancedgamer@mentallysanemainliners.org>

Define the Ruff rule set. Add type information for the optional parser module. Resolve the fallback executable paths. Use PyO3 0.29 for Python 3.14. Apply the standard Python and Rust formats.

The extractor did not extract a nested image zip when the zip held loose raw partition images. Such a zip has no super image and no payload file. Pixel factory images use this layout. They ship system.img, vendor.img, and product.img directly. The marker check missed these files. Therefore the extractor skipped the nested zip. The system partition did not reach the output. The system folder assertion then failed. Add a marker pattern for loose raw partition images. Anchor the pattern on the known partition names. Allow an optional A/B slot suffix. Allow the image file extensions that get_raw_image accepts. The pattern does not match super_empty.img. The pattern does not match an unrelated image file. Derive the partition names from get_partition_names_with_alias. Add the return type to two partition helper functions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The vendored Python payload parser is single-threaded, iterates partitions and operations serially, and shells out to xzcat/bzcat once per operation. On a 3.5 GB Pixel foldable OTA (grizzly, ~40 partitions), that hits our 1-hour bot timeout; running otadump on the same input finishes in ~1-2 min. Use the otadump binary when it's on PATH (Rayon-parallel per partition, in-process zstd/xz/bzip2), fall back to the vendored parser otherwise. Stage otadump's output in a sibling tempdir and move on success so a partial failure doesn't leave stray images next to payload.bin. Fallback triggers only on binary-absent, not on error — the Python impl is less strict about hash verification and silently falling back could hide corruption.

Since Xiaomi HyperOS 4, Xiaomi added new partition called "mi_product". For now it's quite empty, but it might change in the future.

feat: mi_product support

akhilnarang · 2026-06-28T09:36:17Z