Skip to content

Implement YAML 1.2.2 byte-pattern encoding detection #56

Description

Context

YAML 1.2.2 requires processors to determine a stream's character encoding from a leading byte order mark or, when no mark is present, from the null-byte pattern in the first bytes. The module currently documents strict file decoding, but the remaining conformance work needs to cover BOM-less UTF-16/UTF-32 detection and explicit encoding mismatch behavior.

Official YAML 1.2.2 reference:

Current State

Import-Yaml detects byte order marks and defaults unmarked files to the caller's fallback encoding, defaulting to UTF-8. BOM-less UTF-16 and UTF-32 streams are not deduced from their null-byte pattern. A byte order mark currently wins over an explicit encoding argument instead of reporting a mismatch.

Objective

Make YAML file input conform to YAML 1.2.2 encoding detection while keeping PowerShell usage explicit, discoverable, and safe for pipeline-based file import.

PowerShell Design Considerations

  • Keep Import-Yaml as the byte-stream/file boundary; string-based commands should continue to operate on already-decoded text.
  • Preserve rich PowerShell errors with stable identifiers and path-aware target objects.
  • Treat -Encoding as an intentional caller assertion, not as an advisory fallback when a conflicting byte order mark exists.
  • Keep parameter behavior symmetric with any future file-import parameter sets and avoid ambiguous parameter sets that mix path resolution semantics with decoding semantics.

Tasks

  • Detect BOM-less UTF-16LE, UTF-16BE, UTF-32LE, and UTF-32BE streams from the YAML 1.2.2 null-byte patterns.
  • Reject an explicit encoding that conflicts with a byte order mark using YamlImportEncodingMismatch.
  • Consume matching byte order marks without exposing a BOM character to parsed content.
  • Keep strict decoding behavior and return YamlImportEncodingFailed for malformed byte sequences.
  • Validate document-boundary byte order mark consistency within one stream.
  • Document the behavior in command help and file-command guide examples.

Benefits

  • Correctly imports valid YAML 1.2.2 files that omit a BOM but use UTF-16 or UTF-32.
  • Prevents silently decoding files under a different encoding than the caller explicitly requested.
  • Makes file import behavior predictable in automation and CI.

Acceptance Criteria

  • UTF-16LE/BE and UTF-32LE/BE YAML files without byte order marks import successfully when their first bytes match the YAML 1.2.2 detection table.
  • A file with a byte order mark that conflicts with explicit -Encoding terminates with YamlImportEncodingMismatch.
  • A file with malformed bytes under the selected encoding terminates with YamlImportEncodingFailed and emits no replacement characters.
  • Empty files and UTF-8 files without byte order marks keep current observable behavior.
  • Tests cover byte order mark precedence, byte-pattern detection, explicit encoding mismatch, and strict decoding.

Related

  • Spec folder: spec/Encoding-Detection.md

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions