Uh oh!
There was an error while loading. Please reload this page.
Go SDK: airflow-go-pack tool for self-contained bundle binaries - #67156
Merged
jason810496 merged 9 commits intoJun 8, 2026
Conversation
This was referenced May 19, 2026
jason810496force-pushed
the
refactor/go-sdk/airflow-go-pack
branch
from
June 3, 2026 04:14
8c5d62d to
157254cComparejason810496force-pushed
the
refactor/go-sdk/airflow-go-pack
branch
3 times, most recently
from
June 3, 2026 09:37
138b86e to
c2110d9CompareRecord dag and task identity in registration order and expose it via a new EnumerableBundle interface (OrderedDags). This lets tooling read a bundle's dag/task ids without executing any task. AddDag/AddTask signatures are unchanged.
Introduce the airflow-go-pack tool (delivered via the go.mod tool
directive) that builds a Go bundle binary, queries it for its dag/task
identity, and appends the source plus an airflow-metadata.yaml manifest
plus an AFBNDL01 trailer into a single self-contained executable bundle
(ADR 0002/0004).
The bundle binary exposes a single --airflow-metadata flag that prints
the manifest as JSON conforming to airflow-metadata.schema.json
(airflow_bundle_metadata_version, sdk.{language,version,
supervisor_schema_version}, dags). The manifest wire types live once in
internal/airflowmetadata, shared by the producer (pkg/execution) and the
packer so the two cannot drift. The footer reader/writer lives in
internal/bundlefooter and backs the `inspect` subcommand.…ss-arch packing Match the 64-byte AFBNDL01 trailer (with binary_sha256 over the binary region) that the executable spec and the Python ExecutableCoordinator already require; the Go packer previously wrote an incompatible 32-byte trailer that the consumer rejected. Read the manifest from a host-native build so packing works when the deployable artefact is cross-compiled, and recover from an unrunnable foreign-arch --executable by building the --source package on the host solely to read its metadata. Adds a cross-architecture end-to-end test.
…able When --output (or the default output derived from the package directory name) resolved to the same file as the pre-built --executable, the copy step truncated that file before reading it, destroying the binary and appending a footer to an empty region. Resolve the output path before introspection and reject when it aliases the executable or source (cleaned-abs comparison plus os.SameFile for inode-level links). Assemble the bundle through a temp file and atomically rename it into place so a failed pack never leaves a truncated or half-written artefact at the output path.
jason810496force-pushed
the
refactor/go-sdk/airflow-go-pack
branch
2 times, most recently
from
June 4, 2026 05:47
820b526 to
0dd4970Comparejason810496
commented
Jun 4, 2026
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
- Reject --executable together with --goos/--goarch: --executable packs the binary as-is and never builds, so it cannot cross-compile. Silently ignoring the flags contradicted the fail-fast guidance that points unrunnable-binary users at the cross-build path. - Resolve the source path and reject a directory/colliding --output before any go build, so a misconfigured output fails fast instead of after a (cross) compile. - Fix stale docs: the manifest is YAML-by-default (not "the JSON shape"), and the bundle output is a self-contained executable with an appended AFBNDL01 trailer, not a ZIP.
jason810496force-pushed
the
refactor/go-sdk/airflow-go-pack
branch
from
June 4, 2026 06:14
0dd4970 to
43a330eComparejason810496
marked this pull request as ready for review
June 4, 2026 06:15
amoghrajesh
approved these changes
Jun 8, 2026
amoghrajesh
left a comment
Contributor
There was a problem hiding this comment.
Looks solid. Nice work.
One thing worth raising at the PR level rather than inline: the AFBNDL01 trailer is one half of a Go to Python wire contract with ExecutableCoordinator, but the tests on this side are go only round trips. Could we add a checked-in golden rule or a hex fixture that both bundlefooter/footer_test.go and the Python side load? Otherwise the format can desync silently the next time either side gets refactored. (can be a follow up if you like)
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
The default bundle output filename is derived from the main package directory name (matching `go build`), resolved before the build so a bad `--output` fails fast. It never came from `BundleInfo.Name`, and the manifest carries no name field. Fix the stale renderManifest comment and ADR 0002 claim that said otherwise.
rejectOutputAlias already refused an --output that aliased the executable or source; extend it to the supplied --airflow-metadata file so `--output X --airflow-metadata X` fails fast instead of clobbering the manifest when the bundle is renamed into place.
Cover the inspect command's formatting and the bundlefooter.Read happy path with a table test exercising output with and without --source.
jason810496
commented
Jun 8, 2026
MemberAuthor
Thanks Amogh for the review!
This is #67956 (based on top of this one) comes to play! The Go SDK E2E test will involved the |
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A Go bundle ships as a single self-contained executable that also carries its own Dag source and manifest, so Airflow can discover and integrity-check it without running it. This PR adds
airflow-go-pack, the tool that produces and inspects that artifact.What
airflow-go-pack(cmd/airflow-go-pack/): runsgo buildon the user's bundle package, reads the binary's--airflow-metadatamanifest, and appends<source><manifest><trailer>so the output is one runnable bundle.inspect <bundle>prints the embedded manifest (and optionally the source) without executing the binary.--executablepacks a pre-built binary; if it cannot run on the host (foreign arch), the manifest is read by rebuilding the--sourcepackage for the host.internal/bundlefooter: theAFBNDL0164-byte trailer — source/metadata lengths, footer version, and a SHA-256 over the binary region for integrity. Matches what the PythonExecutableCoordinatorreads back.internal/airflowmetadata: the sharedairflow-metadata.yamlmanifest shape emitted by a bundle and rendered into the footer.bundle/bundlev1/bundlev1server: serves--airflow-metadataso a built bundle reports its DAG/task identity for packing.Was generative AI tooling used to co-author this PR?