refactor: better memory footprint for thin svd - #1753
Merged
Merged
Conversation
polvalente
marked this pull request as ready for review
May 21, 2026 21:36
nshkrdotcom
added a commit
to nshkrdotcom/trinity_coordinator
that referenced
this pull request
May 21, 2026
This commit picks up the Nx-side fix that closes the Apple Silicon SVD-OOM blocker upstream and bumps the surrounding dep stack accordingly. Validated end-to-end on CUDA; validated end-to-end on Apple by polvalente (Nx core) prior to this commit. Deps: - nx pinned to GitHub elixir-nx/nx@6424c89 (post-v0.12.0 main, carries elixir-nx/nx#1753 — better memory footprint for thin SVD; both EMLX and EXLA benefit; the Apple OOM on the 151,936 x 1024 embedder is gone). - exla pinned to the same Nx repo at the same commit (sparse: 'exla', v0.12.0). - bumblebee bumped from github.com/elixir-nx/bumblebee@0fd8114 (pre-v0.7.0) to github.com/elixir-nx/bumblebee@d0774e8 (post-v0.7.0 main; required for Nx 0.12 compat). - xla 0.10.x is the resolved version (was 0.9.x). cuda13 is newly accepted by the XLA preflight; cuda12 remains recommended default. - EMLX is deliberately NOT in our deps. optional: true does not prevent Mix from starting it on Linux/CUDA hosts where its Metal/MLX NIF cannot load. Apple users add {:emlx, '~> 0.3'} to their parent app; the :emlx runtime profile resolves the backend at runtime via Code.ensure_loaded?/1. Documentation: - README, guides/onboarding.md, docs/production_qwen_slm_profile.md updated with the new resolved dep versions. - guides/troubleshooting.md 'XLA_TARGET=cuda13' section rewritten: cuda13 is now accepted; the rejection example uses cuda14. - guides/troubleshooting.md 'EMLX OOM on the embedder SVD' section rewritten to credit both fixes (EMLX 0.3.0 thin-SVD fall-through + Nx PR #1753 default-impl refactor). - guides/runtime_profiles.md EMLX caveats section updated to credit PR #1753 and note that polvalente confirmed end-to-end Apple validation (37/37 prompt eval pass) without EMLXAxon rewrites. - docs/bumblebee_unpin_playbook.md updated to reflect the new ref. Tests: - test/build_support/xla_target_validator_test.exs updated for the cuda13-now-accepted reality and the new bundled xla 0.10.x message. Version: - mix.exs @Version 0.1.0 -> 0.2.0. - CHANGELOG: new 0.2.0 entry summarising Phase B, C, D. Gates (all green on CUDA): - mix format - mix compile --warnings-as-errors - mix test: 262 tests, 0 failures (was 261; +1 'accepts cuda13' case) - mix credo --strict: 0 issues - mix dialyzer: 0 errors - mix docs --warnings-as-errors clean - XLA_TARGET=cuda12 mix run examples/qwen_router_prompt_eval.exs --snapshot examples/fixtures/qwen_router_prompt_eval_logits.json --determinism-runs 2 -> 37/37 PASS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This changes the implementation of "non-full" SVD to one that requires less memory.