Refactor tokenizer test and add to cmake - #8450

Merged
facebook-github-bot merged 1 commit into
mainfrom
lfq.cmake-tokenizer-test
Feb 15, 2025
Merged

Refactor tokenizer test and add to cmake#8450
facebook-github-bot merged 1 commit into
mainfrom
lfq.cmake-tokenizer-test

Conversation

@lucylq

@lucylqlucylq commented Feb 13, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Refactor tokenizer test to use env instead of resource path.

Add to cmake tests.

Test plan

build et

./install_executorch.sh

build test

CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install

test

cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 3.67 sec

@pytorch-bot

pytorch-botBot commented Feb 13, 2025•
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8450

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit b1a9127 with merge base 8148603 (image):

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-botfacebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 13, 2025
@lucylq
lucylqforce-pushed the lfq.cmake-tokenizer-test branch 6 times, most recently from 41b88a2 to 15d4cd0CompareFebruary 14, 2025 05:14
@lucylqlucylq changed the title tokenizerRefactor tokenizer test and add to cmakeFeb 14, 2025
@lucylq
lucylqforce-pushed the lfq.cmake-tokenizer-test branch from 15d4cd0 to 71976c7CompareFebruary 14, 2025 05:17
@lucylq
lucylq marked this pull request as ready for review February 14, 2025 05:18
@facebook-github-bot

facebook-github-bot commented Feb 14, 2025

Copy link
Copy Markdown
Contributor

@lucylq has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@lucylqlucylq added the release notes: build Changes related to build, including dependency upgrades, build flags, optimizations, etc. label Feb 14, 2025
facebook-github-bot pushed a commit that referenced this pull request Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
ddccce2
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot

facebook-github-bot commented Feb 14, 2025

Copy link
Copy Markdown
Contributor

This pull request was exported from Phabricator. Differential Revision: D69642007

facebook-github-bot pushed a commit that referenced this pull request Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
a72e252
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot

facebook-github-bot commented Feb 14, 2025

Copy link
Copy Markdown
Contributor

This pull request was exported from Phabricator. Differential Revision: D69642007

@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
b1a9127
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot

facebook-github-bot commented Feb 14, 2025

Copy link
Copy Markdown
Contributor

This pull request was exported from Phabricator. Differential Revision: D69642007

@facebook-github-bot
facebook-github-bot deleted the lfq.cmake-tokenizer-test branch February 15, 2025 01:26
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@larryliu0820larryliu0820larryliu0820 approved these changes

Assignees

No one assigned

Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedrelease notes: buildChanges related to build, including dependency upgrades, build flags, optimizations, etc.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@lucylq@facebook-github-bot@larryliu0820
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Refactor tokenizer test and add to cmake - #8450
Merged
facebook-github-bot merged 1 commit into
mainfrom 
lfq.cmake-tokenizer-test
Feb 15, 2025
Merged
Refactor tokenizer test and add to cmake#8450
facebook-github-bot merged 1 commit into
mainfrom 
lfq.cmake-tokenizer-test
Conversation
@lucylq
@lucylqlucylq
commented
Feb 13, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test plan
build et
./install_executorch.sh

build test
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install

test
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 3.67 sec

@pytorch-bot
pytorch-botBot
commented
Feb 13, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8450
Note: Links to docs will display an error until the docs builds have been completed.
✅ You can merge normally! (1 Unrelated Failure)
As of commit b1a9127 with merge base 8148603 (image):
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
@facebook-github-botfacebook-github-bot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Feb 13, 2025
@lucylq
lucylqforce-pushed
the
lfq.cmake-tokenizer-test
branch
6 times, most recently
from
41b88a2 to
15d4cd0CompareFebruary 14, 2025 05:14
@lucylqlucylq
changed the title
tokenizerRefactor tokenizer test and add to cmakeFeb 14, 2025
@lucylq
lucylqforce-pushed
the
lfq.cmake-tokenizer-test
branch
from
15d4cd0 to
71976c7CompareFebruary 14, 2025 05:17
@lucylq
lucylq
marked this pull request as ready for review
February 14, 2025 05:18
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
@lucylq has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
@lucylqlucylq
added
the 
release notes: build
Changes related to build, including dependency upgrades, build flags, optimizations, etc.
label
Feb 14, 2025
facebook-github-bot pushed a commit
that referenced
this pull request
Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
ddccce2
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
facebook-github-bot pushed a commit
that referenced
this pull request
Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
a72e252
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
b1a9127
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
@facebook-github-bot
facebook-github-bot
deleted the

lfq.cmake-tokenizer-test

branch
February 15, 2025 01:26
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@larryliu0820larryliu0820larryliu0820 approved these changes

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedrelease notes: buildChanges related to build, including dependency upgrades, build flags, optimizations, etc.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@lucylq@facebook-github-bot@larryliu0820

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Refactor tokenizer test and add to cmake - #8450
Merged
facebook-github-bot merged 1 commit into
mainfrom 
lfq.cmake-tokenizer-test
Feb 15, 2025
Merged
Refactor tokenizer test and add to cmake#8450
facebook-github-bot merged 1 commit into
mainfrom 
lfq.cmake-tokenizer-test
Conversation
@lucylq
@lucylqlucylq
commented
Feb 13, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test plan
build et
./install_executorch.sh

build test
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install

test
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 3.67 sec

@pytorch-bot
pytorch-botBot
commented
Feb 13, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8450
Note: Links to docs will display an error until the docs builds have been completed.
✅ You can merge normally! (1 Unrelated Failure)
As of commit b1a9127 with merge base 8148603 (image):
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
@facebook-github-botfacebook-github-bot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Feb 13, 2025
@lucylq
lucylqforce-pushed
the
lfq.cmake-tokenizer-test
branch
6 times, most recently
from
41b88a2 to
15d4cd0CompareFebruary 14, 2025 05:14
@lucylqlucylq
changed the title
tokenizerRefactor tokenizer test and add to cmakeFeb 14, 2025
@lucylq
lucylqforce-pushed
the
lfq.cmake-tokenizer-test
branch
from
15d4cd0 to
71976c7CompareFebruary 14, 2025 05:17
@lucylq
lucylq
marked this pull request as ready for review
February 14, 2025 05:18
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
@lucylq has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
@lucylqlucylq
added
the 
release notes: build
Changes related to build, including dependency upgrades, build flags, optimizations, etc.
label
Feb 14, 2025
facebook-github-bot pushed a commit
that referenced
this pull request
Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
ddccce2
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
facebook-github-bot pushed a commit
that referenced
this pull request
Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
a72e252
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
b1a9127
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
@facebook-github-bot
facebook-github-bot
deleted the

lfq.cmake-tokenizer-test

branch
February 15, 2025 01:26
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@larryliu0820larryliu0820larryliu0820 approved these changes

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedrelease notes: buildChanges related to build, including dependency upgrades, build flags, optimizations, etc.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@lucylq@facebook-github-bot@larryliu0820

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Refactor tokenizer test and add to cmake - #8450
Merged
facebook-github-bot merged 1 commit into
mainfrom 
lfq.cmake-tokenizer-test
Feb 15, 2025
Merged
Refactor tokenizer test and add to cmake#8450
facebook-github-bot merged 1 commit into
mainfrom 
lfq.cmake-tokenizer-test
Conversation
@lucylq
@lucylqlucylq
commented
Feb 13, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test plan
build et
./install_executorch.sh

build test
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install

test
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 3.67 sec

@pytorch-bot
pytorch-botBot
commented
Feb 13, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8450
Note: Links to docs will display an error until the docs builds have been completed.
✅ You can merge normally! (1 Unrelated Failure)
As of commit b1a9127 with merge base 8148603 (image):
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
@facebook-github-botfacebook-github-bot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Feb 13, 2025
@lucylq
lucylqforce-pushed
the
lfq.cmake-tokenizer-test
branch
6 times, most recently
from
41b88a2 to
15d4cd0CompareFebruary 14, 2025 05:14
@lucylqlucylq
changed the title
tokenizerRefactor tokenizer test and add to cmakeFeb 14, 2025
@lucylq
lucylqforce-pushed
the
lfq.cmake-tokenizer-test
branch
from
15d4cd0 to
71976c7CompareFebruary 14, 2025 05:17
@lucylq
lucylq
marked this pull request as ready for review
February 14, 2025 05:18
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
@lucylq has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
@lucylqlucylq
added
the 
release notes: build
Changes related to build, including dependency upgrades, build flags, optimizations, etc.
label
Feb 14, 2025
facebook-github-bot pushed a commit
that referenced
this pull request
Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
ddccce2
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
facebook-github-bot pushed a commit
that referenced
this pull request
Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
a72e252
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
b1a9127
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
@facebook-github-bot
facebook-github-bot
deleted the

lfq.cmake-tokenizer-test

branch
February 15, 2025 01:26
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@larryliu0820larryliu0820larryliu0820 approved these changes

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedrelease notes: buildChanges related to build, including dependency upgrades, build flags, optimizations, etc.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@lucylq@facebook-github-bot@larryliu0820

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Refactor tokenizer test and add to cmake - #8450
Merged
facebook-github-bot merged 1 commit into
mainfrom 
lfq.cmake-tokenizer-test
Feb 15, 2025
Merged
Refactor tokenizer test and add to cmake#8450
facebook-github-bot merged 1 commit into
mainfrom 
lfq.cmake-tokenizer-test
Conversation
@lucylq
@lucylqlucylq
commented
Feb 13, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test plan
build et
./install_executorch.sh

build test
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install

test
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 3.67 sec

@pytorch-bot
pytorch-botBot
commented
Feb 13, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8450
Note: Links to docs will display an error until the docs builds have been completed.
✅ You can merge normally! (1 Unrelated Failure)
As of commit b1a9127 with merge base 8148603 (image):
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
@facebook-github-botfacebook-github-bot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Feb 13, 2025
@lucylq
lucylqforce-pushed
the
lfq.cmake-tokenizer-test
branch
6 times, most recently
from
41b88a2 to
15d4cd0CompareFebruary 14, 2025 05:14
@lucylqlucylq
changed the title
tokenizerRefactor tokenizer test and add to cmakeFeb 14, 2025
@lucylq
lucylqforce-pushed
the
lfq.cmake-tokenizer-test
branch
from
15d4cd0 to
71976c7CompareFebruary 14, 2025 05:17
@lucylq
lucylq
marked this pull request as ready for review
February 14, 2025 05:18
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
@lucylq has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
@lucylqlucylq
added
the 
release notes: build
Changes related to build, including dependency upgrades, build flags, optimizations, etc.
label
Feb 14, 2025
facebook-github-bot pushed a commit
that referenced
this pull request
Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
ddccce2
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
facebook-github-bot pushed a commit
that referenced
this pull request
Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
a72e252
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
b1a9127
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
@facebook-github-bot
facebook-github-bot
deleted the

lfq.cmake-tokenizer-test

branch
February 15, 2025 01:26
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@larryliu0820larryliu0820larryliu0820 approved these changes

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedrelease notes: buildChanges related to build, including dependency upgrades, build flags, optimizations, etc.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@lucylq@facebook-github-bot@larryliu0820

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Refactor tokenizer test and add to cmake - #8450
Merged
facebook-github-bot merged 1 commit into
mainfrom 
lfq.cmake-tokenizer-test
Feb 15, 2025
Merged
Refactor tokenizer test and add to cmake#8450
facebook-github-bot merged 1 commit into
mainfrom 
lfq.cmake-tokenizer-test
Conversation
@lucylq
@lucylqlucylq
commented
Feb 13, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test plan
build et
./install_executorch.sh

build test
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install

test
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 3.67 sec

@pytorch-bot
pytorch-botBot
commented
Feb 13, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8450
Note: Links to docs will display an error until the docs builds have been completed.
✅ You can merge normally! (1 Unrelated Failure)
As of commit b1a9127 with merge base 8148603 (image):
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
@facebook-github-botfacebook-github-bot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Feb 13, 2025
@lucylq
lucylqforce-pushed
the
lfq.cmake-tokenizer-test
branch
6 times, most recently
from
41b88a2 to
15d4cd0CompareFebruary 14, 2025 05:14
@lucylqlucylq
changed the title
tokenizerRefactor tokenizer test and add to cmakeFeb 14, 2025
@lucylq
lucylqforce-pushed
the
lfq.cmake-tokenizer-test
branch
from
15d4cd0 to
71976c7CompareFebruary 14, 2025 05:17
@lucylq
lucylq
marked this pull request as ready for review
February 14, 2025 05:18
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
@lucylq has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
@lucylqlucylq
added
the 
release notes: build
Changes related to build, including dependency upgrades, build flags, optimizations, etc.
label
Feb 14, 2025
facebook-github-bot pushed a commit
that referenced
this pull request
Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
ddccce2
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
facebook-github-bot pushed a commit
that referenced
this pull request
Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
a72e252
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
b1a9127
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
@facebook-github-bot
facebook-github-bot
deleted the

lfq.cmake-tokenizer-test

branch
February 15, 2025 01:26
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@larryliu0820larryliu0820larryliu0820 approved these changes

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedrelease notes: buildChanges related to build, including dependency upgrades, build flags, optimizations, etc.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@lucylq@facebook-github-bot@larryliu0820

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Refactor tokenizer test and add to cmake - #8450
Merged
facebook-github-bot merged 1 commit into
mainfrom 
lfq.cmake-tokenizer-test
Feb 15, 2025
Merged
Refactor tokenizer test and add to cmake#8450
facebook-github-bot merged 1 commit into
mainfrom 
lfq.cmake-tokenizer-test
Conversation
@lucylq
@lucylqlucylq
commented
Feb 13, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test plan
build et
./install_executorch.sh

build test
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install

test
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 3.67 sec

@pytorch-bot
pytorch-botBot
commented
Feb 13, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8450
Note: Links to docs will display an error until the docs builds have been completed.
✅ You can merge normally! (1 Unrelated Failure)
As of commit b1a9127 with merge base 8148603 (image):
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
@facebook-github-botfacebook-github-bot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Feb 13, 2025
@lucylq
lucylqforce-pushed
the
lfq.cmake-tokenizer-test
branch
6 times, most recently
from
41b88a2 to
15d4cd0CompareFebruary 14, 2025 05:14
@lucylqlucylq
changed the title
tokenizerRefactor tokenizer test and add to cmakeFeb 14, 2025
@lucylq
lucylqforce-pushed
the
lfq.cmake-tokenizer-test
branch
from
15d4cd0 to
71976c7CompareFebruary 14, 2025 05:17
@lucylq
lucylq
marked this pull request as ready for review
February 14, 2025 05:18
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
@lucylq has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
@lucylqlucylq
added
the 
release notes: build
Changes related to build, including dependency upgrades, build flags, optimizations, etc.
label
Feb 14, 2025
facebook-github-bot pushed a commit
that referenced
this pull request
Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
ddccce2
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
facebook-github-bot pushed a commit
that referenced
this pull request
Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
a72e252
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
b1a9127
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
@facebook-github-bot
facebook-github-bot
deleted the

lfq.cmake-tokenizer-test

branch
February 15, 2025 01:26
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@larryliu0820larryliu0820larryliu0820 approved these changes

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedrelease notes: buildChanges related to build, including dependency upgrades, build flags, optimizations, etc.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@lucylq@facebook-github-bot@larryliu0820

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Refactor tokenizer test and add to cmake - #8450
Merged
facebook-github-bot merged 1 commit into
mainfrom 
lfq.cmake-tokenizer-test
Feb 15, 2025
Merged
Refactor tokenizer test and add to cmake#8450
facebook-github-bot merged 1 commit into
mainfrom 
lfq.cmake-tokenizer-test
Conversation
@lucylq
@lucylqlucylq
commented
Feb 13, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test plan
build et
./install_executorch.sh

build test
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install

test
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 3.67 sec

@pytorch-bot
pytorch-botBot
commented
Feb 13, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8450
Note: Links to docs will display an error until the docs builds have been completed.
✅ You can merge normally! (1 Unrelated Failure)
As of commit b1a9127 with merge base 8148603 (image):
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
@facebook-github-botfacebook-github-bot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Feb 13, 2025
@lucylq
lucylqforce-pushed
the
lfq.cmake-tokenizer-test
branch
6 times, most recently
from
41b88a2 to
15d4cd0CompareFebruary 14, 2025 05:14
@lucylqlucylq
changed the title
tokenizerRefactor tokenizer test and add to cmakeFeb 14, 2025
@lucylq
lucylqforce-pushed
the
lfq.cmake-tokenizer-test
branch
from
15d4cd0 to
71976c7CompareFebruary 14, 2025 05:17
@lucylq
lucylq
marked this pull request as ready for review
February 14, 2025 05:18
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
@lucylq has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.
@lucylqlucylq
added
the 
release notes: build
Changes related to build, including dependency upgrades, build flags, optimizations, etc.
label
Feb 14, 2025
facebook-github-bot pushed a commit
that referenced
this pull request
Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
ddccce2
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
facebook-github-bot pushed a commit
that referenced
this pull request
Feb 14, 2025
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
a72e252
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
@lucylq@facebook-github-bot
Refactor tokenizer test and add to cmake (#8450)
b1a9127
Summary:
Refactor tokenizer test to use env instead of resource path.
Add to cmake tests.
Test Plan:
## Internal
```
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbsource//xplat/executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_tiktoken
buck2 test fbcode//executorch/extension/llm/tokenizer/test:test_bpe_tokenizer
```
## OSS
build et
```
./install_executorch.sh
```
build test
```
CMAKE_PREFIX_PATH="$(python3 -c 'import torch as _; print(_.__path__[0])')"
cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DCMAKE_PREFIX_PATH="${CMAKE_PREFIX_PATH}" \
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_LLM=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=OFF \
-DEXECUTORCH_BUILD_TESTS=ON \
-Bcmake-out
cmake --build cmake-out -j9 --target install
```
test
```
cd cmake-out
ctest -R tokenizer
Test project /data/users/lfq/executorch/cmake-out
Start 54: extension_llm_tokenizer_test
1/1 Test #54: extension_llm_tokenizer_test ..... Passed 3.66 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) =--sanitized--
Differential Revision: D69642007
Pulled By: lucylq
@facebook-github-bot
facebook-github-bot
commented
Feb 14, 2025

Copy link


Copy Markdown

Contributor
This pull request was exported from Phabricator. Differential Revision: D69642007
@facebook-github-bot
facebook-github-bot
deleted the

lfq.cmake-tokenizer-test

branch
February 15, 2025 01:26
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@larryliu0820larryliu0820larryliu0820 approved these changes

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedrelease notes: buildChanges related to build, including dependency upgrades, build flags, optimizations, etc.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@lucylq@facebook-github-bot@larryliu0820