[Windows] Re-enable tokenizers

[GregoryComer]

Summary

After meta-pytorch/tokenizers#121 and #13888, we should be able to enable tokenizers on Windows.

Test plan

CI. Also verified that install functions on a local Windows machine.

cc @larryliu0820

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14180

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures, 1 Unrelated Failure

As of commit caafb73 with merge base f1ca55a ():

NEW FAILURES - The following jobs have failed:

• Build Windows Wheels / pytorch/executorch / build-wheel-py3_10-cpu (gh)
  Process completed with exit code 1.
• Build Windows Wheels / pytorch/executorch / upload / upload-wheel-py3_10-cpu (gh)
  Unable to download artifact(s): Artifact not found for name: pytorch_executorch__3.10_cpu_x64
• pull / test-qnn-wheel-packages-linux (3.10) / linux-job (gh)
  RuntimeError: Command docker exec -t af1e8caee1b1a1b879e7aadc041cbec6e5e2dd69d14114dc85fab2c31cc30fd8 /exec failed with exit code 127
• pull / test-qnn-wheel-packages-linux (3.11) / linux-job (gh)
  RuntimeError: Command docker exec -t e9a52c60c1abf88e1725baae02a208f8c348547f7edcf3d699c1280a1c584027 /exec failed with exit code 127
• pull / test-qnn-wheel-packages-linux (3.12) / linux-job (gh)
  RuntimeError: Command docker exec -t 04c1236300c24766da7ad6568cbfecc3d1524904cdfacb7fcc59b0c20c3b2528 /exec failed with exit code 127

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / test-moshi-linux / linux-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[GregoryComer]

I'm seeing some path length issues when building tokenizers in-place. I'm going to see if I can set up a symlink in the windows setup CI script.

[GregoryComer]

Rebasing to test with #13923 to be safe, but will land today if everything looks good.

[GregoryComer]

Removing 1.0 milestone from this PR in favor of using the pip package, when available. I'd need to make some more aggressive infra changes to the wheel build to make this work when building from source.

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.