rust-tinysegmenter

Rust implementation of TinySegmenter, which is a compact Japanese tokenizer.

Install

Adding the following to the Cargo.toml in your project:

[dependencies]
tinysegmenter = "0.1"

and import using extern crate:

externcrate tinysegmenter;

let tokens = tinysegmenter::tokenize("私の名前は中野です");println!("{}",&tokens.join("|"));// 私|の|名前|は|中野|です

Licensed under the MIT license.