rust-autocomplete

rust-autocomplete is a rudimentary word completion library built in rust. It is inspired by Rodrigo Palacios's ELI5 explanation of an autocompletion AI found here. I figured it would be a good way to continue learning Rust.

Usage

Using SimpleWordPredictor is easy. First it must be trained.

Training

Autocomplete will work better with a larger corpus of training data available to it. In this repository are provided two types of training data. The first is the file named training_data.csv. This is an already processed count of a large amount of input text. The other is the file named big.txt which is provided by [Peter Norvig] (http://norvig.com). This is a raw collection of several books.

I also recommend watching Peter Norvig's lecture titled The Unreasonable Effectiveness of Data.

With training_data.csv

If you use the provided training data found in training_data.csv, you only have to call SimpleWordPredictor::from_file() with a path to the training data.

With big.txt or other corpus

This method requires you to do a bit more heavy lifting. You will need open your corpus and make sure the only characters are the ones that you want in your training data. I settled on the characters [a-z] and spaces. You then feed this data into SimpleWordTrainer using its train_str() method. Before you can predict, SimpleWordTrainer must be converted to SimpleWordPredictor, which changes its internal representation of the training data.

This is how I trained autocomplete to create training_data.csv:

fnclean_line(line:String) -> String{letmut new_string = String::new();let line_bytes = line.bytes();for byte in line_bytes {if byte == 32{
new_string.push(from_u32(byte asu32).unwrap());}elseif byte >= 97 && byte <= 122{
new_string.push(from_u32(byte asu32).unwrap());}elseif byte >= 64 && byte <= 90{
new_string.push(from_u32((byte + 32)asu32).unwrap());}}
new_string
}fntrain_model(model:&mutSimpleWordTrainer,path:Path){letmut file = BufferedReader::new(File::open(&path));for line in file.lines(){let cleaned_line = clean_line(line.unwrap());
model.train_str(cleaned_line.as_slice());}}fnmain(){letmut model = SimpleWordTrainer::new();let file_path = Path::new("big.txt");println!("Training.");train_model(&mut model, file_path);println!("Finalizing.");let predictor = model.finalize();// Save predictor here or use it to run predictions}

Predicting

Call SimpleWordPredictor.predict() with a &str to get back a vec<PredictionEntry>. PredictionEntry has public fields score and word.

loop{print!("Input: ");let input = old_io::stdin().read_line().ok().expect("Failed to read line.");let output = predictor.predict(input.trim());println!("Score\tWord");for entry in output {println!("{}\t{}", entry.score, entry.word);}}