APPLICATION GENERATOR FOR CONFLATION ALGORITHMS

This is an application generator for conflation algorithms in perl language.

This package includes the following:

- application generator for stemmers

* stemgen.pl - main perl routine
* apgen.pm - application generator class
* corpus.pm - corpus parsing class
* ngram.pm - utility class supporting n-gram specific operations
* porterd.pm - class implementing porter algorithm (developed version)
* sv.pm - utility class supporting successof variety specific operations
* util.pm - generic utlity class
* tempstem.pm - stemmer template used by the application generator

- rule files for the following stemmers:

* porter
* lovins
* k-stem
* sremoval
* paice
* successor variety

- unprocesses corpus file (in a separate zip file)

* unprocessed_corpus.txt

- processed corpus file

* corpus.txt

- usd dictionary

* usd.txt

- file containing the list of words that have been tested

* w.txt

USAGE

Usage:

	perl stemgen.pl -o [g|r|p] [-a algorithm/rule_file] [-k category]
[-c corpus_file] [-d dictionary]]
[-s word] [-t dice_threshold] [-u successor_var_cutoff] [-w word_file] [-g debug_level] [-p output_file]

Options:

 -o r - to run a stemmer
supported algorithms:
* porterd - developed porter
* and any other algorithm generated by using -g option
-w file with words to generate stem for
-s word to generate stem for
-d dictioary name (valid for dictionary type of stemmers)
-c corpus name (valid for n-gram, successor variety)
-t dice thrshold of ngram (valid for n-gram type of stemmers)
-u successor variety cutoff (valid for successor variety type of stemmers)
-o g - to generate a stemmer from a rule file
-k type of the stemmer: n - n-gram type algorithm
s - successor variety type algorithm
a - affix removal type algorithm
d - dictionary based algorithm
-r rule_file with .rl extension
rule files in this package:
* porter
* lovins
* kstem
* sremoval
* paice
* succvar (successor variety)
-o p - to parse a corpus file
-p output file (valid for parsing a corpus)

Examples:

 -- to generate a stemmer:
perl stemgen.pl -o g -a porter -k a
perl stemgen.pl -o g -a lovins -k a
perl stemgen.pl -o g -a sremoval -k a
perl stemgen.pl -o g -a paice -k a
perl stemgen.pl -o g -a kstem -k d
perl stemgen.pl -o g -a succvar -k s
-- to run a stemmer: perl stemgen.pl -o r -a porterd -w w.txt
perl porter_gen.pl -o r -a porter -s greatings
perl paice_gen.pl -o r -a paice -w w.txt
perl lovins_gen.pl -o r -a lovins -w w.txt
perl sremoval_gen.pl -o r -a sremoval -w w.txt
perl kstem_gen.pl -o r -a kstem -d usd.txt -w w.txt
perl succvar_gen.pl -o r -a succvar -c corpus.txt -w -w.txt
-- to parse a corpus file:
perl stemgen.pl -o p -c unprocessed_corpus.txt -p c.txt