ocrmultieval

Proof-of-concept for extensible evaluation of (intermediate) results of an OCR workflow

Installation

make deps install

Architecture

All evaluation functionality is provided by backends.

Every backend inherits from EvalBackend and must implement a compare_files method, that accepts paths to and media types of the Ground Truth and detection results, does the actual evaluation and returns an EvalReport.

An EvalReport is a map of metrics to their resp. value and can be serialized as JSON or CSV for further processing/analysis.

The glue code for running the backends is in ocrmultieval.runner.py.

Usage

CLI

The ocrmultieval compare command line tool allows evaluating individual pages of GT and detection with any of the available backends.

Usage: ocrmultieval compare [OPTIONS] {dinglehopper|ocrevalUAtion|PrimaTextEva
l|CorAsvAnnEval|CorAsvAnnCompare|OcrdSegmentEvalua
te|IsriOcreval} GT_FILE OCR_FILE
Options:
--gt-mediatype TEXT
--ocr-mediatype TEXT
--format [csv|json|yaml|xml]
-g, --pageId TEXT pageId to uniquely identify pages in a work
--help Show this message and exit.

OCR-D processor

The ocrd-ocrmultieval command line tool implments the OCR-D processor API and can be used to process complete workspaces.

Usage: ocrd-ocrmultieval [OPTIONS]
Evaluate
> Eval processor
Options:
-I, --input-file-grp USE File group(s) used as input
-O, --output-file-grp USE File group(s) used as output
-g, --page-id ID Physical page ID(s) to process
--overwrite Remove existing output pages/images
(with --page-id, remove only those)
-p, --parameter JSON-PATH Parameters, either verbatim JSON string
or JSON file path
-P, --param-override KEY VAL Override a single JSON object key-value pair,
taking precedence over --parameter
-m, --mets URL-PATH URL or file path of METS to process
-w, --working-dir PATH Working directory of local workspace
-l, --log-level [OFF|ERROR|WARN|INFO|DEBUG|TRACE]
Log level
-C, --show-resource RESNAME Dump the content of processor resource RESNAME
-L, --list-resources List names of processor resources
-J, --dump-json Dump tool description as JSON and exit
-h, --help This help message
-V, --version Show version
Parameters:
"backend" [string - "PrimaTextEval"]
Backend to use
Possible values: ["PrimaTextEval", "ocrevalUAtion", "dinglehopper",
"OcrdSegmentEvaluate", "IsriOcreval", "CorAsvAnnCompare"]
"format" [string - "csv"]
Output format
Possible values: ["csv", "json", "yaml", "xml"]
"config" [object]
Configuration to override default
Default Wiring:
['GT,OCR1'] -> ['GT_VS_OCR1']