Add validation command to CASE-Utilities-Python

.gitmodules

@@ -1,3 +1,6 @@
+[submodule "dependencies/CASE"]
+	path = dependencies/CASE
+	url = https://github.com/casework/CASE.git
 [submodule "dependencies/CASE-Examples-QC"]
 	path = dependencies/CASE-Examples-QC
 	url = https://github.com/ajnelson-nist/CASE-Examples-QC.git

CONTRIBUTE.md

@@ -0,0 +1,34 @@
+# Contributing to CASE-Utilities-Python
+
+
+## Deploying a new ontology version
+
+1. After cloning this repository, ensure the CASE submodule is checked out. This can be done with either `git submodule init && git submodule update`, `make .git_submodule_init.done.log`, or `make check`.
+2. Update the CASE submodule pointer to the new tagged release.
+3. The version of CASE is also hard-coded in [`case_utils/ontology/version_info.py`](case_utils/ontology/version_info.py). Edit the variable `CURRENT_CASE_VERSION`.
+4. From the top source directory, run `make clean`. This guarantees a clean state of this repository as well as the ontology submodules.
+5. Still from the top source directory, run `make`.
+6. Any new `.ttl` files will be created under [`case_utils/ontology/`](case_utils/ontology/). Use `git add` to add each of them. (The patch-weight of these files could overshadow manual revisions, so it is fine to commit the built files after the manual changes are committed.)
+
+Here is a sample sequence of shell commands to run the build:
+
+```bash
+# (Starting from fresh `git clone`.)
+make check
+pushd dependencies/CASE
+ git checkout master
+ git pull
+popd
+git add dependencies/CASE
+# (Here, edits should be made to case_utils/ontology/version_info.py)
+make
+pushd case_utils/ontology
+ git add case-0.6.0.ttl # Assuming CASE 0.6.0 was just released.
+ # and/or
+ git add uco-0.8.0.ttl # Assuming UCO 0.8.0 was adopted in CASE 0.6.0.
+popd
+make check
+# Assuming `make check` passes:
+git commit -m "Update CASE ontology pointer to version 0.6.0" dependencies/CASE case_utils/ontology/version_info.py
+git commit -m "Build CASE 0.6.0.ttl" case_utils/ontology/case-0.6.0.ttl
+```

Makefile

@@ -15,7 +15,13 @@ SHELL := /bin/bash
 
 PYTHON3 ?= $(shell which python3.9 2>/dev/null || which python3.8 2>/dev/null || which python3.7 2>/dev/null || which python3.6 2>/dev/null || which python3)
 
-all:
+case_version := $(shell $(PYTHON3) case_utils/ontology/version_info.py)
+ifeq ($(case_version),)
+$(error Unable to determine CASE version)
+endif
+
+all: \
+ .ontology.done.log
 
 .PHONY: \
  download
@@ -35,10 +41,28 @@ all:
 	$(MAKE) \
  --directory dependencies/CASE-Examples-QC/tests \
  ontology_vocabulary.txt
+	test -r dependencies/CASE/ontology/master/case.ttl \
+ || (git submodule init dependencies/CASE && git submodule update dependencies/CASE)
+	test -r dependencies/CASE/ontology/master/case.ttl
+	$(MAKE) \
+ --directory dependencies/CASE \
+ .git_submodule_init.done.log \
+ .lib.done.log
+	touch $@
+
+.ontology.done.log: \
+ dependencies/CASE/ontology/master/case.ttl
+	# Do not rebuild the current ontology file if it is already present. It is expected not to change once built.
+	# touch -c: Do not create the file if it does not exist. This will convince the recursive make nothing needs to be done if the file is present.
+	touch -c case_utils/ontology/case-$(case_version).ttl
+	$(MAKE) \
+ --directory case_utils/ontology
+	# Confirm the current monolithic file is in place.
+	test -r case_utils/ontology/case-$(case_version).ttl
 	touch $@
 
 check: \
- .git_submodule_init.done.log
+ .ontology.done.log
 	$(MAKE) \
  PYTHON3=$(PYTHON3) \
  --directory tests \
@@ -49,12 +73,32 @@ clean:
  --directory tests \
  clean
 	@rm -f \
- .git_submodule_init.done.log
+ .*.done.log
+	@# 'clean' in the ontology directory should only happen when testing and building new ontology versions. Hence, it is not called from the top-level Makefile.
+	@test ! -r dependencies/CASE/README.md \
+ || $(MAKE) \
+ --directory dependencies/CASE \
+ clean
+	@# Restore CASE validation output files that do not affect CASE build process.
+	@test ! -r dependencies/CASE/README.md \
+ || ( \
+ cd dependencies/CASE \
+ && git checkout \
+ -- \
+ tests/examples \
+ || true \
+ )
 	@#Remove flag files that are normally set after deeper submodules and rdf-toolkit are downloaded.
 	@rm -f \
  dependencies/CASE-Examples-QC/.git_submodule_init.done.log \
  dependencies/CASE-Examples-QC/.lib.done.log
 
+# This recipe guarantees timestamp update order, and is otherwise intended to be a no-op.
+dependencies/CASE/ontology/master/case.ttl: \
+ .git_submodule_init.done.log
+	test -r $@
+	touch $@
+
 distclean: \
  clean
 	@rm -rf \

README.md

@@ -20,6 +20,33 @@ Installation is demonstrated in the `.venv.done.log` target of the [`tests/`](te
 ## Usage
 
 
+### `case_validate`
+
+This repository provides `case_validate` as an adaptation of the `pyshacl` command from [RDFLib's pySHACL](https://github.com/RDFLib/pySHACL). The command-line interface is adapted to run as though `pyshacl` were provided the full CASE ontology (and adopted full UCO ontology) as both a shapes and ontology graph. "Compiled" (or, "aggregated") CASE ontologies are in the [`case_utils/ontology/`](case_utils/ontology/) directory, and are installed with `pip`, so data validation can occur without requiring networking after this repository is installed.
+
+To see a human-readable validation report of an instance-data file:
+
+```bash
+case_validate input.json
+```
+
+If `input.json` is not conformant, a report will be emitted, and `case_validate` will exit with status `1`. (This is a `pyshacl` behavior, where `0` and `1` report validation success. Status of >`1` is for other errors.)
+
+To produce the validation report as a machine-readable graph output, the `--format` flag can be used to modify the output format:
+
+```bash
+case_validate --format turtle input.json > result.ttl
+```
+
+To use one or more supplementary ontology files, the `--ontology-graph` flag can be used, more than once if desired, to supplement the selected CASE version:
+
+```bash
+case_validate --ontology-graph internal_ontology.ttl --ontology-graph experimental_shapes.ttl input.json
+```
+
+Other flags are reviewable with `case_validate --help`.
+
+
 ### `case_file`
 
 To characterize a file, including hashes:
@@ -86,10 +113,9 @@ This project follows [SEMVER 2.0.0](https://semver.org/) where versions are decl
 
 ## Ontology versions supported
 
-This repository supports the ontology versions that are linked as submodules in the [CASE Examples QC](https://github.com/ajnelson-nist/CASE-Examples-QC) repository. Currently, the ontology versions are:
+This repository supports the CASE ontology version that is linked as a submodule [here](dependencies/CASE). The CASE version is encoded as a variable (and checked in unit tests) in [`case_utils/ontology/version_info.py`](case_utils/ontology/version_info.py), and used throughout this code base, as `CURRENT_CASE_VERSION`.
 
-* CASE - 0.4.0
-* UCO - 0.6.0
+For instructions on how to update the CASE version for an ontology release, see [`CONTRIBUTE.md`](CONTRIBUTE.md).
 
 
 ## Repository locations

case_utils/case_validate/__init__.py

@@ -0,0 +1,191 @@
+#!/usr/bin/env python3
+
+# This software was developed at the National Institute of Standards
+# and Technology by employees of the Federal Government in the course
+# of their official duties. Pursuant to title 17 Section 105 of the
+# United States Code this software is not subject to copyright
+# protection and is in the public domain. NIST assumes no
+# responsibility whatsoever for its use by other parties, and makes
+# no guarantees, expressed or implied, about its quality,
+# reliability, or any other characteristic.
+#
+# We would appreciate acknowledgement if the software is used.
+
+"""
+This script provides a wrapper to the pySHACL command line tool,
+available here:
+https://github.com/RDFLib/pySHACL
+
+Portions of the pySHACL command line interface are preserved and passed
+through to the underlying pySHACL validation functionality.
+
+Other portions of the pySHACL command line interface are adapted to
+CASE, specifically to support CASE and UCO as ontologies that store
+subclass hierarchy and node shapes together (rather than as separate
+ontology and shape graphs). More specifically to CASE, if no particular
+ontology or shapes graph is requested, the most recent version of CASE
+will be used. (That most recent version is shipped with this package as
+a monolithic file; see case_utils.ontology if interested in further
+details.)
+"""
+
+__version__ = "0.1.0"
+
+import argparse
+import importlib.resources
+import logging
+import os
+import pathlib
+import sys
+import typing
+
+import rdflib.util # type: ignore
+import pyshacl # type: ignore
+
+import case_utils.ontology
+
+from case_utils.ontology.version_info import *
+
+_logger = logging.getLogger(os.path.basename(__file__))
+
+def main() -> None:
+ parser = argparse.ArgumentParser(description="CASE wrapper to pySHACL command line tool.")
+
+ # Configure debug logging before running parse_args, because there
+ # could be an error raised before the construction of the argument
+ # parser.
+ logging.basicConfig(level=logging.DEBUG if ("--debug" in sys.argv or "-d" in sys.argv) else logging.INFO)
+
+ case_version_choices_list = ["none", "case-" + CURRENT_CASE_VERSION]
+
+ # Add arguments specific to case_validate.
+ parser.add_argument(
+ '-d',
+ '--debug',
+ action='store_true',
+ help='Output additional runtime messages.'
+ )
+ parser.add_argument(
+ "--built-version",
+ choices=tuple(case_version_choices_list),
+ default="case-"+CURRENT_CASE_VERSION,
+ help="Monolithic aggregation of CASE ontology files at certain versions. Does not require networking to use. Default is most recent CASE release."
+ )
+ parser.add_argument(
+ "--ontology-graph",
+ action="append",
+ help="Combined ontology (i.e. subclass hierarchy) and shapes (SHACL) file, in any format accepted by rdflib recognized by file extension (e.g. .ttl). Will supplement ontology selected by --built-version. Can be given multiple times."
+ )
+
+ # Inherit arguments from pyshacl.
+ parser.add_argument(
+ '--abort',
+ action='store_true',
+ help='(As with pyshacl CLI) Abort on first invalid data.'
+ )
+ parser.add_argument(
+ '-w',
+ '--allow-warnings',
+ action='store_true',
+ help='(As with pyshacl CLI) Shapes marked with severity of Warning or Info will not cause result to be invalid.',
+ )
+ parser.add_argument(
+ "-f",
+ "--format",
+ choices=('human', 'turtle', 'xml', 'json-ld', 'nt', 'n3'),
+ default='human',
+ help="(ALMOST as with pyshacl CLI) Choose an output format. Default is \"human\". Difference: 'table' not provided."
+ )
+ parser.add_argument(
+ '-im',
+ '--imports',
+ action='store_true',
+ help='(As with pyshacl CLI) Allow import of sub-graphs defined in statements with owl:imports.',
+ )
+ parser.add_argument(
+ '-i',
+ '--inference',
+ choices=('none', 'rdfs', 'owlrl', 'both'),
+ default='none',
+ help="(As with pyshacl CLI) Choose a type of inferencing to run against the Data Graph before validating. Default is \"none\".",
+ )
+ parser.add_argument(
+ '-o',
+ '--output',
+ dest='output',
+ nargs='?',
+ type=argparse.FileType('x'),
+ help="(ALMOST as with pyshacl CLI) Send output to a file. If absent, output will be written to stdout. Difference: If specified, file is expected not to exist. Clarification: Does NOT influence --format flag's default value of \"human\". (I.e., any machine-readable serialization format must be specified with --format.)",
+ default=sys.stdout,
+ )
+
+ parser.add_argument("in_graph")
+
+ args = parser.parse_args()
+
+ data_graph = rdflib.Graph()
+ data_graph.parse(args.in_graph)
+
+ ontology_graph = rdflib.Graph()
+ if args.built_version != "none":
+ ttl_filename = args.built_version + ".ttl"
+ _logger.debug("ttl_filename = %r.", ttl_filename)
+ ttl_data = importlib.resources.read_text(case_utils.ontology, ttl_filename)
+ ontology_graph.parse(data=ttl_data, format="turtle")
+ if args.ontology_graph:
+ for arg_ontology_graph in args.ontology_graph:
+ _logger.debug("arg_ontology_graph = %r.", arg_ontology_graph)
+ ontology_graph.parse(arg_ontology_graph)
+
+ # Determine output format.
+ # pySHACL's determination of output formatting is handled solely
+ # through the -f flag. Other CASE CLI tools handle format
+ # determination by output file extension. case_validate will defer
+ # to pySHACL behavior, as other CASE tools don't (at the time of
+ # this writing) have the value "human" as an output format.
+ validator_kwargs : typing.Dict[str, str] = dict()
+ if args.format != "human":
+ validator_kwargs['serialize_report_graph'] = args.format
+
+ validate_result : typing.Tuple[
+ bool,
+ typing.Union[Exception, bytes, str, rdflib.Graph],
+ str
+ ]
+ validate_result = pyshacl.validate(
+ data_graph,
+ shacl_graph=ontology_graph,
+ ont_graph=ontology_graph,
+ inference=args.inference,
+ abort_on_first=args.abort,
+ allow_warnings=True if args.allow_warnings else False,
+ debug=True if args.debug else False,
+ do_owl_imports=True if args.imports else False,
+ **validator_kwargs
+ )
+
+ # Relieve RAM of the data graph after validation has run.
+ del data_graph
+
+ conforms = validate_result[0]
+ validation_graph = validate_result[1]
+ validation_text = validate_result[2]
+
+ # NOTE: The output logistics code is adapted from pySHACL's file
+ # pyshacl/cli.py. This section should be monitored for code drift.
+ if args.format == "human":
+ args.output.write(validation_text)
+ else:
+ if isinstance(validation_graph, rdflib.Graph):
+ raise NotImplementedError("rdflib.Graph expected not to be created from --format value %r." % args.format)
+ elif isinstance(validation_graph, bytes):
+ args.output.write(validation_graph.decode("utf-8"))
+ elif isinstance(validation_graph, str):
+ args.output.write(validation_graph)
+ else:
+ raise NotImplementedError("Unexpected result type returned from validate: %r." % type(validation_graph))
+
+ sys.exit(0 if conforms else 1)
+
+if __name__ == "__main__":
+ main()