- CLARIN-FCS Core 2.0 query language grammar and parser
- based on Github: clarin-eric/fcs-ql and Github: clarin-eric/fcs-simple-endpoint
- for more details visit: CLARIN FCS Technical Details
Install from PyPI:
python3 -m pip install fcs-ql-parserOr install from source:
git clone https://github.com/Querela/fcs-ql-python.git
cd fcs-ql-python
uv build
# built package
python3 -m pip install dist/fcs_ql_parser-<version>-py3-none-any.whl
# or
python3 -m pip install dist/fcs_ql_parser-<version>.tar.gz
# for local development
python3 -m pip install -e .The high-level interface fcsql.parser.QueryParser wraps the ANTLR4 parse tree into a simplified query node tree that is easier to work with. The fcsql-parser exposes a simple parsing function with fcsql.parse(input: str, enableSourceLocations: bool = True) -> fcsql.parser.QueryNode:
importfcsql## parsing a valid query into a query node tree# our query input stringinput='[ pos = "NOUN" ]'# parse into QueryNode treesc=fcsql.parse(input)
# print stringified treeprint(str(sc))
## handling possibly invalid queriesinput="[ kaputt ]"try:
fcsql.parse(input)
exceptfcsql.QueryParserExceptionasex:
print(f"Error: {ex}")You can also use the more low-level ANTLR4 framework to parse the query string. A handy wrapper is provided with fcsql.antlr_parse(input: str) -> LexParser.QueryContext.
fromantlr4importCommonTokenStream, InputStreamfromfcsql.parserimportFCSLexer, FCSParserinput='"test"'input_stream=InputStream(input)
lexer=FCSLexer(input_stream)
stream=CommonTokenStream(lexer)
parser=FCSParser(stream)
tree: FCSParser.QueryContext=parser.query()Parsed queries can also be checked against their specification conformance.
fromfcsqlimportQueryParserfromfcsql.validationimportFCSQLValidator, SpecificationValidationErrorparser=QueryParser(enableSourceLocations=True)
query='"Banane"'node=parser.parse(query)
validator=FCSQLValidator()
validator.validate(node, query=query)
len(validator.errors) ==0# no errors# or to raise an error on first violationquery='[ post = "NOUN" ]'node=parser.parse(query)
validator=FCSQLValidator(raise_at_first_violation=True)
validator.validate(node, query=query) # raises SpecificationValidationErrorA convenience method is provded with fcsql.validate(query: str):
fromfcsqlimportvalidate# simple boolean returnsvalidate("'apples'") # => Truevalidate("apples") # => False (parse error, invalid construct, not a simple string or token)validate('[ pos = "NOUNT" ]{3,0}') # => False (repetition max must be >= min)# or with list of errorserror=validate("pos = NOUN", return_errors=True)[0] # has one errorerror.message# "mismatched input 'pos' expecting {'(', '[', REGEXP}"error.type# "syntax-error"error.fragment# "pos"error.position.start# 0 (start offset in query string)error.position.stop# 3 ( end offset in query string)Fetch (or update) grammar files:
git clone https://github.com/clarin-eric/fcs-ql.git
cp fcs-ql/src/main/antlr4/eu/clarin/sru/fcs/qlparser/fcs/*.g4 src/fcsql/(Re-)Generate python parser code:
# setup environment
uv sync --extra antlr
# NOTE: you can activate the environment (if you do not want to prefix everything with `uv run`)# NOTE: `uv` does not play nicely with `pyenv` - if you use `pyenv`, sourcing does NOT work!source .venv/bin/activate
cd src/fcsql
uv run antlr4 -Dlanguage=Python3 *.g4 -listener -visitorRun style checks:
# setup environment
uv sync --extra style
uv run isort --check --diff .
uv run black --check .
uv run flake8 . --show-source --statistics
uv run mypy srcRun tests (pytest with coverage, clarity and randomly plugins):
# setup environment
uv sync --extra test
uv run pytest
# to see output and run a specific test file
uv run pytest -v -rP tests/validation/test_validation.py
# with logs
uv run pytest -v -rP -o log_cli=true -o log_cli_level="DEBUG"Build documentation:
# setup environment
uv sync --extra docs
# or if standalone
python3 -m pip install -r ./docs/requirements.txt
# build documentation and check links ...
uv run sphinx-build -b html docs dist/docs
uv run sphinx-build -b linkcheck docs dist/docsRun check before publishing:
# setup environment
uv sync --extra build
# build the package
uv build
# run metadata check# uv run python3 -m build
uv run twine check --strict dist/*# (manual) check of package contents
tar tvf dist/fcs_ql_parser-*.tar.gz