diff --git a/docs/assets/quality_checks/actions_logs_navigation.png b/docs/assets/quality_checks/actions_logs_navigation.png new file mode 100644 index 000000000..47eef66cc Binary files /dev/null and b/docs/assets/quality_checks/actions_logs_navigation.png differ diff --git a/docs/assets/quality_checks/actions_workflow_summary.png b/docs/assets/quality_checks/actions_workflow_summary.png new file mode 100644 index 000000000..cf2e39367 Binary files /dev/null and b/docs/assets/quality_checks/actions_workflow_summary.png differ diff --git a/docs/assets/quality_checks/dayamlchecker_cli_preview.png b/docs/assets/quality_checks/dayamlchecker_cli_preview.png new file mode 100644 index 000000000..4d101cc1a Binary files /dev/null and b/docs/assets/quality_checks/dayamlchecker_cli_preview.png differ diff --git a/docs/assets/quality_checks/word_diff_preview.png b/docs/assets/quality_checks/word_diff_preview.png new file mode 100644 index 000000000..5625fc2a0 Binary files /dev/null and b/docs/assets/quality_checks/word_diff_preview.png differ diff --git a/docs/assets/quality_checks/word_diff_run_summary.png b/docs/assets/quality_checks/word_diff_run_summary.png new file mode 100644 index 000000000..981b02561 Binary files /dev/null and b/docs/assets/quality_checks/word_diff_run_summary.png differ diff --git a/docs/authoring/docx_templates.md b/docs/authoring/docx_templates.md index 910911d25..2030d3012 100644 --- a/docs/authoring/docx_templates.md +++ b/docs/authoring/docx_templates.md @@ -493,8 +493,9 @@ manually. ## Quality control checklist -- [ ] The DOCX template has valid Jinja syntax. You can use the [ALDashboard](https://github.com/SuffolkLITLab/docassemble-ALDashboard)'s "Validate DOCX template" widget, - installed on your own docassemble server, to check. +- [ ] The DOCX template has valid Jinja syntax. You can validate templates locally with [DAYamlChecker](../automated_quality_checks/dayamlchecker.md), in your browser with the [ALDashboard](../components/ALDashboard/validate_docx.md)'s "Validate DOCX template" widget, or automatically in GitHub pull requests using [ALActions `valid_jinja2`](../automated_quality_checks/github_actions.md#valid_jinja2). +- [ ] Template visual diffs and text changes have been reviewed in pull requests using [ALActions `word_diff`](../automated_quality_checks/github_actions.md#word_diff). +- [ ] Document accessibility has been checked (headings structure, image alt text, table headers, no floating text boxes) using [DAYamlChecker's DOCX accessibility checks](../automated_quality_checks/dayamlchecker.md#4-docx-template-accessibility-accessibility). - [ ] All of the expected standard field names are recognized by the Weaver (marked in bold when you upload and view the quality control screen) - [ ] Custom field names are all properly spelled, without typos - [ ] Run the interview with short, long, and one and many items for repeated lists to ensure that the full range of common scenarios looks proper. diff --git a/docs/authoring/github.md b/docs/authoring/github.md index a069392f8..272ab43a5 100644 --- a/docs/authoring/github.md +++ b/docs/authoring/github.md @@ -276,4 +276,14 @@ If you think your GitHub account does have permission, try [redoing the GitHub i #### If you see `CONFLICT (content): Merge conflict` -The text `CONFLICT (content): Merge conflict` means the branch you are trying to push to changed while you were making your changes. It is what a [merge conflict](#resolve-conflicts) looks like in docassemble in git. Simply make a new branch and commit the changes there. Work out the merge conflicts afterwards. \ No newline at end of file +The text `CONFLICT (content): Merge conflict` means the branch you are trying to push to changed while you were making your changes. It is what a [merge conflict](#resolve-conflicts) looks like in docassemble in git. Simply make a new branch and commit the changes there. Work out the merge conflicts afterwards. + +## Automated quality checks in GitHub + +Once your package is on GitHub, add the +**[Assembly Line GitHub Actions](../automated_quality_checks/github_actions.md)** to check +every pull request: the package build, interview YAML, Word and PDF template +accessibility, broken links, and Python style and tests. Start with the +**[automated quality checks overview](../automated_quality_checks/overview.md)**, or with +**[running the same checks locally](../automated_quality_checks/running_checks_locally.md)** +before you push. diff --git a/docs/authoring/pdf_templates.md b/docs/authoring/pdf_templates.md index eab2db9da..ce217f18d 100644 --- a/docs/authoring/pdf_templates.md +++ b/docs/authoring/pdf_templates.md @@ -151,3 +151,4 @@ allowing for a quick visual inspection. - [ ] Checkboxes all use a consistent fill-in style. We recommend using the cross (`X`) to fill in the checkbox (under the Checkbox Properties | Options dialog in Adobe Acrobat). - [ ] Checkboxes use the export value "Yes" (less important, but can resolve some rare issues) - [ ] Signature fields use the `digital signature` field type in Adobe Acrobat or Gavel, not a standard text field. +- [ ] PDF document is tagged for accessibility (PDF/UA-1 standard). You can audit this automatically in CI via [ALActions `da_build` with veraPDF](../automated_quality_checks/github_actions.md#da_build). diff --git a/docs/automated_quality_checks/dayamlchecker.md b/docs/automated_quality_checks/dayamlchecker.md new file mode 100644 index 000000000..f38dbea8f --- /dev/null +++ b/docs/automated_quality_checks/dayamlchecker.md @@ -0,0 +1,317 @@ +--- +id: dayamlchecker +title: "DAYamlChecker: static analysis and linting" +sidebar_label: DAYamlChecker +slug: dayamlchecker +--- + +# DAYamlChecker: static analysis and linting + +`dayamlchecker` is a static analysis tool for Docassemble packages. It reads interview +YAML, embedded Python and Mako, Python test modules, and Word (`.docx`) templates, and +reports broken interview logic, syntax errors, broken URLs, style problems, and Web +Content Accessibility Guidelines (WCAG) failures. + +![A terminal running dayamlchecker over a questions directory, listing findings with codes EA502, EA509, WA518, EG414, and WG123](../assets/quality_checks/dayamlchecker_cli_preview.png) + +--- + +## Installation + +`dayamlchecker` requires Python 3.12 or later. + +```bash +# With pip +pip install dayamlchecker + +# Or as a standalone tool with uv +uv tool install dayamlchecker +``` + +To work on `dayamlchecker` itself: + +```bash +git clone https://github.com/SuffolkLITLab/DAYamlChecker.git +cd DAYamlChecker +pip install -e . +``` + +Installing provides two commands, `dayamlchecker` and `dayamlchecker-fmt` (a formatter +for interview YAML). The examples below use `python3 -m dayamlchecker`, which is +equivalent to running `dayamlchecker` and works even when the script directory is not on +your `PATH`. + +--- + +## Running DAYamlChecker + +Pass files or directories. Directories are searched recursively, skipping `.git*`, +`.github*`, `build`, `dist`, `node_modules`, and `sources` unless you pass `--check-all`. + +```bash +# One interview +python3 -m dayamlchecker docassemble/MyPackage/data/questions/interview.yml + +# Every interview in a package +python3 -m dayamlchecker docassemble/MyPackage/data/questions/ + +# Word templates +python3 -m dayamlchecker docassemble/MyPackage/data/templates/ + +# Both at once +python3 -m dayamlchecker docassemble/MyPackage/data/ +``` + +--- + +## What DAYamlChecker checks + +Findings belong to one of four classes: `general`, `accessibility`, `style`, and +`translatability`. Each finding has a code whose first letter is its severity (`E` error, +`W` warning, `I` info) and whose second letter is its class (`G`, `A`, `S`, `T`). + +### 1. YAML structure and Docassemble integrity (`general`) + +| Check area | Description | Example codes | +| :--- | :--- | :--- | +| YAML syntax | Unclosed quotes, indentation mistakes, invalid characters | `EG102` (YAML parse error) | +| Duplicate keys | A key repeated inside one block | `EG101` | +| Duplicate block ids | An `id:` reused across blocks, where Docassemble silently keeps the last one | `EG104` | +| Docassemble keys | Unrecognized keys in a block | `EG301` | +| Question ids | A `question` block with no `id:` | `EG414` | +| Mandatory logic | More than one unconditioned `mandatory: True` block | `WG415` | +| Field definitions | Empty `fields:`, fields that collect no input, empty variable targets | `EG418`, `EG420`, `EG421` | +| Test modules | A Python test module that does not start with `# do not pre-load`, which makes Docassemble import it during server startup | `EG105` | + +### 2. Embedded Python, Mako, and JavaScript (`general`) + +- **Python code blocks**: syntax errors (`EG122`), and a warning when a code block defines + a function that belongs in a Python module instead (`WG123`). +- **Mako**: `${ ... }` expressions and `% if ...` control lines inside questions, + subquestions, and field values (`EG111`, `EG112`), plus malformed Markdown links in + Mako-rendered text (`WG113`). +- **JavaScript**: `datatype: js` expressions, including `.val()` calls and references to + fields that are not on the screen (`EG204`–`EG207`). + +### 3. WCAG and web accessibility (`accessibility`) + +Accessibility checks run by default. Disable them with `--no-wcag`. + +- **Heading structure**: skipped Markdown heading levels such as `##` to `####` + (`EA506`), and the same in HTML (`EA507`). +- **Alt text**: Markdown images (`EA505`), HTML `` tags, and Docassemble + `[FILE ...]` tags. Write `[FILE logo.png, alt="Suffolk LIT Lab logo"]`. +- **Link text**: empty links (`EA508`), non-descriptive text such as "click here", + "here", "read more", and Spanish equivalents like "haga clic aquí" (`EA509`), the same + link text pointing at different destinations (`WA518`), and links that open a new tab + without telling the user (`WA519`). +- **Field labels**: `no label` and empty labels on screens with two or more fields + (`EA502`), non-descriptive labels (`WA512`), and duplicate labels on one screen + (`WA515`). +- **Comboboxes**: off by default while the rule is evaluated. Turn it on with + `--accessibility-error-on-widget combobox` (`EA501`). +- **Custom theme contrast**: when a theme is loaded with `features: bootstrap theme:`, + the CSS is parsed and body text, navbar links, dropdown items, and buttons are checked + against a 4.5:1 minimum ratio (`EA504`). +- **Display templates**: templates shown with `display_template()` need a `subject` + (`WA529`). +- **PDF tagging**: an informational note when an attachment does not set + `tagged pdf: True` (`IA503`). + +WCAG checks still report YAML parse errors, so one run surfaces both broken YAML and +accessibility failures. This is source analysis: it cannot see problems that only appear +in a rendered page or in JavaScript at runtime. + +### 4. DOCX template accessibility (`accessibility`) + +Any `.docx` file you pass is opened and its OpenXML structure inspected. These checks run +by default; `--no-docx-accessibility` skips them. + +- **Alt text**: images (`WA541`), shapes and other objects (`WA542`), placeholder alt + text (`WA548`), and alt text on images marked decorative (`WA547`). +- **Links**: empty link text (`WA543`), a raw URL used as link text (`WA549`), and + ambiguous link text (`WA550`). +- **Document metadata**: missing document language (`WA545`) and missing document title + (`IA561`). +- **Headings**: documents with no headings (`WA556`), empty headings (`WA555`), a first + heading below level 1 (`WA557`), and skipped levels (`WA558`). +- **Tables**: no header row marker (`WA552`), merged cells (`WA546`), and tables that + look like they are being used for layout (`WA551`). +- **Reading order**: floating objects (`WA553`) and text boxes (`WA554`) that pull text + out of the main flow. +- **Spacing**: long runs of empty paragraphs used instead of paragraph spacing + (`IA565`) and manually typed list numbering (`IA566`). + +Because a DOCX has no line numbers, each finding names the package part it came from +(`word/document.xml`, `word/header1.xml`) and quotes up to 80 characters of nearby text +so you can search for it in Word: + +```text +WARN [WA552] docassemble/MyPackage/data/templates/discovery.docx + a table in word/document.xml has no obvious header row marker + (table begins "Certificate of Service") +``` + +:::tip DOCX findings are warnings by default +Every DOCX finding is capped at **warning** severity, so turning these checks on reports +problems without failing a build. Most existing templates have findings today, and the +intent is for authors to work through them over time. Pass +`--docx-accessibility-severity error` to restore each rule's own severity and fail on +errors. +::: + +#### Why DOCX is checked here and PDF is checked with veraPDF + +`dayamlchecker` checks DOCX templates directly, while PDF templates are validated by +**veraPDF** inside the [`da_build`](./github_actions.md#da_build) action. A DOCX is a zip +of XML that pure Python can read, so the check is fast and runs anywhere. Validating +PDF/UA-1 means implementing a large ISO specification, and veraPDF is the reference +implementation — but it is a Java application, and bundling a JRE into a Python linter +would be a heavy price for local use. So `da_build` installs veraPDF in CI, where a Java +runtime is already available. + +### 5. Broken URLs + +Absolute HTTP and HTTPS links in interview files, and in the `data/templates` files +those interviews sit alongside, are requested concurrently. A broken link in a question +file is an error; a broken link in a template is a warning by default. + +```bash +# URL checking is on by default +python3 -m dayamlchecker docassemble/MyPackage/data/questions/ + +# Skip endpoints that block CI or are known to be flaky +python3 -m dayamlchecker --url-check-ignore-urls "https://flaky-court.gov,https://status.example.org" docassemble/MyPackage/data/questions/ + +# Turn it off, for example when working offline +python3 -m dayamlchecker --no-url-check docassemble/MyPackage/data/questions/ +``` + +### 6. Style and translatability (`style`, `translatability`) + +Assembly Line style checks are opt in. + +```bash +# Deterministic style and translatability rules +python3 -m dayamlchecker --style docassemble/MyPackage/data/questions/ + +# Add rules backed by an OpenAI-compatible model +OPENAI_API_KEY="sk-..." python3 -m dayamlchecker --style-llm docassemble/MyPackage/data/questions/ +``` + +`--style-llm` implies `--style`, and reads `OPENAI_BASE_URL`, `OPENAI_API_KEY`, and +`OPENAI_MODEL` from the environment when the matching flags are not given. + +Style rules cover plain language, screen titles, overlong labels, too many fields on one +screen, review screens, and inclusive gender and pronoun fields. They also include the +rule that `#` and `

` belong to the `question` itself and should not appear in a +`subquestion` (`ES701`). + +Translatability rules use `WT` codes: choice lists whose labels are translated but whose +stored values are not invariant (`WT701`), user-facing strings written inside Python code +(`WT702`), and conditional expressions or Mako blocks that change only part of a sentence +and so cannot be translated as a unit (`WT703`, `WT704`). + +--- + +## Suppressing findings + +Suppress by code (`EA509`) or by finding class (`accessibility`, `style`, +`translatability`, `general`). `ALL` or `*` suppresses everything at that location. + +### On one line + +```yaml +subquestion: | + Please [click here](https://example.com) for details. # no-dayc: EA509 +``` + +### For a whole block + +Put `# no-dayc-block:` anywhere inside the block, usually just after the `---` separator: + +```yaml +--- +# no-dayc-block: accessibility, EG301 +question: Custom widget +fields: + - no label: custom_widget_data +``` + +### For a whole run + +```bash +python3 -m dayamlchecker --suppress accessibility,EG101 docassemble/MyPackage/ +``` + +:::note DOCX findings can only be suppressed on the command line +`# no-dayc:` and `# no-dayc-block:` comments live in YAML, and a Word document has +nowhere to put them. Silence a noisy DOCX rule with `--suppress` instead, for example +`--suppress IA561` for the missing document title. +::: + +--- + +## Command line reference + +| Flag | Description | Default | +| :--- | :--- | :--- | +| `files` | One or more YAML, Python, or DOCX files or directories | Required | +| `--suppress CODE` | Suppress a code or finding class; repeatable and comma-separated | none | +| `--check-all` | Search ignored directories too (`.git*`, `.github*`, `build`, `dist`, `node_modules`, `sources`) | off | +| `--no-wcag` | Turn off WCAG accessibility checks | WCAG on | +| `--accessibility-error-on-widget WIDGET` | Treat a widget, such as `combobox`, as an error | none | +| `--style` | Turn on Assembly Line style and translatability rules | off | +| `--style-llm` | Add LLM-backed style rules; implies `--style` | off | +| `--openai-base-url URL` | Base URL for an OpenAI-compatible API | `OPENAI_BASE_URL` | +| `--openai-api-key KEY` | API key for `--style-llm` | `OPENAI_API_KEY` | +| `--openai-model MODEL` | Model name for `--style-llm` | `OPENAI_MODEL`, else `gpt-4o-mini` | +| `--url-check` / `--no-url-check` | Check that URLs resolve | on | +| `--url-check-timeout SECONDS` | Per-request HTTP timeout | `10` | +| `--url-check-root PATH` | Repository root used to find related template files | inferred | +| `--url-check-ignore-urls URLS` | Comma- or newline-separated URLs to skip | none | +| `--url-check-skip-templates` | Do not check URLs inside `data/templates` files | off | +| `--template-url-severity CHOICE` | Severity for broken template URLs: `error`, `warning`, `ignore` | `warning` | +| `--unreachable-url-severity CHOICE` | Severity for URLs that could not be reached at all | `warning` | +| `--docx-accessibility` / `--no-docx-accessibility` | Check DOCX templates | on | +| `--docx-accessibility-severity CHOICE` | Severity cap for DOCX findings: `warning` or `error` | `warning` | +| `--format CHOICE` | `text`, or `github` for GitHub Actions annotations | `text` | +| `--max-warnings N` | Fail if more than N warnings are reported | no limit | + +--- + +## Python API + +```python +from dayamlchecker import ( + RuntimeOptions, + find_errors_from_string, + find_style_findings_from_string, +) + +yaml_code = """ +id: user_income +question: What is your income? +fields: + - Income: user_income + datatype: currency +""" + +for finding in find_errors_from_string(yaml_code, input_file="interview.yml"): + print(f"[{finding.code}] {finding.message} (line {finding.line_number})") + +style_findings = find_style_findings_from_string( + yaml_code, + input_file="interview.yml", + runtime_options=RuntimeOptions(style_enabled=True), +) +``` + +--- + +## Related documentation + +- **[Running checks before you push](./running_checks_locally.md)** +- **[GitHub Actions](./github_actions.md)** +- **[Logs and artifacts](./navigating_logs_and_artifacts.md)** +- **[Making Docassemble interviews accessible](../coding_style/accessibility.md)** diff --git a/docs/automated_quality_checks/github_actions.md b/docs/automated_quality_checks/github_actions.md new file mode 100644 index 000000000..fd887e2b0 --- /dev/null +++ b/docs/automated_quality_checks/github_actions.md @@ -0,0 +1,484 @@ +--- +id: github_actions +title: "Assembly Line GitHub Actions (ALActions)" +sidebar_label: GitHub Actions +slug: github_actions +--- + +# Assembly Line GitHub Actions (ALActions) + +**[SuffolkLITLab/ALActions](https://github.com/SuffolkLITLab/ALActions)** holds the +reusable GitHub Actions the Document Assembly Line uses for Docassemble packages: package +builds, YAML and template checking, readable Word diffs, PDF accessibility validation +with veraPDF, Python linting and tests, playground deployments, and uptime monitoring. + +All of them are [composite +actions](https://docs.github.com/en/actions/creating-actions/creating-a-composite-action). +Reference them at `@main` so that fixes reach your repositories without a version bump: + +```yaml +- uses: SuffolkLITLab/ALActions/da_build@main +``` + +--- + +## Action catalog + +| Action | What it does | Typical triggers | What it produces | +| :--- | :--- | :--- | :--- | +| **[`da_build`](#da_build)** | Package build, Python compile, YAML checks, DOCX and PDF accessibility, URL checks | `push`, `pull_request` | Inline annotations, a PDF accessibility step summary | +| **[`valid_jinja2`](#valid_jinja2)** | Compiles Jinja2 expressions in changed `.docx` templates | `push`, `pull_request` on `.docx` paths | Step summary table, `jinja-validation` artifact | +| **[`word_diff`](#word_diff)** | Converts changed `.docx` templates to Markdown and HTML diffs | `pull_request`, `workflow_dispatch` | Diffs in the step summary, `word-doc-diff` artifact | +| **[`black-formatting`](#black-formatting)** | Runs Black over the repository | `push`, `pull_request` | A failing job when files need reformatting | +| **[`docsig`](#docsig)** | Checks Google-style docstrings against signatures | `push`, `pull_request` | Docstring mismatches in the job log | +| **[`pythontests`](#pythontests)** | Mypy, Bandit, and `pytest` | `push`, `pull_request` | Type, security, and test output | +| **[`da_playground_install`](#da_playground_install)** | Installs the branch into a playground project | `push` on feature branches | A live interview to click through | +| **[`da_package`](#da_package)** | Installs the package server-wide | `push` on `main` or a tag | A deployed package | +| **[`hall_monitor`](#hall_monitor)** | Checks that installed interviews still load | `schedule` | Email or Teams alerts on failure | + +--- + +## `da_build`: build and check the package {#da_build} + +`da_build` is the main check for a Docassemble package. It runs, in order: + +1. **Python compile**: `python -m compileall .`, so no Python file has a syntax error. +2. **Package build**: `uv build`, producing a wheel and an sdist in `dist/`. +3. **YAML and DOCX checks**: [`dayamlchecker`](./dayamlchecker.md) over the package's + interview files and Word templates, reporting findings as GitHub annotations. +4. **URL checks**: every absolute URL in question and template files is requested. A + broken link in a question file fails the job; one in a template is a warning. +5. **PDF accessibility**: veraPDF is installed and every PDF template is validated + against **PDF/UA-1** (ISO 14289-1). + +### Sample workflow + +Create `.github/workflows/build_and_check.yml`. `da_build` checks out the repository +itself, so you do not need an `actions/checkout` step: + +```yaml +name: Build and check package + +on: + push: + branches: [main] + pull_request: + branches: [main] + workflow_dispatch: + +jobs: + build-and-validate: + runs-on: ubuntu-latest + steps: + - uses: SuffolkLITLab/ALActions/da_build@main + with: + python-version: "3.12" + # Optional: skip endpoints that block CI or are known to be flaky + ignore-urls: | + https://example.com/known-flaky-endpoint + https://another.example.org/blocked-from-ci +``` + +### Inputs + +| Input | Description | Default | +| :--- | :--- | :--- | +| `python-version` | Python version used for the build environment | `"3.12"` | +| `skip-url-check` | `"true"` skips all URL network calls | `"false"` | +| `skip-templates` | `"true"` skips URLs found inside `data/templates` files | `"false"` | +| `ignore-urls` | Comma- or newline-separated URLs to ignore during URL checks | `""` | +| `docx-validation-mode` | `"warning"` annotates without failing, `"error"` fails on DOCX accessibility errors, `"off"` skips DOCX checks | `"warning"` | +| `pdf-validation-mode` | `"warning"` annotates without failing, `"error"` fails on PDF/UA-1 failures, `"off"` skips the check and the veraPDF install | `"warning"` | +| `pdf-strict` | `"true"` also enforces tab-order and annotation-structure rules on form fields | `"false"` | + +### Which files are checked + +`da_build` does not check every file in the repository. It matches: + +- `docassemble/*/data/questions/**/*.yml` for interview checks, so that workflow files + and ALKiln fixtures are not mistaken for interviews, and +- `docassemble/*/data/templates/**/*.docx` for document accessibility, so that only the + documents users actually receive are checked. Word lock files (`~$…`) are skipped. + +`build/`, `dist/`, and `.venv/` are pruned, because the `uv build` step above has already +copied the package into `build/lib`. Without pruning, every finding would be reported +twice and the annotations would point at the copy rather than at the file you edit. + +Note that templates are found by scanning the directory, not by following references from +a YAML file. A template that no interview mentions yet is still checked. + +### Adjusting what fails the build + +- **DOCX accessibility** is reported as warnings by default, because most existing + templates have findings and the intent is to work through them over time. Set + `docx-validation-mode: "error"` once a package is clean, or `"off"` to skip it. +- **PDF accessibility** is likewise `"warning"` by default. Set + `pdf-validation-mode: "error"` to block merges on PDF/UA-1 failures. Leave `pdf-strict` + at `"false"` unless your forms stay fillable for the user: tab-order and annotation + rules are suppressed by default because Assembly Line forms are usually flattened + before anyone sees them. +- **URLs**: set `skip-templates: "true"` to ignore links inside `data/templates`, or + `skip-url-check: "true"` to make no network calls at all, which is what you want on a + runner without outbound internet access. +- **Individual findings**: suppress them in the YAML itself with `# no-dayc:` and + `# no-dayc-block:` comments. DOCX findings have no YAML to annotate, so a noisy + document rule has to be silenced with `--suppress` when running + [`dayamlchecker` locally](./dayamlchecker.md#suppressing-findings), or turned off for + the whole job with `docx-validation-mode`. + +:::note URLs are checked twice +`dayamlchecker` checks URLs as part of its own run, and `da_build` then runs the URL +checker again as a separate step so that warnings can be surfaced as a job annotation. +Seeing the same link reported in two steps is expected. +::: + +To audit templates locally at full strictness before pushing: + +```bash +python3 -m dayamlchecker --docx-accessibility-severity error docassemble/MyPackage/data/templates/ +``` + +--- + +## `valid_jinja2`: check Jinja2 in Word templates {#valid_jinja2} + +Docassemble builds Word documents with `docxtpl`, which is Jinja2. A typo like +`{{ user.firs_name }}` or an unclosed `{% if %}` only fails when a user tries to download +the document. `valid_jinja2` compiles the templates in a pull request instead. + +- **Syntax errors** fail the job. +- **Unknown filters** are warnings, not failures. The action knows 124 Jinja2 and + Docassemble filters, including `currency`, `format_date`, `title_case`, `nice_number`, + `ordinal`, `word`, and `comma_and_list`. +- **Added and modified** `.docx` files are found with `git diff` against the pull + request base. Only committed changes are compared, so commit a fixed template before + expecting the result to change. +- **A Markdown summary** is always written to the step summary. The HTML artifact is + uploaded only when there is something to report. + +### Sample workflow + +```yaml +name: Validate DOCX templates + +on: + pull_request: + paths: ['**/*.docx'] + push: + paths: ['**/*.docx'] + workflow_dispatch: + +jobs: + validate-templates: + runs-on: ubuntu-latest + steps: + - uses: SuffolkLITLab/ALActions/valid_jinja2@main +``` + +### Inputs + +| Input | Description | Default | +| :--- | :--- | :--- | +| `base_ref` | Git ref to compare against | detected from the event | +| `working_directory` | Directory to run from | `.` | +| `artifact_name` | Name of the uploaded artifact | `jinja-validation` | +| `output_dir` | Directory for the HTML reports | `jinja_validation` | +| `summary_file` | Path for the Markdown summary | `jinja_validation_summary.md` | +| `skip_checkout` | `"true"` if the job already checked out the repository with `fetch-depth: 0` | `"false"` | + +--- + +## `word_diff`: readable diffs for Word templates {#word_diff} + +GitHub shows a changed `.docx` as a replaced binary file, which tells a reviewer nothing. +`word_diff` converts both versions to Markdown and diffs the text. + +It is not a pass/fail check. On any pull request that touches a Word document it produces +a report, whether or not anything is wrong; when no `.docx` changed between the two +commits it says so in the summary and stops. + +![A side-by-side HTML diff of two versions of a Word template, with removed text on the left and added text on the right](../assets/quality_checks/word_diff_preview.png) + +- **In the step summary**: a unified diff per changed file, so a reviewer can read the + change without downloading anything. +- **In the artifact**: side-by-side HTML diffs plus the converted Markdown, with an + `index.html` table of contents. +- **Jinja2 is preserved**, so a change from `{{ user.name }}` to `{{ users[0].name }}` + shows up as a text change like any other. + +### Sample workflow + +```yaml +name: Diff Word documents + +on: + pull_request: + paths: ['**/*.docx'] + workflow_dispatch: + +jobs: + docx-diff: + runs-on: ubuntu-latest + steps: + - uses: SuffolkLITLab/ALActions/word_diff@main +``` + +### Inputs + +| Input | Description | Default | +| :--- | :--- | :--- | +| `base_ref` | Git ref or SHA to compare against | detected from the event | +| `working_directory` | Directory to run from | `.` | +| `artifact_name` | Name of the uploaded artifact | `word-doc-diff` | +| `output_dir` | Directory for the diff files and `index.html` | `word_diffs` | +| `summary_file` | Path for the Markdown summary | `word_diff_summary.md` | +| `skip_checkout` | `"true"` if the job already checked out the repository with `fetch-depth: 0` | `"false"` | + +For a manually dispatched run, pass `base_ref` explicitly; the automatic detection only +covers pull requests and pushes. + +--- + +## `black-formatting`: Python formatting {#black-formatting} + +Runs [Black](https://black.readthedocs.io/en/stable/) over the repository, excluding +`__init__.py` and `setup.py`. The job fails if any file would be reformatted. + +```yaml +jobs: + lint-python: + runs-on: ubuntu-latest + steps: + - uses: SuffolkLITLab/ALActions/black-formatting@main +``` + +Black also reads `pyproject.toml`. Most Assembly Line packages include: + +```toml +[tool.black] +extend-exclude = '(__init__.py|setup.py)' +``` + +:::caution Gate on the job, not the output +`black-formatting` and `pythontests` both declare an output (`linting-passed` and +`tests-passed`). Neither is currently populated with a meaningful value. Depend on +whether the job succeeded instead. +::: + +--- + +## `docsig`: docstrings that match their signatures {#docsig} + +`docsig` checks that every documented parameter exists, that every parameter is +documented, and that the style is consistent. Assembly Line packages use **Google-style** +docstrings. + +```yaml +jobs: + docstrings: + runs-on: ubuntu-latest + steps: + - uses: SuffolkLITLab/ALActions/docsig@main +``` + +The action checks every `.py` file except `test*.py`, `setup.py`, and `__init__.py`, and +disables the `description-not-capitalized` rule. Configure the rest through +`[tool.docsig]` in `pyproject.toml`; see the +[docsig README](https://github.com/jshwi/docsig#commandline). + +--- + +## `pythontests`: types, security, and tests {#pythontests} + +1. Installs the system libraries Docassemble packages tend to need, and sets + `ISUNITTEST=true`. +2. Installs dependencies with `uv sync --group dev` when `pyproject.toml` has a + `[dependency-groups]` table, and `uv sync` otherwise. +3. Runs `mypy . --exclude '^build/' --explicit-package-bases`. +4. Runs Bandit at high severity: + `uv tool run bandit -r . --exclude './scripts,./venv,./.venv,./build' --severity-level=high`. +5. Runs `pytest`. + +```yaml +jobs: + test-python: + runs-on: ubuntu-latest + steps: + - uses: SuffolkLITLab/ALActions/pythontests@main +``` + +Configure each tool through `pyproject.toml`: `[tool.pytest.ini_options]`, `[tool.mypy]`, +and `[tool.bandit]`. + +--- + +## `da_playground_install`: deploy to a playground {#da_playground_install} + +Installs the current branch into a project in a developer's Docassemble playground, so a +reviewer can click through the interview. This action installs whatever is in the working +directory, so the job **must** check out the repository first. + +```yaml +name: Deploy to playground + +on: + push: + branches: ['feature/**'] + +jobs: + deploy-playground: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: SuffolkLITLab/ALActions/da_playground_install@main + with: + SERVER_URL: ${{ secrets.SERVER_URL }} + DOCASSEMBLE_DEVELOPER_API_KEY: ${{ secrets.DOCASSEMBLE_DEVELOPER_API_KEY }} + PROJECT_NAME: "review-${{ github.ref_name }}" +``` + +| Input | Description | Default | Required | +| :--- | :--- | :--- | :--- | +| `SERVER_URL` | Server URL, without a trailing slash, such as `https://apps-dev.example.org` | none | Yes | +| `DOCASSEMBLE_DEVELOPER_API_KEY` | API key for an account that may install packages | none | Yes | +| `PROJECT_NAME` | Playground project to install into | none | Yes | +| `USER_ID` | Numeric user id | the account the API key belongs to | No | +| `RESTART` | Set to `0` to skip the server restart | restart | No | + +--- + +## `da_package`: install server-wide {#da_package} + +Installs the package for every user on a server. It picks its source in this order: + +1. `PYPI_PACKAGE`, if set. +2. `GITHUB_URL` (with `GITHUB_BRANCH`, defaulting to the repository's default branch), + if set. This is usually what you want, because the server can then be updated later + with its own **update** button. +3. Otherwise, a zip of the working directory — which means the job must run + `actions/checkout` first. + +```yaml +name: Deploy package server-wide + +on: + push: + branches: [main] + +jobs: + deploy-package: + runs-on: ubuntu-latest + steps: + - uses: SuffolkLITLab/ALActions/da_package@main + with: + SERVER_URL: ${{ secrets.PROD_SERVER_URL }} + DOCASSEMBLE_DEVELOPER_API_KEY: ${{ secrets.DOCASSEMBLE_DEVELOPER_API_KEY }} + GITHUB_URL: "https://github.com/${{ github.repository }}" + GITHUB_BRANCH: "main" +``` + +| Input | Description | Default | Required | +| :--- | :--- | :--- | :--- | +| `SERVER_URL` | Server URL, without a trailing slash | none | Yes | +| `DOCASSEMBLE_DEVELOPER_API_KEY` | API key for an account that may install packages server-wide | none | Yes | +| `GITHUB_URL` | GitHub URL of the package to install | none | No | +| `GITHUB_BRANCH` | Branch to install from | the repository's default branch | No | +| `PYPI_PACKAGE` | PyPI package name to install instead | none | No | + +--- + +## `hall_monitor`: scheduled checks on a live server {#hall_monitor} + +Docassemble's `/list` page marks any installed interview that failed to load. +`hall_monitor` fetches that page on a schedule and fails the job if any interview is +marked broken — like a hall monitor looking through doorways, without going in. Set +`CHECK_TYPE: "homepage"` to check only that `/` responds. + +When the check fails, the action sends alerts through SendGrid, Mailgun, or a Microsoft +Teams webhook. This is separate from GitHub's own notifications, which cannot email +people who are not watching the repository. + +```yaml +name: Hall monitor + +on: + schedule: + # 07:00 and 19:00 UTC + - cron: "0 7,19 * * *" + workflow_dispatch: + +jobs: + monitor-server: + runs-on: ubuntu-latest + steps: + - uses: SuffolkLITLab/ALActions/hall_monitor@main + with: + SERVER_URL: "https://apps.example.org" + SENDGRID_API_KEY: ${{ secrets.SENDGRID_API_KEY }} + ERROR_EMAIL_FROM: "Monitor " + ERROR_EMAILS: "dev-team@example.org,admin@example.org" +``` + +| Input | Description | Default | Required | +| :--- | :--- | :--- | :--- | +| `SERVER_URL` | Server to check, with or without a trailing slash | none | Yes | +| `CHECK_TYPE` | `"homepage"` to check only `/`; anything else checks `/list` | check `/list` | No | +| `SENDGRID_API_KEY` | SendGrid key for failure emails | none | No | +| `MAILGUN_API_KEY` | Mailgun key for failure emails | none | No | +| `MAILGUN_DOMAIN` | Mailgun sending domain | none | No | +| `ERROR_EMAIL_FROM` | Address the alert is sent from | none | No | +| `ERROR_EMAILS` | Comma-separated recipients | none | No | +| `TEAMS_MONITOR_WEBHOOK` | Microsoft Teams incoming webhook | none | No | + +--- + +## A complete workflow + +`.github/workflows/quality_checks.yml`, combining the checks that run on every change. +`da_build`, `valid_jinja2`, and `word_diff` each check out the repository themselves. + +```yaml +name: Quality checks + +on: + push: + branches: [main] + pull_request: + branches: [main] + workflow_dispatch: + +jobs: + package-build-and-lint: + runs-on: ubuntu-latest + steps: + - uses: SuffolkLITLab/ALActions/da_build@main + with: + python-version: "3.12" + + validate-docx-templates: + runs-on: ubuntu-latest + steps: + - uses: SuffolkLITLab/ALActions/valid_jinja2@main + + diff-word-documents: + if: github.event_name == 'pull_request' + runs-on: ubuntu-latest + steps: + - uses: SuffolkLITLab/ALActions/word_diff@main + + python-quality: + runs-on: ubuntu-latest + steps: + - uses: SuffolkLITLab/ALActions/black-formatting@main + - uses: SuffolkLITLab/ALActions/docsig@main + - uses: SuffolkLITLab/ALActions/pythontests@main +``` + +--- + +## Related documentation + +- **[Logs and artifacts](./navigating_logs_and_artifacts.md)** +- **[DAYamlChecker](./dayamlchecker.md)** +- **[Automated testing with ALKiln](../components/ALKiln/intro.mdx)** diff --git a/docs/automated_quality_checks/navigating_logs_and_artifacts.md b/docs/automated_quality_checks/navigating_logs_and_artifacts.md new file mode 100644 index 000000000..5c0a45046 --- /dev/null +++ b/docs/automated_quality_checks/navigating_logs_and_artifacts.md @@ -0,0 +1,133 @@ +--- +id: navigating_logs_and_artifacts +title: "Navigating logs, summaries, and artifacts" +sidebar_label: Logs & artifacts +slug: logs-and-artifacts +--- + +# Navigating logs, summaries, and artifacts + +When a check reports something, the result can land in one of four places: an inline +annotation on the **Files changed** tab, an annotation box on the run's **Summary** page, +a Markdown step summary, or a downloadable artifact. This page explains which is which +and how to get to each one. + +--- + +## Finding the workflow run + +From a pull request, open the **Checks** tab, or scroll to the merge box at the bottom of +the **Conversation** tab and click **Details** next to a check. From the repository, +open the **Actions** tab and click the run. + +--- + +## Reading the run summary + +![The summary page of a da_build workflow run, showing a successful build job and an annotations panel listing URL checker warnings](../assets/quality_checks/actions_workflow_summary.png) + +- **Annotations**, near the top, collect everything the run reported as a workflow + command: `dayamlchecker` findings, the URL checker's warning block, and veraPDF + failures. +- **Step summaries** are rendered below: the Jinja2 validation table from + `valid_jinja2`, the per-file diffs from `word_diff`, and the PDF accessibility section + from `da_build`. +- **Artifacts**, at the very bottom, are the downloadable `.zip` bundles. + +:::note GitHub only shows the first ten annotations +GitHub keeps at most ten annotations per severity per run. `dayamlchecker` sorts its +findings so that the most severe ones survive that cut, and also prints the complete +report into the job log — so if the annotation list looks suspiciously short, the full +list is in the log for that step. +::: + +--- + +## Annotations on the Files changed tab + +`da_build` runs `dayamlchecker` with `--format github`, so findings become annotations +carrying a file, a line, and the diagnostic code as their title. A finding on a line you +changed appears inline on the **Files changed** tab, directly under the offending line. + +Two kinds of finding cannot be anchored to a line, and appear in the run's annotation box +instead: + +- **DOCX findings**, because a Word document has no line numbers. They name the package + part instead (`word/document.xml`) and quote nearby text. +- **URL checker warnings**, which are collected into a single annotation titled + `URL checker`. + +--- + +## Word diffs + +`word_diff` is not a pass/fail check. On any pull request that touches a `.docx` file it +produces a report, whether or not anything is wrong. + +![A side-by-side HTML diff of two versions of a Word template, with removed text on the left and added text on the right](../assets/quality_checks/word_diff_preview.png) + +The unified diff in the step summary is usually enough. For the side-by-side view: + +1. Open the run, as above, and click **Summary** in the left sidebar — GitHub often opens + the log viewer first. +2. Scroll to **Artifacts** at the bottom and click **`word-doc-diff`** (or whatever + `artifact_name` you configured). GitHub downloads a `.zip`. +3. Unzip it and open **`index.html`** in any browser, then pick a template from the list. + Red or struck-through text was removed; green text was added. + +![The artifacts section of a word_diff run summary, with the word-doc-diff bundle ready to download](../assets/quality_checks/word_diff_run_summary.png) + +:::tip Reviewing Jinja2 without Word +Because `word_diff` diffs the text including its Jinja2 tags, you can confirm a rename +from `{{ user.name }}` to `{{ users[0].name.full() }}` in the browser, with no copy of +Microsoft Word involved. +::: + +--- + +## Searching the job logs + +![The GitHub Actions log viewer with a job's steps expanded](../assets/quality_checks/actions_logs_navigation.png) + +Click a job in the left sidebar, then click a step header — `Run YAML and template +document checker`, `Check URLs in question/template files`, or `Check PDF accessibility +with veraPDF` — to expand it. The search box at the top right of the log pane searches +the whole job: + +- `ERROR` or `WARN` for severity, or `[EA` and `[WA` for accessibility findings + specifically. +- `[EG` for interview structure and syntax findings. +- `HTTP` for URL checker results. +- `veraPDF` for PDF/UA-1 rule failures. + +--- + +## Common failures + +**A YAML error (`EG102`, `EG101`)** — the `dayamlchecker` step fails on a parse error or +a duplicate key. The annotation names the line; fix the indentation, quoting, or +duplicate. + +**A skipped heading (`EA506`)** — a screen jumps from `##` to `####`. Use the next level +down instead. If you wanted smaller text rather than a lower level, keep the level and +style it: `

Heading text

`. + +**A broken URL (`EG602`)** — a link in a question file returned an error. Fix the link, +or, if the destination blocks CI or rate-limits it, add it to `ignore-urls` in your +workflow. Broken links in `data/templates` are warnings rather than failures. + +**A PDF/UA-1 failure** — a PDF template is not properly tagged. Open it in Acrobat Pro, +run the accessibility check, and fix the tagging. If the form is flattened before users +see it, leave `pdf-strict` at `"false"` so tab-order rules do not count against you. + +**A DOCX accessibility finding (`WA5…`)** — a warning by default, so it will not fail the +build. Work through them with the [DOCX accessibility +rules](./dayamlchecker.md#4-docx-template-accessibility-accessibility). + +--- + +## Related documentation + +- **[Automated quality checks overview](./overview.md)** +- **[DAYamlChecker](./dayamlchecker.md)** +- **[GitHub Actions](./github_actions.md)** diff --git a/docs/automated_quality_checks/overview.md b/docs/automated_quality_checks/overview.md new file mode 100644 index 000000000..0234940a4 --- /dev/null +++ b/docs/automated_quality_checks/overview.md @@ -0,0 +1,91 @@ +--- +id: automated_quality_checks_overview +title: Automated quality checks in Assembly Line +sidebar_label: Overview +slug: overview +--- + +# Automated quality checks in Assembly Line + +Docassemble interviews combine several languages in one package: YAML for interview +logic, Python for data models, Mako and Jinja2 for templating, Markdown and HTML for +screen formatting, and DOCX and PDF for the documents users receive. The Document +Assembly Line runs a set of automated checks over all of them, so that broken links, +template typos, and accessibility barriers are caught in a pull request instead of by a +self-represented litigant. + +```mermaid +flowchart LR + A["Local authoring
(VS Code / Playground)"] --> B["dayamlchecker
(local static checks)"] + B --> C["GitHub pull request"] + C --> D["da_build
(YAML, DOCX, PDF, URLs)"] + C --> E["valid_jinja2
(DOCX Jinja2 syntax)"] + C --> F["word_diff
(readable DOCX diffs)"] + C --> G["black, docsig, pythontests
(Python)"] + D & E & F & G --> H{"All checks pass?"} + H -->|Yes| I["Merge and deploy
(da_package / da_playground_install)"] + H -->|No| J["Review annotations and logs"] + I --> K["hall_monitor
(scheduled uptime checks)"] +``` + +--- + +## What the checks catch + +1. **Accessibility barriers**: skipped heading levels, unlabelled fields, missing image + alt text, low contrast in custom themes, and untagged PDFs. +2. **Template and syntax errors**: broken Jinja2 expressions (`{{ user.nam }}` instead of + `{{ user.name }}`), malformed Mako, and invalid YAML, before a user hits a runtime + exception. +3. **Broken links**: every absolute HTTP and HTTPS link in interview screens and + templates is requested, so users never land on a dead page. +4. **Document drift**: the exact text that changed in a Word template, readable in a pull + request without opening Microsoft Word. +5. **Python style and correctness**: Black formatting, docstrings that match their + signatures, type checks, a security scan, and unit tests. +6. **Server health**: scheduled checks that every installed interview on a live server + still loads. + +--- + +## The toolchain + +| Tool or action | Scope | What it does | How it runs | +| :--- | :--- | :--- | :--- | +| **[`dayamlchecker`](./dayamlchecker.md)** | YAML, Python, DOCX, URLs | Static checker for interview structure, WCAG failures, DOCX accessibility, and broken links | Locally from the command line, and inside `da_build` | +| **[`ALActions/da_build`](./github_actions.md#da_build)** | Package build, YAML, DOCX, PDF, URLs | Builds the package, runs `dayamlchecker` over interview YAML and DOCX templates, audits PDF templates with veraPDF, and checks URLs | GitHub Actions | +| **[`ALActions/valid_jinja2`](./github_actions.md#valid_jinja2)** | DOCX templates | Compiles the Jinja2 expressions in changed `.docx` files, recognizing 124 Docassemble and Jinja2 filters | GitHub Actions | +| **[`ALActions/word_diff`](./github_actions.md#word_diff)** | DOCX templates | Converts changed `.docx` files to Markdown and side-by-side HTML diffs | GitHub Actions | +| **[`ALActions/black-formatting`](./github_actions.md#black-formatting)** | Python | Enforces Black formatting | GitHub Actions | +| **[`ALActions/docsig`](./github_actions.md#docsig)** | Python docstrings | Checks that Google-style docstrings match function signatures | GitHub Actions | +| **[`ALActions/pythontests`](./github_actions.md#pythontests)** | Python | Runs Mypy, Bandit, and the `pytest` suite | GitHub Actions | +| **[`ALActions/da_playground_install`](./github_actions.md#da_playground_install)** | Deployment | Installs the branch into a Docassemble playground project for manual testing | GitHub Actions | +| **[`ALActions/da_package`](./github_actions.md#da_package)** | Deployment | Installs the package server-wide on a test or staging server | GitHub Actions | +| **[`ALActions/hall_monitor`](./github_actions.md#hall_monitor)** | Monitoring | Checks that installed interviews on a live server still load, and alerts by email or Teams | GitHub Actions, on a cron schedule | + +:::note Static and dynamic testing are complementary +These checks are **static**: they run in seconds in a lightweight container, reading +source code, templates, and documents without booting a Docassemble server. They cannot +tell you whether an interview actually works. For that, use +**[ALKiln](../components/ALKiln/intro.mdx)**, which drives a headless browser through a +real interview on a running server. +::: + +--- + +## What a run looks like + +![The summary page of a da_build workflow run, showing a successful build job and an annotations panel listing URL checker warnings](../assets/quality_checks/actions_workflow_summary.png) + +--- + +## Next steps + +- **[Running checks before you push](./running_checks_locally.md)**: command line checks + and Git pre-commit hooks. +- **[DAYamlChecker](./dayamlchecker.md)**: what it checks, what the diagnostic codes + mean, and how to suppress a finding. +- **[GitHub Actions](./github_actions.md)**: every action in + `SuffolkLITLab/ALActions`, with workflows you can copy. +- **[Logs and artifacts](./navigating_logs_and_artifacts.md)**: reading annotations, + step summaries, and downloadable reports. diff --git a/docs/automated_quality_checks/running_checks_locally.md b/docs/automated_quality_checks/running_checks_locally.md new file mode 100644 index 000000000..9f92ebf85 --- /dev/null +++ b/docs/automated_quality_checks/running_checks_locally.md @@ -0,0 +1,191 @@ +--- +id: running_checks_locally +title: Running checks before you push to GitHub +sidebar_label: Running checks locally +slug: running-checks-locally +--- + +# Running checks before you push to GitHub + +GitHub Actions checks every pull request, but a round trip through CI takes minutes and +leaves a trail of "fix lint" commits. The same checks run locally in seconds. + +--- + +## From the command line + +```bash +# Interview YAML: structure, WCAG, and broken links +python3 -m dayamlchecker docassemble/MyPackage/data/questions/ + +# Word templates: document accessibility +python3 -m dayamlchecker docassemble/MyPackage/data/templates/ + +# Python: syntax, formatting, docstrings, tests +python3 -m compileall . -q +black . +docsig docassemble/ +pytest +``` + +See the [DAYamlChecker page](./dayamlchecker.md) for what each finding means and how to +suppress one. + +:::tip Skip the network when you are iterating +`dayamlchecker` requests every external link it finds, which is the slowest thing it +does. Pass `--no-url-check` while you are working, and leave the link checking to CI. +::: + +--- + +## Automating with Git pre-commit hooks + +A pre-commit hook runs every time you `git commit`. If a check fails, the commit is +aborted, so a broken template never reaches your history in the first place. There are +two ways to set one up. + +## Option 1: the `pre-commit` framework (recommended) + +[pre-commit](https://pre-commit.com/) installs and runs hooks from many languages, and +only passes it the files you actually staged. + +### 1. Install it + +```bash +uv tool install pre-commit +# or +pip install pre-commit +``` + +### 2. Create `.pre-commit-config.yaml` + +In the root of your package repository: + +```yaml +repos: + # General file hygiene + - repo: https://github.com/pre-commit/pre-commit-hooks + rev: v4.6.0 + hooks: + - id: trailing-whitespace + - id: end-of-file-fixer + - id: check-yaml + # Docassemble interviews are multi-document YAML with Mako in them + exclude: ^docassemble/.*/data/questions/.*\.yml$ + + # Python formatting + - repo: https://github.com/psf/black + rev: 24.8.0 + hooks: + - id: black + + # Python docstrings + - repo: https://github.com/jshwi/docsig + rev: v0.96.0 + hooks: + - id: docsig + args: [--disable=description-not-capitalized] + exclude: (test_.*|setup)\.py$ + + # DAYamlChecker, from your own environment + - repo: local + hooks: + - id: dayamlchecker + name: DAYamlChecker + entry: dayamlchecker --no-url-check + language: system + files: ^docassemble/.*/data/(questions/.*\.ya?ml|templates/.*\.docx)$ + pass_filenames: true +``` + +`--no-url-check` keeps every commit from making live HTTP requests to every link in your +interview. CI still checks the links. + +### 3. Install the hook + +```bash +pre-commit install +``` + +```text +pre-commit installed at .git/hooks/pre-commit +``` + +### 4. What it looks like + +```bash +$ git add docassemble/MyPackage/data/questions/interview.yml +$ git commit -m "Add new interview screen" + +Trim Trailing Whitespace.................................................Passed +Fix End of Files.........................................................Passed +black................................................(no files to check)Skipped +docsig...............................................(no files to check)Skipped +DAYamlChecker............................................................Passed +[feature-branch 8a3f910] Add new interview screen + 1 file changed, 25 insertions(+) +``` + +To run every hook over the whole repository, without committing: + +```bash +pre-commit run --all-files +``` + +--- + +## Option 2: a plain shell hook + +If you would rather not add a dependency, write `.git/hooks/pre-commit` yourself: + +```bash +#!/bin/bash +# Check staged interview files before committing +set -e + +echo "Checking Python syntax..." +python3 -m compileall . -q + +STAGED=$(git diff --cached --name-only --diff-filter=ACM \ + | grep -E '^docassemble/.*/data/(questions/.*\.ya?ml|templates/.*\.docx)$' || true) + +if [ -n "$STAGED" ]; then + echo "Running DAYamlChecker on:" + echo "$STAGED" + # shellcheck disable=SC2086 + dayamlchecker --no-url-check $STAGED +fi + +echo "All pre-commit checks passed." +``` + +Then make it executable: + +```bash +chmod +x .git/hooks/pre-commit +``` + +Unlike the `pre-commit` framework, a hook in `.git/hooks/` is local to your own clone and +is not shared with the rest of the team. + +--- + +## Skipping the hooks + +```bash +git commit --no-verify -m "WIP: save checkpoint" +``` + +:::warning +`--no-verify` only skips the local checks. The same checks run again on your pull +request, so anything you skipped will come back. +::: + +--- + +## Related documentation + +- **[DAYamlChecker](./dayamlchecker.md)** +- **[GitHub Actions](./github_actions.md)** +- **[pre-commit documentation](https://pre-commit.com/)** and + **[Git hooks in the Git book](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks)** diff --git a/docs/coding_style/accessibility.md b/docs/coding_style/accessibility.md index 6226f90fb..5cd650fb5 100644 --- a/docs/coding_style/accessibility.md +++ b/docs/coding_style/accessibility.md @@ -13,24 +13,26 @@ Interview builders must address accessibility, too, so the Document Assembly Lin ## AssemblyLine accessibility tools +The Document Assembly Line provides multiple automated tools to check accessibility during development and continuous integration: -The AssemblyLine software includes a [code linter](https://en.wikipedia.org/wiki/Lint_(software)) used in the [ALDashboard](../components/ALDashboard/overview) and [ALWeaver](../authoring/weaver/weaver_overview) packages. It can check interviews against the Document Assembly Line style guide, interface guidelines, and [WCAG accessibility guidelines](https://www.w3.org/WAI/standards-guidelines/wcag/). (WCAG accessibility checks must be run manually from the Dashboard **Interview style check (lint)** link.) +- **[DAYamlChecker](../automated_quality_checks/dayamlchecker.md)**: A command-line static analysis tool that checks Docassemble YAML interviews, Python code, and DOCX templates for WCAG violations, skipped headings, missing alt text, non-descriptive links, unlabelled fields, low contrast in custom themes, and template accessibility issues. +- **[ALActions `da_build`](../automated_quality_checks/github_actions.md#da_build)**: Runs `dayamlchecker` over interview YAML and Word templates on every pull request, and audits PDF templates against **PDF/UA-1** with **veraPDF**. +- **[ALDashboard & ALWeaver](../components/ALDashboard/aldashboard_overview.md)**: The built-in interview style check (lint) in the ALDashboard provides in-browser linting for interview authors.

ALDashboard icon for the code linter. The text reads "Interview style check (lint)"

-Accessibility checks include: +Automated accessibility checks include: -- Web Content Accessibility Guidelines (WCAG) clear failures -- DOCX and PDF template accessibility -- DAL style guide issues -- Low contrast +- Web Content Accessibility Guidelines (WCAG 2.1/2.2) clear static failures +- DOCX and PDF template accessibility ([veraPDF PDF/UA-1 checks](../automated_quality_checks/github_actions.md#da_build) and [DOCX accessibility rules](../automated_quality_checks/dayamlchecker.md#4-docx-template-accessibility-accessibility)) +- DAL style guide and plain language issues +- Low color contrast in custom themes - Skipped [heading levels](#heading-levels) -- Empty links -- Non-descriptive link text -- Missing [alt text](#alt-text) -- Missing field labels -- [Combobox](#avoid-comboboxes) use -- Translateability +- Empty links and non-descriptive link text +- Missing [alt text](#alt-text) on images (`[FILE ...]`, Markdown, and HTML ``) +- Missing field labels on multi-field screens +- [Combobox](#avoid-comboboxes) usage detection +- Translatability & sentence fragmentation ## General tips for improving interview accessibility @@ -214,8 +216,10 @@ These tools can automatically scan your interview pages for accessibility issues - **[TalkBack](https://support.google.com/accessibility/android/answer/6283677)** is Android's built-in screen reader - **[Switch Access](https://support.google.com/accessibility/android/answer/6122836)** is an Android feature for users with motor disabilities -### Automated testing integration +### Automated testing and CI integration -- **[ALKiln](../components/ALKiln/automated_testing.mdx#accessibility)** is the Document Assembly Line's testing framework with built-in accessibility testing using [axe-core](https://github.com/dequelabs/axe-core) -- **[aXe-core](https://github.com/dequelabs/axe-core)** is an open-source accessibility testing engine used by many tools -- **[Pa11y](https://pa11y.org/)** is a command-line accessibility testing tool that can be integrated into CI/CD pipelines +- **[DAYamlChecker](../automated_quality_checks/dayamlchecker.md)**: Static analysis for interview YAML and Word templates, catching skipped headings, unlabelled fields, missing alt text, and non-descriptive links while you are still editing. +- **[SuffolkLITLab/ALActions](../automated_quality_checks/github_actions.md)**: The GitHub Actions pipeline that runs `dayamlchecker`, checks hyperlinks, and validates PDF templates with **veraPDF**. See the [automated quality checks overview](../automated_quality_checks/overview.md). +- **[ALKiln](../components/ALKiln/automated_testing.mdx#accessibility)**: Dynamic end-to-end browser testing framework with built-in accessibility testing using [aXe-core](https://github.com/dequelabs/axe-core). +- **[aXe-core](https://github.com/dequelabs/axe-core)**: An open-source accessibility testing engine used by ALKiln and browser DevTools. +- **[Pa11y](https://pa11y.org/)**: A command-line accessibility testing tool for rendered HTML pages. diff --git a/docs/coding_style/yaml.md b/docs/coding_style/yaml.md index c6f3407ea..69e4d3041 100644 --- a/docs/coding_style/yaml.md +++ b/docs/coding_style/yaml.md @@ -196,3 +196,7 @@ Remember that the descriptive name is great when you are sure that you will only name in the package. If you might edit the package later to add multiple documents, it might be best to start with `main.yml` right at the start of the interview. ::: + +## Validating YAML files automatically + +To ensure that YAML interview files adhere to correct syntax, do not duplicate keys, and follow docassemble structural rules, use **[DAYamlChecker](../automated_quality_checks/dayamlchecker.md)** locally and enable the **[ALActions `da_build`](../automated_quality_checks/github_actions.md#da_build)** workflow in GitHub. diff --git a/docs/components/ALKiln/intro.mdx b/docs/components/ALKiln/intro.mdx index 61e64f9b1..94650be0c 100644 --- a/docs/components/ALKiln/intro.mdx +++ b/docs/components/ALKiln/intro.mdx @@ -25,6 +25,10 @@ Ready to start testing? Need to quickly refresh your memory on some important po - [Write your first test](setup.mdx#first-test) - [Generate a first draft for more complicated tests](writing_tests.mdx#alkiln_story) +:::tip Looking for fast static checks? +ALKiln performs **dynamic, end-to-end browser testing** on running servers. If you are looking for fast, lightweight static analysis (linting YAML syntax, WCAG checks, broken URL scanning, Jinja2 template verification, and Word document diffing), see our **[Automated quality checks](../../automated_quality_checks/overview.md)** and **[DAYamlChecker](../../automated_quality_checks/dayamlchecker.md)** documentation. +::: + See [a few more reminders here](writing_tests.mdx#refresh). diff --git a/sidebars.js b/sidebars.js index 886d9ab10..cb6276b91 100644 --- a/sidebars.js +++ b/sidebars.js @@ -338,6 +338,17 @@ module.exports = { 'components/ALKiln/alkiln_security', 'components/ALKiln/alkiln_deprecated', ] + }, + { + label: 'Automated quality checks', + type: 'category', + items: [ + 'automated_quality_checks/automated_quality_checks_overview', + 'automated_quality_checks/running_checks_locally', + 'automated_quality_checks/dayamlchecker', + 'automated_quality_checks/github_actions', + 'automated_quality_checks/navigating_logs_and_artifacts', + ] } ] };