Parameters Package and Experiments

experiments/param-opti/.gitignore

@@ -0,0 +1,2 @@
+output/
+repos/

experiments/param-opti/Agent.md

@@ -0,0 +1,31 @@
+# Tool Parameter Extraction
+
+An experiment to extract configuration (hyper)parameters from tools that perform data integration tasks and cluster them to show common options.
+
+The extraction and clustering code is under src/kgpipe_parameters
+The experiment code using this is under experimets/param-opti
+
+Trace the implementation state under spec/
+Reuse its state and extend it for each new feature.
+Tack issues under spec/fix_needed.md
+
+# Implementation Requirements
+- Implement parameter extractors from
+    - docker doc
+    - python lib
+    - cli help
+    - http api doc
+    - repo Readme.md/Doc
+- The extractors can use LLMs or rules/regex patterns
+- Find similar parameters between the single tools implementing a cluster strategy
+  - using sentence transformer embeddings
+  - prompting llms with preselected terms
+- Visualize the clusters
+
+# Success Criteria
+- simple tests for each extractors
+- a working experiment in experiments/param-opti
+- a table with configuration parameters
+- A vizualization output
+
+I want you to check missing features and just implement the next feature required now.

experiments/param-opti/README.md

@@ -0,0 +1,100 @@
+# Parameter Optimization Experiment
+
+This experiment extracts and analyzes configuration parameters from open-source data integration tools using the `kgpipe_parameters` extraction module.
+
+## Directory Structure
+
+```
+param-opti/
+├── input/              # Tool definitions
+│   ├── paris/
+│   │   ├── repo.url    # Git repository URL
+│   │   └── cli.txt     # CLI help output (optional)
+│   └── corenlp_openie/
+│       └── repo.url
+├── repos/              # Cloned repositories (auto-populated)
+├── output/             # Extraction results (JSON)
+├── src/
+│   └── param_opti/     # Experiment code
+└── run_experiment.py   # Main entry point
+```
+
+## Usage
+
+### Run full experiment
+
+```bash
+# From kgpipe root (with venv activated)
+cd experiments/param-opti
+python run_experiment.py
+```
+
+### Run for specific tool
+
+```bash
+python run_experiment.py --tool paris
+python run_experiment.py --tool paris corenlp_openie
+```
+
+### Skip repository cloning
+
+```bash
+python run_experiment.py --no-clone
+```
+
+### Use LLM-based extraction (requires kgpipe_llm)
+
+```bash
+python run_experiment.py --use-llm
+```
+
+## Adding New Tools
+
+1. Create a folder in `input/` with the tool name
+2. Add `repo.url` with the Git repository URL
+3. Optionally add `cli.txt` with CLI help output
+4. Optionally add `config.json` for additional settings:
+
+```json
+{
+    "language": "python",
+    "main_files": ["src/main.py", "cli.py"]
+}
+```
+
+## Output Format
+
+Results are saved as JSON files in `output/`:
+
+```json
+{
+    "tool_name": "paris",
+    "timestamp": "2024-...",
+    "sources": [
+        {
+            "source_type": "cli",
+            "file_path": "input/paris/cli.txt",
+            "parameters_count": 5
+        }
+    ],
+    "parameters": [
+        {
+            "name": "threshold",
+            "native_keys": ["--threshold"],
+            "description": "Matching threshold",
+            "type_hint": "float",
+            "default_value": 0.5,
+            "_source": "cli"
+        }
+    ],
+    "summary": {
+        "total_parameters": 5,
+        "total_sources": 1,
+        "total_errors": 0
+    }
+}
+```
+
+A `_summary.json` file is also generated with aggregate statistics.
+
+

experiments/param-opti/input/corenlp_openie/repo.url

@@ -0,0 +1 @@
+https://github.com/stanfordnlp/CoreNLP.git

experiments/param-opti/input/paris/cli.txt

@@ -0,0 +1,10 @@
+Paris <settingFile>
+
+You can specify a file that has no content.
+PARIS will ask for the necessary data and store it in <settingFile>.
+
+Paris <ontology1> <ontology2> <outputFolder>
+
+Shorthand for the previous form.
+
+Paris <factstore> <dump>

experiments/param-opti/input/paris/repo.url

@@ -0,0 +1 @@
+https://github.com/dig-team/PARIS.git

experiments/param-opti/input/valentine/repo.url

@@ -0,0 +1 @@
+https://github.com/delftdata/valentine.git

experiments/param-opti/run_experiment.py

@@ -0,0 +1,27 @@
+#!/usr/bin/env python3
+"""
+Quick script to run the parameter extraction experiment.
+
+This script can be run directly from the param-opti directory:
+    python run_experiment.py
+    python run_experiment.py --tool paris
+    python run_experiment.py --no-clone
+"""
+
+import sys
+from pathlib import Path
+
+# Add src to path
+src_path = Path(__file__).parent / "src"
+sys.path.insert(0, str(src_path))
+
+# Also ensure kgpipe is importable
+kgpipe_src = Path(__file__).parent.parent.parent / "src"
+sys.path.insert(0, str(kgpipe_src))
+
+from param_opti.__main__ import main
+
+if __name__ == "__main__":
+    sys.exit(main())
+
+

experiments/param-opti/spec/implementation_state.md

@@ -0,0 +1,123 @@
+# Implementation State
+
+Last updated: 2026-02-20
+
+## Parameter Extractors
+
+| Source Type     | Regex | LLM | Tests | Module                          |
+|-----------------|-------|-----|-------|---------------------------------|
+| CLI help        | ✅    | ✅  | ✅    | `extractors/cli.py`             |
+| Python lib      | ✅    | ✅  | ✅    | `extractors/python_lib.py`      |
+| HTTP API doc    | ✅    | ✅  | ✅    | `extractors/http_api.py`        |
+| Docker doc      | ✅    | ✅  | ✅    | `extractors/docker.py`          |
+| Repo README/Doc | ✅    | ✅  | ✅    | `extractors/readme_doc.py`      |
+
+All extractors live under `src/kgpipe_parameters/extraction/extractors/`.
+
+## Core Infrastructure
+
+| Component                  | Status | Location                                      |
+|----------------------------|--------|-----------------------------------------------|
+| Models                     | ✅     | `extraction/models.py`                        |
+| Base classes               | ✅     | `extraction/base.py`                          |
+| Regex patterns             | ✅     | `extraction/patterns.py`                      |
+| Utilities                  | ✅     | `extraction/utils.py`                         |
+| ParameterMiner             | ✅     | `extraction/param_miner.py`                   |
+| Auto source detect         | ✅     | `param_miner._detect_source_type()`           |
+| Keyword chunk filter       | ✅     | `extraction/chunk_filter.py`                  |
+
+## Keyword Chunk Filter
+
+Keyword-based pre-filter that scores chunks before they reach any extractor.
+Counts parameter-signal keywords per language/file-type and skips files below
+a configurable threshold.  No embeddings, zero extra dependencies.
+
+| Language / Type | Keywords cover                                          | Threshold |
+|-----------------|---------------------------------------------------------|-----------|
+| Python          | argparse, click, dataclass, Field, os.environ, …       | 2         |
+| Java            | @Option, @Parameter, getProperty, Properties, @Value,… | 1         |
+| .properties     | `=`, `:`                                                | 1         |
+| XML             | `<property`, `<param`, `<config`, `name=`, `value=`, … | 1         |
+| Docker          | ENV, ARG, EXPOSE, environment:, …                      | 2         |
+| README          | --, default, parameter, configuration, usage, …        | 1*        |
+| Generic         | default, param, config, option, argument, …            | 2         |
+
+\* README files use threshold 1 in `extract_from_repo()`.
+
+**Tests:** `tests/test_chunk_filter.py` — 29 tests.
+
+## Experiment Pipeline
+
+| Step                             | Status | Location                              |
+|----------------------------------|--------|---------------------------------------|
+| Tool discovery from input/       | ✅     | `experiment.py` + `tool.py`           |
+| Repository cloning               | ✅     | `experiment.py`                       |
+| CLI extraction                   | ✅     | `experiment.py:extract_from_cli()`    |
+| README extraction (input)        | ✅     | `experiment.py:extract_from_readme()` |
+| Repo file extraction             | ✅     | `experiment.py:extract_from_repo()`   |
+| Keyword chunk filter in pipeline | ✅     | `experiment.py:extract_from_repo()`   |
+| JSON output + summary            | ✅     | `experiment.py:save_result()`         |
+
+### File types scanned by `extract_from_repo()`
+
+| File type      | Extractor used    | Chunk filter | Limit |
+|----------------|-------------------|--------------|-------|
+| `*.py`         | PYTHON_LIB        | ✅ (≥2)      | 20    |
+| `*.java`       | README (kv+flags) | ✅ (≥1)      | 20    |
+| `*.properties` | README (kv)       | inherent     | 15    |
+| `*.xml` (cfg)  | README            | ✅ (≥1)      | 10    |
+| `Dockerfile*`  | DOCKER            | —            | all   |
+| `docker-compose*` | DOCKER         | —            | all   |
+| `README*`, `*.md`, docs/ | README  | ✅ (≥1)      | 15    |
+
+## Parameter Clustering
+
+Cluster similar parameters across tools using sentence-transformer embeddings
+and agglomerative clustering.
+
+| Component                | Status | Location                                       |
+|--------------------------|--------|-------------------------------------------------|
+| Clustering models        | ✅     | `clustering/models.py`                          |
+| Embedding similarity     | ✅     | `clustering/similarity.py`                      |
+| Agglomerative clusterer  | ✅     | `clustering/clusterer.py`                        |
+| Cluster __init__         | ✅     | `clustering/__init__.py`                         |
+| Experiment integration   | ✅     | `experiment.py:cluster_parameters()`             |
+| CLI flags                | ✅     | `--cluster`, `--cluster-only`, `--distance-threshold` |
+| JSON + CSV output        | ✅     | `_clusters.json`, `_parameter_table.csv`         |
+| Tests                    | ✅     | `tests/test_clustering.py` — 20 tests            |
+
+**Approach:**
+1. Load extracted parameters from per-tool JSON output files.
+2. Build text representation (name + description + native_keys + type_hint).
+3. Encode with `all-MiniLM-L6-v2` sentence-transformer.
+4. L2-normalise embeddings; compute cosine distance matrix.
+5. Apply scikit-learn `AgglomerativeClustering` (average linkage, precomputed
+   distance, configurable threshold — default 0.55).
+6. Label each cluster by most-common parameter name.
+7. Output `_clusters.json` (full cluster details) and `_parameter_table.csv`
+   (flat, one row per parameter with cluster assignment).
+
+## Visualization
+
+Static plots generated from clustering results using matplotlib/seaborn.
+
+| Component                | Status | Location                                          |
+|--------------------------|--------|---------------------------------------------------|
+| ParameterVisualizer      | ✅     | `visualization/kgpipe_parameter_explorer.py`      |
+| Cluster size bar chart   | ✅     | `_viz_cluster_sizes.png`                          |
+| Tool × cluster heatmap   | ✅     | `_viz_tool_heatmap.png`                           |
+| 2-D PCA embedding scatter| ✅     | `_viz_embedding_scatter.png`                      |
+| CLI `--visualize` flag   | ✅     | `__main__.py`                                     |
+| Experiment integration   | ✅     | `experiment.py:visualize_clusters()`              |
+| Tests                    | ✅     | `tests/test_visualization.py` — 7 tests           |
+
+**Plots:**
+1. **Cluster sizes** — horizontal bar chart (top-30), cross-tool clusters highlighted.
+2. **Tool × cluster heatmap** — parameter count per tool per cross-tool cluster.
+3. **Embedding scatter** — PCA-2D projection of parameter embeddings, coloured by tool.
+
+## Still TODO
+
+- [x] Visualization (`visualization/kgpipe_parameter_explorer.py`)
+- [ ] Optimization (`optimization/` is empty)
+- [ ] Embedding-based RAG for LLM prompts (later, when keyword filter plateaus)