data-agent-eval-kit

Production-ready evaluation templates for Data Agent, AI ask-data, AI fetch-data and NL2SQL workflows.

这个仓库解决一个很具体的问题：

AI 问数 Demo 很容易做出来，但上线前到底该怎么测？

很多团队只验证“模型能不能生成 SQL”。这不够。生产环境真正要验证的是：

• 指标口径是否正确
• SQL 是否可执行
• 时间归属是否正确
• 订单数、用户数、GMV 是否按正确粒度计算
• 敏感字段是否被拦截
• 大表查询是否带分区
• Prompt / Skill / 模型改动后是否发生回归
• Bad Case 能不能被归因和修复

data-agent-eval-kit 提供一套可以直接改造落地的模板。

What is inside

data-agent-eval-kit
├── templates
│ ├── gold_case.schema.yaml
│ ├── bad_case_taxonomy.yaml
│ └── judge_rubric.yaml
├── examples
│ ├── trade_gold_cases.yaml
│ └── baseline_sql
├── rules
│ └── sql_review_rules.yaml
├── evaluator
│ ├── evaluate_cases.py
│ └── requirements.txt
├── reports
│ └── sample_eval_report.md
└── docs
├── HOW_TO_BUILD_DATA_AGENT_EVALS.md
└── PRODUCTION_READINESS_CHECKLIST.md

Evaluation framework

The evaluation is not a single score.

It is a production gate:

User question
-> intent / metric / time / dimension extraction
-> schema and metric matching
-> SQL generation
-> SQL review
-> baseline comparison
-> permission check
-> bad case attribution
-> regression report

Example Gold Case

case_id: TRADE_GMV_001task_type: ask_datapriority: P0question: 昨天平台支付 GMV 是多少？business_context: 电商交易主题，统计支付成功商品行金额。required_metrics:
- pay_gmvtime_policy:
field: pay_success_daterange: yesterdaymust_use:
- sum(sku_pay_amount)
- dt = '${yesterday}'must_not_use:
- sum(order_amount)
- create_timebaseline_sql_file: examples/baseline_sql/TRADE_GMV_001.sqlexpected_checks:
- metric_consistency
- time_policy_correctness
- partition_filter_requirederror_tags:
- metric_error
- time_policy_error
- partition_missing

Quick start

cd evaluator
pip install -r requirements.txt
python evaluate_cases.py \
--cases ../examples/trade_gold_cases.yaml \
--sql-dir ../examples/generated_sql \
--rules ../rules/sql_review_rules.yaml

The evaluator is intentionally simple. It does not replace your real SQL engine or business validation. It helps you run a first-pass gate:

• required SQL fragments
• forbidden SQL fragments
• sensitive field detection
• partition filter checks
• dangerous SQL operation checks
• case-level report generation

Who should use this

• Data warehouse engineers building AI ask-data systems
• Data platform teams building internal Data Agent
• Analytics engineers evaluating NL2SQL
• AI product teams building ChatBI / Text2SQL
• Teams trying to move from demo to production

Why this matters

AI ask-data fails silently.

The dangerous case is not when SQL fails to execute. The dangerous case is when SQL executes successfully and returns a wrong number.

This repo is built around one principle:

Data Agent should not be trusted because it sounds fluent. It should be trusted only after it survives real business cases.

Roadmap

• Add more business domains: user growth, inventory, marketing, finance
• Add report-generation evaluation cases
• Add Langfuse / LangSmith trace examples
• Add dbt semantic layer examples
• Add MCP tool evaluation templates
• Add CI workflow for regression checks