Lilac + Eval Protocol: Diverse Dataset Curation

Use Lilac to automatically cluster and sample diverse examples from your LLM traces for evaluation.

This is an example showing how to use Eval Protocol's utilities with Lilac for intelligent data curation.

What This Does

When evaluating LLMs, running on all your production traces is expensive and often redundant—many queries are semantically similar. This integration:

1. Pulls traces from Langfuse (or any supported observability platform)
2. Clusters them semantically using embeddings + HDBSCAN
3. Samples diverse examples from each cluster
4. Evaluates the representative subset

Result: Instead of evaluating 1000 similar traces, you evaluate 30 diverse ones that cover all query types.

100 traces → Lilac clustering → 6 semantic groups → 12 diverse samples

Quick Start

1. Setup

# Clone this repo
git clone <repo-url>cd lilac-eval-example
# Run setup script (creates venv and installs everything)
./setup.sh

2. Configure API Keys

cp env.template .env
# Edit .env with your keys

3. Run

source .venv/bin/activate
pytest test_lilac_preprocessing.py -v -s

---

How Lilac Integration Works

The Integration Point

The key is the preprocess_fn parameter in DynamicDataLoader. This function receives ALL loaded rows and returns a filtered/transformed subset:

@evaluation_test(data_loaders=DynamicDataLoader(generators=[langfuse_traces_generator],preprocess_fn=lilac_cluster_and_sample, # ← Your Lilac logic here! ), ...)deftest_my_evaluation(row: EvaluationRow) ->EvaluationRow:
returnevaluate(row)

The Preprocessing Pipeline

deflilac_cluster_and_sample(rows: List[EvaluationRow]) ->List[EvaluationRow]:
""" 1. Convert to DataFrame (for Lilac compatibility) 2. Create Lilac dataset 3. Cluster on user queries 4. Sample from each cluster 5. Convert back to EvaluationRows """importlilacasll# Step 1: Convert to DataFrame using eval-protocol utilitydf=evaluation_rows_to_dataframe(rows)
df["user_query"] =df["messages_json"].apply(extract_first_user_message)
# Step 2: Create Lilac datasetconfig=ll.DatasetConfig(
namespace="local",
name="my_dataset",
source=ll.PandasSource(df),
)
dataset=ll.create_dataset(config)
# Step 3: Cluster (Lilac handles embedding + UMAP + HDBSCAN)dataset.cluster("user_query")
# Step 4: Sample diverse examples from each clusterdf=dataset.to_pandas(include_signals=True)
# ... sampling logic per cluster ...# Step 5: Convert back using eval-protocol utilityreturndataframe_to_evaluation_rows(df)

Clustering Pipeline (What Lilac Does)

┌─────────────┐ ┌──────────────┐ ┌─────────────┐ ┌──────────────┐
│ User Queries │ ──▶ │ Embed with │ ──▶ │ UMAP │ ──▶ │ HDBSCAN │
│ (text) │ │ Transformers │ │ (dim reduce)│ │ (clustering) │
└─────────────┘ └──────────────┘ └─────────────┘ └──────────────┘
│
▼
┌─────────────┐ ┌──────────────┐ ┌─────────────────────────────────┐
│ Output: │ ◀── │ Sample N per │ ◀── │ Clusters with auto-generated │
│ Diverse Set │ │ cluster │ │ titles (via LLM, optional) │
└─────────────┘ └──────────────┘ └─────────────────────────────────┘

1. Embeds each user query using sentence transformers (jina-embeddings-v2-small-en)
2. Reduces dimensions with UMAP (512 → 5 dimensions)
3. Clusters with HDBSCAN (automatically determines cluster count)
4. Names clusters using an LLM (optional, requires API_MODEL env var)
5. Samples N examples from each cluster for diversity

Example Output

============================================================
🌸 LILAC PREPROCESSING
============================================================
📥 Input: 100 rows
🧮 Clustering user queries...
Method: Embed → UMAP → HDBSCAN
Cluster naming: LLM (gpt-4o-mini)
📊 Found 6 clusters:
--------------------------------------------------
Cluster 0 "Account Management Requests": 14 items
e.g., "Update phone number on account"
Cluster 1 "Order Returns and Refunds": 26 items
e.g., "ORD-54656 shipping status?"
Cluster 2 "Customer Service Inquiries": 17 items
e.g., "Recovery options change"
✅ Output: 12 diverse samples
Strategy: 2 per cluster, max 30 total
============================================================

---

Eval Protocol Utilities Used

This example uses several Eval Protocol utilities that enable the Lilac integration:

1. DataFrame Conversion - Bridge Between EvaluationRows and Pandas

fromeval_protocol.adapters.lilacimport (
evaluation_rows_to_dataframe,
dataframe_to_evaluation_rows,
)
# Convert EvaluationRows → DataFrame (for Lilac/pandas processing)df=evaluation_rows_to_dataframe(rows)
# ... do clustering, filtering, transformations with pandas/Lilac ...# Convert DataFrame → EvaluationRows (back to eval-protocol format)filtered_rows=dataframe_to_evaluation_rows(df)

2. Trace Adapters - Pull Data from Observability Platforms

fromeval_protocolimportcreate_langfuse_adapter# Create adapter for your platformadapter=create_langfuse_adapter()
# Pull traces and convert to EvaluationRowsrows=adapter.get_evaluation_rows(
limit=100, # How many traceshours_back=168, # Time window (7 days)include_tool_calls=True# Include function calls
)

Supported platforms:

• create_langfuse_adapter() - Langfuse
• create_langsmith_adapter() - LangSmith
• create_braintrust_adapter() - Braintrust
• create_fireworks_tracing_adapter() - Fireworks Tracing

3. DynamicDataLoader - Flexible Data Loading with Preprocessing

fromeval_protocolimportDynamicDataLoaderdata_loader=DynamicDataLoader(
generators=[my_data_generator], # Functions that return EvaluationRowspreprocess_fn=my_preprocess_fn, # Transform rows before evaluation
)

---

Configuration

Edit these constants in test_lilac_preprocessing.py:

SAMPLES_PER_CLUSTER=2# How many samples from each clusterMAX_TOTAL_SAMPLES=30# Cap on total output rowsLANGFUSE_LIMIT=100# How many traces to pull from Langfuse

---

Bring Your Own Data

From Different Trace Sources

# Langfuseadapter=create_langfuse_adapter()
rows=adapter.get_evaluation_rows(limit=100)
# LangSmithadapter=create_langsmith_adapter()
rows=adapter.get_evaluation_rows(limit=100)
# From a JSONL filedefload_from_file():
withopen("traces.jsonl") asf:
return [EvaluationRow.from_dict(json.loads(line)) forlineinf]

Custom Preprocessing (Without Lilac)

You can write any preprocessing logic:

defmy_custom_preprocess(rows: List[EvaluationRow]) ->List[EvaluationRow]:
# Filter by lengthrows= [rforrinrowsiflen(r.last_user_message().content) >10]
# Deduplicateseen=set()
unique= []
forrinrows:
key=r.last_user_message().content[:100]
ifkeynotinseen:
seen.add(key)
unique.append(r)
# Random samplereturnrandom.sample(unique, min(50, len(unique)))

---

Advanced Lilac Features

Beyond clustering, Lilac offers:

Semantic Search

dataset.search("user_query", "password reset", limit=10)

Concept Detection

fromlilac.signalsimportPIISignaldataset.compute_signal(PIISignal(), "user_query")

Lilac Web UI

importlilacasllll.start_server() # Interactive UI at localhost:5432

📚 Full Lilac Documentation:https://docs.lilacml.com/

---

API Keys Reference

Required:

LANGFUSE_PUBLIC_KEY=pk-lf-... # Langfuse public key
LANGFUSE_SECRET_KEY=sk-lf-... # Langfuse secret key
LANGFUSE_HOST=https://cloud.langfuse.com
FIREWORKS_API_KEY=fw_... # For model evaluation

Optional (for LLM cluster naming):

OPENAI_API_KEY=sk-... # OpenAI API key
API_MODEL=gpt-4o-mini # Model for naming clusters

---

Troubleshooting

Test skipped with "No Langfuse credentials"

Ensure .env has valid LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY.

"you must provide a model parameter"

Set API_MODEL for LLM cluster naming, or it falls back to generic names.

"HDBSCAN: X noise points"

Normal! Uncertain points are assigned to nearest cluster automatically.

Slow first run

First run downloads embedding model (~400MB). Subsequent runs use cache.

---

Requirements

• Python 3.10+
• ~2GB disk space for embedding model cache
• API keys for trace source + evaluation model

License

MIT