A FastAPI-based service that translates natural language instructions into structured geometric edits using GPT-4 with schema validation and constraint preservation.
NLPatch bridges the gap between natural language and formal geometric representations. It uses large language models (GPT-4) to interpret design intent and generate validated edit operations on geometric scene graphs, enabling intuitive CAD manipulation through conversational interfaces.
- Natural Language Processing: Interpret complex design instructions in plain English
- Structured Output Generation: Convert intent to validated JSON patch operations
- 17-Point Schema Validation: Ensure geometric consistency and constraint preservation
- Multi-Operation Support: Handle parameter updates, node creation/deletion, constraint management
- Fast Inference: <2s end-to-end latency for typical queries
- Production-Ready: FastAPI REST API with async support
# Create virtual environment
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate# Install dependencies
pip install fastapi uvicorn openai pydantic python-dotenv jsonschema
# Set OpenAI API keyecho"OPENAI_API_KEY=sk-..."> .env# Development mode with auto-reload
uvicorn main:app --reload --port 8001
# Production mode
uvicorn main:app --host 0.0.0.0 --port 8001 --workers 4curl -X POST http://localhost:8001/edit \
-H "Content-Type: application/json" \
-d '{ "instruction": "Make the box twice as wide and add a distance constraint of 15mm between box1 and sphere1", "current_scene": { "nodes": { "box1": { "op": "sdf.box", "params": {"width": 10.0, "height": 5.0, "depth": 3.0} }, "sphere1": { "op": "sdf.sphere", "params": {"radius": 8.0} } }, "constraints": [] } }'Response:
{
"patch": {
"actions": [
{
"type": "set_param",
"node_id": "box1",
"param_name": "width",
"value": 20.0
},
{
"type": "add_constraint",
"constraint": {
"id": "C1",
"type": "distance",
"targets": ["box1", "sphere1"],
"value": 15.0,
"weight": 1.0
}
}
]
},
"validation": {
"valid": true,
"checks_passed": 17
},
"latency_ms": 1847
}User Input (Natural Language)
↓
GPT-4 with System Prompt
↓
JSON Patch (Structured Edits)
↓
17-Point Schema Validator
↓
Validated Patch Output
The orchestrator supports 5 core operation types:
- set_param: Update parameter values
- add_node: Create new geometric nodes
- add_constraint: Add optimization constraints
- delete_node: Remove geometric nodes
- delete_constraint: Remove constraints
Convert natural language instruction to geometric patch.
Request Body:
{
"instruction": "string", // Natural language command"current_scene": { // Current scene graph"nodes": {
"node_id": {
"op": "string",
"params": { ... },
"inputs": ["string"]
}
},
"constraints": [ ... ]
},
"conversation_history": [ // Optional: previous exchanges
{
"role": "user",
"content": "string"
}
]
}Response:
{
"patch": {
"actions": [
{
"type": "set_param | add_node | delete_node | add_constraint | delete_constraint",
...
}
]
},
"validation": {
"valid": true,
"checks_passed": 17,
"errors": []
},
"reasoning": "string", // LLM's reasoning (if requested)"latency_ms": 1847
}Health check endpoint.
Response:
{
"status": "healthy",
"openai_configured": true,
"version": "1.0.0"
}Validate a patch without LLM inference.
Request Body:
{
"patch": { ... },
"current_scene": { ... }
}Response:
{
"valid": true,
"checks_passed": 17,
"errors": []
}Update a parameter value on an existing node.
{
"type": "set_param",
"node_id": "box1",
"param_name": "width",
"value": 20.0
}Create a new geometric node in the scene.
{
"type": "add_node",
"node_id": "cyl1",
"op": "sdf.cylinder",
"params": {
"radius": 5.0,
"height": 10.0
},
"inputs": []
}Remove a geometric node and clean up dependent references.
{
"type": "delete_node",
"node_id": "box1"
}Add an optimization constraint between nodes.
{
"type": "add_constraint",
"constraint": {
"id": "C1",
"type": "distance",
"targets": ["box1", "sphere1"],
"value": 15.0,
"weight": 1.0,
"samples": [
{
"node": "box1",
"point": [0, 0, 0]
},
{
"node": "sphere1",
"point": [0, 0, 0]
}
]
}
}Constraint Types:
distance: Maintain distance between geometriestangent: Make surfaces tangent at contact pointsangle: Maintain angle between surface normals
Remove an optimization constraint.
{
"type": "delete_constraint",
"constraint_id": "C1"
}The orchestrator performs 17 validation checks before returning patches:
- ✓ Patch has 'actions' array
- ✓ Each action has 'type' field
- ✓ Action types are valid
- ✓ All referenced nodes exist
- ✓ No circular dependencies in inputs
- ✓ Parameter names match operator schema
- ✓ Parameter values within valid ranges
- ✓ Constraint targets have geometry nodes
- ✓ Constraint types are valid
- ✓ Constraint samples reference valid nodes
- ✓ No duplicate node IDs
- ✓ No duplicate constraint IDs
- ✓ Numeric params are numbers
- ✓ String params are strings
- ✓ Array params are arrays
- ✓ Inputs array contains valid node IDs
- ✓ Transform parameters have correct dimensionality
The orchestrator understands 17 geometric operators:
sdf.box- Axis-aligned boxsdf.sphere- Spheresdf.cylinder- Cylindersdf.cone- Cone/frustumsdf.torus- Torussdf.capsule- Capsule/pill shapesdf.halfspace- Infinite plane
transform.translate- Translationtransform.rotate- Rotationtransform.scale- Scaling
boolean.union- Unionboolean.intersect- Intersectionboolean.subtract- Subtractionboolean.smooth_union- Smooth union
chart.bspline- B-spline curvechart.extrude- Linear extrusionchart.revolve- Surface of revolution
importrequests# Initial designscene= {
"nodes": {
"base": {"op": "sdf.box", "params": {"width": 100, "height": 20, "depth": 50}}
},
"constraints": []
}
# Iterative refinementinstructions= [
"Make the base 50% wider",
"Add a cylinder on top with radius 15 and height 30",
"Add a distance constraint of 5mm between base and cylinder"
]
forinstructionininstructions:
response=requests.post('http://localhost:8001/edit', json={
'instruction': instruction,
'current_scene': scene
})
patch=response.json()['patch']
scene=apply_patch(scene, patch) # Apply patch to sceneprint("Final scene:", scene)# Complex constraint setup via natural languageinstruction="""Create a gear assembly:1. Add a cylinder gear1 with radius 20mm and height 10mm2. Add a second cylinder gear2 with radius 15mm and height 10mm3. Add a distance constraint of 35mm between their centers4. Add a tangent constraint where they mesh"""response=requests.post('http://localhost:8001/edit', json={
'instruction': instruction,
'current_scene': empty_scene
})
patch=response.json()['patch']
# Patch will contain 4 add_node actions + 2 add_constraint actions# Conversational editing with historyhistory= []
# First editresponse1=requests.post('http://localhost:8001/edit', json={
'instruction': "Create a box named housing with dimensions 100x50x30",
'current_scene': scene,
'conversation_history': history
})
history.append({"role": "user", "content": "Create a box named housing..."})
history.append({"role": "assistant", "content": response1.json()['reasoning']})
# Second edit with contextresponse2=requests.post('http://localhost:8001/edit', json={
'instruction': "Make it 20% taller", # "it" refers to housing'current_scene': updated_scene,
'conversation_history': history
})# Required
OPENAI_API_KEY=sk-...
# Optional
OPENAI_MODEL=gpt-4 # Default: gpt-4
OPENAI_TEMPERATURE=0.0 # Default: 0.0 (deterministic)
OPENAI_MAX_TOKENS=2000 # Default: 2000
LOG_LEVEL=INFO # Default: INFO
PORT=8001 # Default: 8001Edit system_prompt.txt to customize LLM behavior:
# main.pywithopen('system_prompt.txt') asf:
SYSTEM_PROMPT=f.read()
# Modify prompt to emphasize specific behaviorsSYSTEM_PROMPT+="""Additional Guidelines:- Always preserve existing constraints unless explicitly asked to remove them- Prefer smooth_union over sharp union for organic shapes- Use standard gear modules (1, 1.5, 2, 2.5, 3, 4, 5mm)"""fromfunctoolsimportlru_cacheimporthashlib@lru_cache(maxsize=1000)defcached_llm_call(instruction_hash, scene_hash):
# Cache repeated queriesreturncall_openai(instruction, scene)
# Usageinstruction_hash=hashlib.md5(instruction.encode()).hexdigest()
scene_hash=hashlib.md5(json.dumps(scene, sort_keys=True).encode()).hexdigest()
result=cached_llm_call(instruction_hash, scene_hash)fromfastapiimportBackgroundTasks@app.post("/edit_async")asyncdefedit_async(request: EditRequest, background_tasks: BackgroundTasks):
task_id=generate_task_id()
background_tasks.add_task(process_edit, task_id, request)
return {"task_id": task_id, "status": "processing"}
@app.get("/status/{task_id}")asyncdefget_status(task_id: str):
result=get_result(task_id)
return {"status": result.status, "patch": result.patchifresult.doneelseNone}@app.post("/edit_batch")asyncdefedit_batch(requests: List[EditRequest]):
# Process multiple edits in paralleltasks= [edit_single(req) forreqinrequests]
results=awaitasyncio.gather(*tasks)
return {"results": results}FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8001
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8001"]docker build -t nlpatch:latest .
docker run -p 8001:8001 -e OPENAI_API_KEY=sk-... nlpatch:latestapiVersion: apps/v1kind: Deploymentmetadata:
name: nlpatchspec:
replicas: 2selector:
matchLabels:
app: orchestratortemplate:
metadata:
labels:
app: orchestratorspec:
containers:
- name: apiimage: nlpatch:latestports:
- containerPort: 8001env:
- name: OPENAI_API_KEYvalueFrom:
secretKeyRef:
name: openai-secretkey: api-keyresources:
requests:
memory: "512Mi"cpu: "500m"limits:
memory: "1Gi"cpu: "1000m"
---
apiVersion: v1kind: Servicemetadata:
name: orchestrator-servicespec:
selector:
app: orchestratorports:
- port: 80targetPort: 8001type: LoadBalancer# lambda_handler.pyfrommangumimportMangumfrommainimportapphandler=Mangum(app)Deploy with AWS SAM:
# template.yamlAWSTemplateFormatVersion: '2010-09-09'Transform: AWS::Serverless-2016-10-31Resources:
OrchestratorFunction:
Type: AWS::Serverless::FunctionProperties:
CodeUri: .Handler: lambda_handler.handlerRuntime: python3.11Timeout: 30Environment:
Variables:
OPENAI_API_KEY: !Ref OpenAIKeyEvents:
EditAPI:
Type: ApiProperties:
Path: /editMethod: post# Example error response
{
"error": {
"type": "ValidationError",
"message": "Parameter 'width' out of range [0.1, 1000.0]",
"details": {
"node_id": "box1",
"param_name": "width",
"value": 5000.0,
"valid_range": [0.1, 1000.0]
}
},
"patch": null,
"validation": {
"valid": false,
"checks_passed": 16,
"errors": ["Check 7 failed: Parameter values within valid ranges"]
}
}# Run unit tests
pytest tests/
# Run integration tests with live API
pytest tests/integration/ --api-url http://localhost:8001
# Test with mock LLM (no API key required)
pytest tests/ --mock-llm# Clone repository
git clone https://github.com/YOUR_USERNAME/nlpatch.git
cd nlpatch
# Create environment
python -m venv venv
source venv/bin/activate
# Install dependencies
pip install -r requirements.txt
pip install -r requirements-dev.txt # For testing# Run tests
pytest
# Start server
uvicorn main:app --reloadSee examples/ directory:
basic_editing.py- Simple parameter updatesconstraint_setup.py- Adding multiple constraintsconversational_cad.py- Multi-turn editing sessionbatch_processing.py- Batch edit operations
- LLM Dependencies: Requires active OpenAI API access
- Latency: 1-3s per request depending on instruction complexity
- Cost: GPT-4 API costs ~$0.03-0.06 per 1000 requests
- Context Limits: Very large scenes may exceed token limits (use scene summarization)
If you use NLPatch in academic work, please cite:
@software{modelspace_orchestrator,
title={NLPatch: LLM-Driven Geometric Editing},
author={Your Name},
year={2025},
url={https://github.com/YOUR_USERNAME/nlpatch}
}MIT License - see LICENSE file for details
Contributions welcome! See CONTRIBUTING.md for guidelines.
- Issues: https://github.com/YOUR_USERNAME/nlpatch/issues
- Documentation: https://nlpatch.readthedocs.io
- Email: support@example.com