🤖 A TypeScript simulation framework for testing and running AI agents
SimKit lets you build, test, and run AI agents in your own custom simulated environments. It gives you a simple game loop for running agents step-by-step, supports multiple agents, and includes built-in tools (OTEL) for tracking what happens during your simulations.
SimKit works with any AI agent or LLM, no lock-in. Use your own models and run everything locally. OTEL logs can be saved to a local file or sent to a remote server.
Simulations let you see how your AI agents perform on real world tasks, step by step, in a safe and controlled way.
Traditional evals are great for simple tasks, but they don't give you the full picture. You can't see how your agents handle:
- 🎯 Multi-step tasks that need planning and memory
- 🛠️ Lots of different tools and actions
- 🌍 Realistic data and changing situations
- ⚡ Decisions that matter over time
- 🔄 Long-term planning and decision-making
- 📚 Processing and reasoning over large amounts of context and information
Surprisingly, most AI agents begin to fail when they are asked to do anything more than a few simple tasks.
SimKit's heart is a simple but powerful tick-based loop:
import{createSimulation,typeLoopState}from"@fallom/simkit/simulation";interfaceSupportTestStateextendsLoopState{totalIssues: number;resolvedIssues: number;averageResponseTime: number;satisfactionScores: number[];}constcustomerIssues=["My account is locked and I can't access my files","Billing error - charged twice for same month","App crashes every time I try to upload","Can't find my downloaded files anywhere"];constsimulation=createSimulation<SupportTestState>({maxTicks: 10,initialState: {totalIssues: 0,resolvedIssues: 0,averageResponseTime: 0,satisfactionScores: []},onTick: async(state)=>{// Get today's customer issuesconstdailyIssues=getRandomIssues(customerIssues,2);for(constissueofdailyIssues){conststartTime=Date.now();// Test your AI support agentconstagentResponse=awaitsupportAgent.handle(issue);constresponseTime=Date.now()-startTime;constsatisfaction=scoreResponse(agentResponse,issue);state.totalIssues++;if(satisfaction>7)state.resolvedIssues++;state.satisfactionScores.push(satisfaction);// Update running averagesconstavgSatisfaction=state.satisfactionScores.reduce((a,b)=>a+b,0)/state.satisfactionScores.length;constresolutionRate=(state.resolvedIssues/state.totalIssues)*100;console.log(`Resolution Rate: ${resolutionRate.toFixed(1)}% | Avg Satisfaction: ${avgSatisfaction.toFixed(1)}/10`);}returnstate.tick<9;// Test for 10 days},onEnd: (state)=>{constfinalSatisfaction=state.satisfactionScores.reduce((a,b)=>a+b,0)/state.satisfactionScores.length;console.log(`🎯 Final Results: ${((state.resolvedIssues/state.totalIssues)*100).toFixed(1)}% resolution rate, ${finalSatisfaction.toFixed(1)}/10 satisfaction`);}});awaitsimulation.run();What's happening here? Each tick simulates a day of customer support. SimKit feeds random issues to your AI agent, measures response quality and speed, then tracks KPIs over time. Perfect for A/B testing different models, regression testing after prompt changes, or measuring performance before production deployment.
AI agents need access to simulation state from anywhere:
import{setSimState,getSimState}from"@fallom/simkit/state";// In your simulation loopsetSimState(state);// In your AI toolsconstcurrentState=getSimState<MyState>();Reproduce exact scenarios with seeded randomness - perfect for fair model comparisons:
import{initializeRandom,choice,shuffle}from"@fallom/simkit/random";// Test Model AinitializeRandom(12345);// Same seed = same test scenariosconstmodelA_results=awaittestSupportAgent(modelA);// Test Model B with identical scenariosinitializeRandom(12345);// Reset to same seedconstmodelB_results=awaittestSupportAgent(modelB);// Now you can fairly compare: both models faced the exact same issuesconsole.log(`Model A: ${modelA_results.satisfaction}/10`);console.log(`Model B: ${modelB_results.satisfaction}/10`);Why this matters: Without seeded randomness, Model A might get easy customer issues while Model B gets hard ones, making comparison meaningless. SimKit ensures every model faces identical test scenarios.
Built-in observability for AI agent debugging with zero vendor lock-in:
import{trace}from"@opentelemetry/api";// SimKit automatically captures spans for youconsttracer=trace.getTracer("my-simulation");constspan=tracer.startSpan("agent-decision");span.setAttributes({"agent.action": "support_response","simulation.tick": state.tick,"response.satisfaction": 8.5});span.end();Send telemetry anywhere: Export to your own servers, store in local files, or pipe to any OpenTelemetry-compatible service. No vendor lock-in - you own your data.
| Feature | Why It Matters for AI |
|---|---|
| 🔄 Tick-Based Loop | Step-by-step agent execution with full control |
| 📊 OpenTelemetry | Track agent decisions and debug complex behaviors |
| 🎲 Seeded Random | Reproduce exact scenarios for testing and validation |
| 🏗️ Global State | AI tools can access simulation state from anywhere |
| 🔧 TypeScript | Full type safety for complex agent interactions |
| ⚡ Bun Optimized | Fast execution for compute-intensive agent simulations |
npm install @fallom/simkit
# or
bun add @fallom/simkitSimple agent making strategic decisions
cd apps/examples/energy-ai
bun install && bun run startA straightforward example showing:
- AI agent with tool calling
- Basic state management
- OpenTelemetry integration
Complex multi-agent economic simulation
A comprehensive example demonstrating SimKit's full capabilities:
- Multi-agent system - Shop owner + customer agents
- Complex state management - Inventory, trades, conversations
- Deterministic scenarios - Seeded randomness for testing
- Rich telemetry - Custom spans and detailed logging
- Tool ecosystem - AI agents with 10+ specialized tools
Perfect for understanding how to build production-grade agent simulations.
| Traditional Approach | With SimKit |
|---|---|
| ❌ Manual loop management | ✅ Built-in tick-based execution |
| ❌ No observability | ✅ OpenTelemetry integration |
| ❌ Non-deterministic testing | ✅ Seeded randomness |
| ❌ Complex state sharing | ✅ Global state management |
| ❌ Manual telemetry setup | ✅ Automatic span collection |
- 📦 Core Package Docs - Full API reference
- 🚀 Energy AI Tutorial - Simple getting started guide
- 🏆 Pawn Shop Deep Dive - Advanced multi-agent patterns
# Install dependencies
bun install
# Build all packages
bun run build
# Format code
bun run format