🤖 FlowLLM

Production-ready SDK for building AI agents with MCP tools

Quick Start • Features • Examples • Architecture • Documentation

---

What is FlowLLM?

FlowLLM is a production-ready SDK that makes building AI agents with Model Context Protocol (MCP) tools ridiculously easy.

Unlike existing frameworks that are either too low-level, too opinionated, or missing MCP support, FlowLLM gives you:

• ✅ Model-agnostic by default (OpenAI, Anthropic, Gemini, local models)
• ✅ MCP-native integration (use any MCP server as agent tools)
• ✅ Production primitives (streaming, retries, error handling, cost tracking)
• ✅ Deploy anywhere (works with any Node.js hosting platform)

import{defineAgent,openai}from'@targetly-labs/flowllm';constagent=defineAgent({provider: openai('gpt-4o'),systemPrompt: 'You are a helpful assistant.',});constresponse=awaitagent.execute('What is the capital of France?');console.log(response.content);

---

🚀 Quick Start

Installation

npm install @targetly-labs/flowllm

Basic Usage

import{defineAgent,openai}from'@targetly-labs/flowllm';// 1. Define your agentconstagent=defineAgent({provider: openai('gpt-4o'),systemPrompt: 'You are a helpful assistant.',temperature: 0.7,});// 2. Execute a single queryconstresponse=awaitagent.execute('Tell me a joke');console.log(response.content);// 3. Or stream responsesconststream=awaitagent.stream('Write a poem about TypeScript');forawait(constchunkofstream){process.stdout.write(chunk.content||'');}

Switch Providers Easily

import{openai,anthropic,gemini}from'@targetly-labs/flowllm/providers';// OpenAIconstgptAgent=defineAgent({provider: openai('gpt-4o'),systemPrompt: 'You are a helpful assistant.',});// AnthropicconstclaudeAgent=defineAgent({provider: anthropic('claude-3-5-sonnet-20240620'),systemPrompt: 'You are a helpful assistant.',});// Google GeminiconstgeminiAgent=defineAgent({provider: gemini('gemini-pro'),systemPrompt: 'You are a helpful assistant.',});

---

✨ Features

🤖 Agent Framework

• Goal-driven agent execution
• Multi-turn conversations with memory
• Tool selection and orchestration
• Streaming responses

🧠 LLM Client (Model-Agnostic)

• Single API for multiple providers (OpenAI, Anthropic, Gemini)
• Automatic retries and error handling
• Built-in cost tracking
• Token counting and management

🔌 MCP Integration (Coming Soon)

• Native MCP protocol support
• Automatic tool discovery from MCP servers
• Type-safe tool calls
• Works with any MCP server

💬 Conversation & Memory

• Short-term conversation memory
• Token window management
• Custom memory strategies
• Context persistence

⚡ Streaming & Real-time

• Token-by-token streaming
• Server-sent events support
• Progress updates
• UI-friendly streaming APIs

🛠️ Tool / Function Calling

• Type-safe tool definitions
• Schema validation with Zod
• Retry and fallback handling
• Custom function support

📊 Production Features

• Automatic retry logic for transient errors
• Cost and token tracking (per request/session)
• Request/response middleware
• Structured logging with Pino
• Performance monitoring

---

📚 Examples

Basic Conversation

import{defineAgent,openai}from'@targetly-labs/flowllm';constagent=defineAgent({provider: openai('gpt-4o'),systemPrompt: 'You are a helpful coding assistant.',});constresponse=awaitagent.execute('How do I reverse a string in JavaScript?');console.log(response.content);

Streaming Responses

constagent=defineAgent({provider: openai('gpt-4o'),systemPrompt: 'You are a creative writer.',});conststream=awaitagent.stream('Write a short story about a robot');forawait(constchunkofstream){process.stdout.write(chunk.content||'');}

Custom Tools

import{defineAgent,defineTool,openai}from'@targetly-labs/flowllm';constweatherTool=defineTool({name: 'get_weather',description: 'Get current weather for a location',parameters: {type: 'object',properties: {location: {type: 'string',description: 'City name',},},required: ['location'],},execute: async({ location })=>{// Your weather API logic herereturn{temp: 72,condition: 'sunny', location };},});constagent=defineAgent({provider: openai('gpt-4o'),tools: [weatherTool],systemPrompt: 'You are a helpful weather assistant.',});constresponse=awaitagent.execute('What\'s the weather in Tokyo?');console.log(response.content);

Multi-Provider Support

import{defineAgent,openai,anthropic}from'@targetly-labs/flowllm';// Create agents with different providersconstagents={gpt: defineAgent({provider: openai('gpt-4o'),systemPrompt: 'You are GPT-4.',}),claude: defineAgent({provider: anthropic('claude-3-5-sonnet-20240620'),systemPrompt: 'You are Claude.',}),};// Use the right agent for the jobconsttechnicalResponse=awaitagents.claude.execute('Explain async/await');constcreativeResponse=awaitagents.gpt.execute('Write a haiku');

---

🏗️ Architecture

FlowLLM uses a layered architecture for maximum flexibility and maintainability:

┌─────────────────────────────────────────────┐
│ FlowLLM SDK │
│ ┌─────────────────────────────────────┐ │
│ │ Agent Framework │ │
│ │ - Conversation management │ │
│ │ - Tool orchestration │ │
│ │ - Memory handling │ │
│ └─────────────────────────────────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ LLM │ │ MCP │ │ Tools │ │
│ │ Client │ │Connector │ │ System │ │
│ └──────────┘ └──────────┘ └──────────┘ │
└────┬──────────────┬──────────────┬─────────┘
│ │ │
▼ ▼ ▼
┌─────────┐ ┌────────────┐ ┌──────────┐
│ OpenAI │ │ MCP Server │ │ Custom │
│Anthropic│ │ (Future) │ │ Function │
│ Gemini │ │ │ │ Tools │
└─────────┘ └────────────┘ └──────────┘

Core Components

• Agent: Orchestrates conversation flow, memory, and tool execution
• LLMClient: Unified interface for all LLM providers
• ToolRegistry: Manages and executes custom tools and MCP tools
• Memory: Handles conversation history and token management
• CostTracker: Tracks token usage and costs across requests
• RetryHandler: Implements exponential backoff for transient errors

For detailed architecture diagrams and data flows, see ARCHITECTURE.md.

---

🎯 Use Cases

💼 AI-Powered SaaS Features

Build AI features into your product with ease.

constsupportAgent=defineAgent({provider: openai('gpt-4o'),systemPrompt: 'Help customers with their support tickets.',});

🤖 Autonomous Agents

Build agents that can take actions and make decisions.

constcodeReviewAgent=defineAgent({provider: anthropic('claude-3-5-sonnet-20240620'),systemPrompt: 'Review code and suggest improvements.',});

💬 Conversational Apps

Build chatbots, Discord bots, Slack bots with LLM capabilities.

constdiscordBot=defineAgent({provider: gemini('gemini-pro'),systemPrompt: 'Helpful Discord bot.',});

🔧 Internal Tools

Build internal AI assistants for your team.

constopsAgent=defineAgent({provider: openai('gpt-4o'),systemPrompt: 'Help with DevOps tasks and incident response.',});

---

📖 Documentation

• 🎓 Getting Started - Complete setup guide
• 🏗️ Architecture - Technical architecture details
• 🚀 Implementation Guide - Advanced patterns
• 🔐 Security - Security best practices
• 🤝 Contributing - Contribution guidelines
• 📝 Changelog - Version history

---

🛣️ Roadmap

✅ Phase 1 – Core SDK (Current)

• Core agent framework
• Multi-provider LLM client (OpenAI, Anthropic, Gemini)
• Conversation memory
• Custom tool calling
• Streaming responses
• Error handling & retries
• Cost tracking
• TypeScript support

🔄 Phase 2 – MCP Integration (Next)

• Native MCP protocol support
• Automatic tool discovery
• Type-safe MCP tool calls
• MCP server integration examples

📅 Phase 3 – Advanced Features

• Advanced memory systems (long-term, user profiles)
• Multi-agent orchestration
• Prompt versioning
• Execution tracing and analytics

---

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

# Clone the repository
git clone https://github.com/targetly-labs/flowllm.git
# Install dependenciescd flowllm
npm install
# Run tests
npm test# Build the project
npm run build

---

📄 License

MIT License - See LICENSE for details

---

🌟 Show Your Support

If you find FlowLLM useful, please consider:

• ⭐ Starring the repository
• 🐛 Reporting bugs and issues
• 💡 Suggesting new features
• 🤝 Contributing code
• 📢 Sharing with others

---

🔗 Links

• NPM Package: @targetly-labs/flowllm
• GitHub: targetly-labs/flowllm
• Issues: Report a bug
• Documentation: Full Docs

---

Built with ❤️ by the Targetly Labs team