Transform complex, multi-stage workflows into reliable, resumable AI orchestration
hey! im a grad student building this framework for catering every type of user who work with LLMs. It all started, when LLMs were missing important pieces of context in long shot conversations, tasks like research writing, market analysis, code migration were stunted. to tackle this, i made this framework, to make LLMs take notes, plan in advance, maintain a tracker (blueprint). i hope to make this production ready one day :D
this framework enables llms to execute sophisticated, long-duration projects through intelligent task orchestration. By combining built-in connectors, human oversight, and multi-provider llm support, weaver bridges the gap between simple ai interactions and complex, real-world automation.
🔄 Never Lose Progress Again
- Stateful execution with SQLite persistence
- Resume interrupted workflows from any point
- Comprehensive error handling and retry logic
🤝 Human-AI Collaboration
- Review and edit AI-generated task plans
- Approve results before proceeding to next steps
- Export/import workflows via intuitive CSV interface
🛡️ Production-Ready Architecture
- Local-first security - your data never leaves your machine
- Multi-provider LLM support through litellm integration
- Modular, extensible design for custom workflows
📊 Intelligent Task Management
- Automatic dependency resolution
- Cost tracking across all LLM providers
- Detailed execution logs and timestamps
pip install python-weaverfromweaver.projectimportProject# Initialize a new projectproject=Project(
project_name="market_analysis",
project_goal="Generate a comprehensive market analysis report for renewable energy trends"
)
# Ingest your data sourcesproject.ingest([
"data/market_report.pdf",
"data/competitor_analysis.txt",
"https://example.com/industry-trends"
])
# Generate an AI-powered execution planproject.plan()
# 📝 Review and edit the generated blueprint.csv# Execute with human oversightproject.run(human_feedback=True)# Initialize project
weaver init my_project "Analyze customer feedback and generate improvement recommendations"# Add data sources
weaver ingest my_project reports/*.pdf feedback_data.csv
# Generate execution plan
weaver plan my_project
# Execute with human feedback
weaver run my_project
# Or run fully automated
weaver run my_project --no-human-feedback --steps 5
Weaver's architecture ensures reliability and transparency:
- Blueprint Database: SQLite-backed task tracking with complete state persistence
- Agent System: Intelligent task execution with automatic retry and error handling
- Connector Framework: Extensible ingestion system for PDFs, URLs, and custom sources
- Multi-Provider Support: Seamless integration with 10+ LLM providers via litellm
- Automated Decomposition: Break complex goals into manageable, sequential tasks
- Dependency Management: Automatic resolution of task dependencies and prerequisites
- Resource Optimization: Smart LLM selection and load balancing across providers
- Persistent State: All progress saved to local SQLite database
- Resume Capability: Pick up exactly where you left off after interruptions
- Audit Trail: Complete execution history with timestamps and cost tracking
- Plan Review: Edit AI-generated task plans before execution
- Result Approval: Review and modify outputs at each step
- CSV Interface: Intuitive spreadsheet-based workflow management
- OpenAI: GPT-4, GPT-4 Turbo, GPT-3.5
- Anthropic: Claude 3 Haiku, Sonnet, Opus
- Google: Gemini 1.5 Pro, Flash
- Azure OpenAI: Enterprise-grade OpenAI models
- Local Models: Ollama, LM Studio integration
- Custom Providers: Easy integration via litellm
- Python 3.8+
- At least one LLM provider API key
# Install from PyPI
pip install python-weaver
# Verify installation
weaver checkSet your API keys as environment variables:
# OpenAIexport OPENAI_API_KEY="sk-..."# Anthropicexport ANTHROPIC_API_KEY="sk-ant-..."# Googleexport GOOGLE_API_KEY="AIza..."# Azure OpenAIexport AZURE_API_KEY="..."export AZURE_API_BASE="https://your-resource.openai.azure.com/"Create a weaver.toml file for advanced settings:
[litellm]
set_verbose = truedrop_params = true
[api_bases]
openai = "https://api.openai.com/v1"anthropic = "https://api.anthropic.com"
[models]
default_orchestrator = "gpt-4o-mini"Extend Weaver with custom data sources:
fromweaver.connectors.base_connectorimportBaseConnectorclassDatabaseConnector(BaseConnector):
defingest(self, source: str) ->str:
# Your custom ingestion logicreturnextracted_text# Register and useproject.register_connector("db", DatabaseConnector())
project.ingest(["db://my_database/table"])fromweaver.blueprintimportBlueprintfromweaver.agentimportAgent# Direct blueprint manipulationblueprint=Blueprint("my_project.db")
task_id=blueprint.add_task(
task_name="Data Analysis",
llm_config_key="claude-3-sonnet",
prompt_template="Analyze the following data: {data}",
dependencies="1,2"
)
# Execute specific tasksagent=Agent(blueprint, "My project goal")
agent.execute_task(task_id)# Configure cost-aware executionproject=Project("cost_optimized", "Generate report")
# Use different models for different task typesproject.blueprint.add_task(
task_name="Quick Summary",
llm_config_key="gpt-4o-mini", # Cost-effective for simple tasksprompt_template="Summarize: {content}"
)
project.blueprint.add_task(
task_name="Deep Analysis", llm_config_key="gpt-4o", # High-quality for complex analysisprompt_template="Provide detailed analysis: {content}"
)project=Project("market_research", "Comprehensive renewable energy market analysis")
project.ingest([
"reports/iea_renewable_2024.pdf",
"data/market_data.csv",
"https://irena.org/statistics"
])
project.plan()
project.run()project=Project("literature_review", "Systematic review of AI ethics literature")
project.ingest([
"papers/*.pdf",
"abstracts.txt",
"https://arxiv.org/search/cs.AI"
])
project.plan()
project.run(steps=3) # Process in batchesproject=Project("quarterly_report", "Generate Q4 business performance report")
project.ingest([
"financials/q4_data.xlsx",
"metrics/kpi_dashboard.csv",
"feedback/customer_surveys.txt"
])
project.plan()
project.run(human_feedback=True)# Clone repository
git clone https://github.com/python-weaver/python-weaver.git
cd python-weaver
# Install in development mode
pip install -e .[dev]
# Run tests
pytest
# Run with coverage
pytest --cov=weaver --cov-report=htmlpython-weaver/
├── weaver/
│ ├── __init__.py
│ ├── project.py # Main user interface
│ ├── blueprint.py # SQLite task management
│ ├── agent.py # LLM task execution
│ ├── config.py # Provider configurations
│ ├── cli.py # Command-line interface
│ └── connectors/ # Data ingestion modules
├── tests/ # Comprehensive test suite
├── examples/ # Usage examples
└── docs/ # Documentation
could use all the help i can get. i'll make a contributing guide soon to help you easily navigate and develop!
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
- 🔄 Async/Await Patterns
- 📊 Entirely customizable connectors
- 🌐 Web Interface: Browser-based project management dashboard
- Documentation and Official Website
- DB Optimization
- Better Memory Management
- RAG For long contexts
- Security Concerns addressing
Coming Soon!
This project is licensed under the MIT License - see the LICENSE file for details.