AutoTask - Local Workflow Capture Tool

A macOS desktop application for capturing professional workflows with AI-powered semantic understanding. Record screen video + OS-level interactions (mouse, keyboard, apps) to create documentation and build a searchable knowledge base.

🎯 Project Vision

AutoTask addresses two critical needs:

1. Documentation & Knowledge Base: Record workflows and query them intelligently with AI
2. Future Automation: Learn from demonstrations to automate repetitive tasks

Inspired by ShowUI-Aloha, AutoTask captures not just what you do, but why you do it through AI-generated semantic traces.

🏗️ Architecture

• Frontend: React + TypeScript + Tailwind CSS
• Backend: Tauri (Rust) for native macOS performance
• Database: SQLite with WAL mode
• Recording: ScreenCaptureKit API + Accessibility API
• AI: Multi-provider support (OpenAI, Anthropic, Google)
• RAG: Multi-modal embeddings with hybrid search

Key Innovation: Semantic Traces

Unlike traditional screen recorders, AutoTask generates semantic traces with AI:

• Observation: What's visible on screen
• Think: Why this action is being taken (reasoning)
• Action: What was done
• Expectation: What should happen next

This enables both human-readable documentation and future workflow automation.

📋 Current Status

Phase 1: Recorder Stage (In Progress)

✅ Completed

• Project structure set up (Tauri + React)
• Database schema with semantic_traces table
• SQLite initialization with migrations
• Permission checking framework
• Basic UI skeleton
• Module structure for recording components

🚧 In Progress

• ScreenCaptureKit integration
• Accessibility API event tracking
• Video encoding pipeline

📝 Next Steps

1. Complete screen capture implementation
2. Implement OS-level interaction tracking
3. Build recording UI with start/stop controls
4. Test recording with real workflows

🚀 Development Setup

Prerequisites

All prerequisites should already be installed:

• Node.js v25.6.1
• npm 11.9.0
• Rust 1.93.1
• Cargo 1.93.1

Running the App

# Development mode (starts frontend + backend)
npm run tauri dev
# Build for production
npm run tauri build

Project Structure

autotask/
├── src/ # React frontend
│ ├── App.tsx # Main app component
│ ├── main.tsx # Entry point
│ └── components/ # UI components (coming soon)
├── src-tauri/ # Rust backend
│ ├── src/
│ │ ├── main.rs # Tauri entry point
│ │ ├── storage/
│ │ │ └── db.rs # Database layer
│ │ └── recording/
│ │ ├── permissions.rs # Permission checking
│ │ ├── capture.rs # Screen recording
│ │ └── interaction_tracker.rs # OS event tracking
│ └── Cargo.toml # Rust dependencies
├── Initial Plan/ # Implementation plan
│ └── implementation-plan.md # Comprehensive technical plan
└── README.md # This file

🔒 Required Permissions

AutoTask requires two macOS permissions:

1. Screen Recording (System Settings > Privacy & Security > Screen Recording)

   • Required for video capture
   • Checked automatically on app start

2. Accessibility (System Settings > Privacy & Security > Accessibility)

   • Required for mouse/keyboard tracking
   • Must be granted manually by user
   • Used to capture semantic workflow context

📊 Database Schema

The SQLite database includes these key tables:

• captures: Video recordings with metadata
• semantic_traces: AI-generated step-by-step workflow understanding
• projects: Organizational structure
• tags: Categorization system
• ai_analysis: LLM-generated summaries and insights
• transcripts: Audio transcription (optional)
• embeddings: Multi-modal vectors for RAG search
• token_usage: AI cost tracking

🗺️ Roadmap

Phase 1: Recorder (Current)

• Screen capture with ScreenCaptureKit
• OS-level interaction tracking
• Video + interaction log storage
• Basic playback UI

Phase 2: Learner (Next)

• Semantic trace generation with GPT-4o Vision
• Audio transcription with Deepgram
• Workflow summarization
• Trace viewer UI

Phase 3: RAG System

• Multi-modal embeddings
• Hybrid search (vector + full-text)
• Chat interface for querying workflows
• Citation linking to video timestamps

Phase 4: Production Polish

• Project/tag organization
• Multi-provider AI support
• Token usage dashboard
• Cost management

Phase 5+: Automation

• Workflow replay
• Task adaptation to variants
• Planner + Actor + Executor stages

💡 Key Features (Planned)

• 🎥 Screen + Interaction Recording: Complete workflow capture
• 🧠 AI Semantic Understanding: Know why actions happen
• 🔍 RAG-Powered Search: "How did I create that pivot table?"
• 📚 Documentation Export: Generate tutorials from recordings
• 🤖 Future Automation: Replay learned workflows
• 💰 Cost Tracking: Monitor AI API usage across providers
• 🔐 Privacy-First: All data stored locally, no cloud uploads

📖 Documentation

See Initial Plan/implementation-plan.md for:

• Detailed technical architecture
• ShowUI-Aloha integration strategy
• Database schema documentation
• AI processing pipeline
• Security and privacy considerations
• Cost estimates and risk mitigations

🛠️ Technology Stack

Frontend:

• React 18.2
• TypeScript 5.1
• Tailwind CSS 3.4
• Zustand (state management)
• React Player (video playback)

Backend:

• Tauri 2.x
• Rust 1.93
• SQLite + sqlx
• ScreenCaptureKit (macOS screen recording)
• Accessibility API (event tracking)

AI Services:

• OpenAI (GPT-4o, GPT-4o Vision, embeddings)
• Anthropic Claude (alternative)
• Google Gemini (future)
• Deepgram (transcription)

📝 License

Private project - All rights reserved

🙏 Acknowledgments

Inspired by:

• ShowUI-Aloha - Human-taught computer-use agent
• Modern screen recording tools with semantic understanding