Turn your workday into structured workflows agents can execute.
Computer-use agents work every task out from scratch, even one you've done a hundred times. And between tasks, your agent has no idea what you've been doing all day, so it starts every conversation blind.
activity-frames fixes both. It records your screen locally and compiles what it sees into structured activity frames: bounded, deterministic records of the tasks you actually did. The recurring ones become workflows an agent can execute instead of working out again. So your repetitive computer tasks get done cheaper (running a compiled workflow costs almost no tokens) and more reliable (the same steps, grounded the same way every time, instead of guessing from a screenshot) - and everything else becomes context your agent can use.
pip install activity-frames
aframes record # start capturing (local, audio off by default)
aframes context # your last 2 hours, agent-readyWant to go deeper?docs/mcp.md covers the six MCP tools, docs/python.md the Python API, docs/cli.md every flag, docs/troubleshooting.md the errors you might actually hit. AI agent integrating this? AGENTS.md is written for you.
Capture stores instants: thousands of snapshot rows a day, each one saying "at 22:53:05, Chrome showed linkedin.com/in/...". Useless to reason over.
activity-frames compiles those instants into activity frames:
- id: f-0007app: Google Chromesite: linkedin.comstart: "20:24:04"end: "20:42:11"duration_min: 18.0pages:
- {kind: people_search, entity: "cto berlin", count: 2}
- {kind: profile, entity: john-doe}
- {kind: company, entity: acme-ai}input: {keys: 214, clicks: 31}evidence: {frame_ids: "99871..100147"}And into a compact context block for any system prompt:
USER ACTIVITY (2026-08-10, local time; measured from screen capture, no interpretation):
coverage: 10:29-22:55, 529 active min, 15 apps
away: 13:02-14:39 (97m)
- 10:54-11:23 Google Chrome/app.cal.com (28.7m): dashboard:video x59
- 12:49-12:52 cmux (2.7m): ⠂ Build nocta-recorder ground-up screen recorder
- 12:52-13:02 Google Chrome/calendar.google.com (9.7m): calendar x20
- 20:53-20:56 Google Chrome/github.com (3.3m): user:bethvourc x5; repo:nossa-y/activity-frames x2
- 20:56-21:01 Google Chrome/hackernoon.com (5.2m): page:i-compiled-55-days-of-screen-activity-into-episodic-memory-for-my-ai-agent x12
That is a real day of mine (excerpt; the full block has 44 frames). It compiles to 1,371 tokens; the raw snapshot rows for the same day are 247,563. Compiles in under a second, costs zero tokens, no LLM in the loop.
Drop it into a prompt and your agent knows your day - ask "what was that article I had open around 9pm?" and it answers in one line, off the block alone:
Real exchange, MCP disabled - the answer comes from the compiled block only.
Computer-use agents re-derive every task from scratch - screenshot, reason, act, repeat - even for a workflow they've run a hundred times. That re-derivation is where the token cost goes, and it's waste: the workflow hasn't changed.
Because activity-frames compiles recurring activity deterministically, a task you've demonstrated becomes an executable script:
aframes steps --find "message john doe"{
"steps": [
{"t": "20:24:09", "op": "focus", "target": "Google Chrome · LinkedIn", "n": 1},
{"t": "20:24:14", "op": "click", "target": "Search", "role": "TextField", "url": "https://www.linkedin.com/feed/", "n": 2},
{"t": "20:24:16", "op": "type", "chars": 8, "text": "john doe", "n": 3},
{"t": "20:24:21", "op": "click", "target": "John Doe", "role": "Link", "url": "https://www.linkedin.com/search/results/people/", "n": 4},
{"t": "20:24:29", "op": "click", "target": "Message", "role": "Button", "url": "https://www.linkedin.com/in/john-doe/", "n": 5},
{"t": "20:24:35", "op": "type", "chars": 71, "text": "hey, loved your post on agent memory - open to a quick chat next week?", "n": 6}
],
"step_count": 6,
"unresolved_clicks": 0
}That's the replay view of a demonstrated run - ordered clicks grounded by element name and role, typed runs, focus changes. An agent repeats the task instead of re-deriving it: fill the slots with new values (a different name, the same steps) and execute. On the happy path it replays at zero model calls; anything unexpected halts and asks instead of guessing.
We measured how much agents overpay to re-derive workflows they've already performed - the Routine Overhead Ratio - on weeks of real activity, replicated it on a public web-task dataset, and built a deterministic executor that replays a compiled workflow in a real browser. Instrument, measurements, and executor: research/.
Passively-captured activity becomes deterministic action - and the cheapest computer task is the one an agent never reasons through twice.
Agent memory today means conversation memory: what you told the model. What you actually did is the missing half - and the hard part is representing it without lying.
activity-frames enforces a two-tier contract (SPEC.md):
- Tier 1, measured (this package): everything is derivable by deterministic code from capture data - sessions, durations, typed page entities, input volume, coverage gaps. No interpretation, no intent labels. Same input, same output, every time.
- Tier 2, inferred (optional extension): tools that add interpretation must namespace it, tag confidence (
high | medium | speculative), and link evidence. Facts and guesses can never silently mix.
Every frame carries evidence pointers back to raw capture rows. Every document declares its blind spots. What the system did not see, it says it did not see.
# Claude Code
claude mcp add activity-frames -- aframes mcpAny MCP client works: command aframes, args ["mcp"]. Six tools: get_context, get_activity, get_steps (expand one activity frame into its ordered click-by-click script - the replay view of a demonstrated run, so an agent can repeat the task instead of re-deriving it), get_day_summary, get_patterns (repetitive-workflow detection: repeated clicks, action sequences, URL patterns, app-switching loops, daily habits), and get_communications (email/messaging surfaces with the window titles seen on each — for many clients the title carries the subject or conversation name; a client that doesn't title its windows with the conversation leaves only its presence to report. Titles only, measured tier: message bodies are never read).
fromactivity_framesimportActivityLoglog=ActivityLog()
doc=log.day() # today, structureddoc=log.recent(hours=2) # last 2 hoursprint(log.context(hours=2)) # paste-ready context block- Local only. Capture, storage, and compilation all happen on your machine. Nothing is uploaded anywhere, ever.
- Read-only compilation. The compiler opens the capture database read-only.
- Content opt-in at the output. Compiled documents carry input counts by default; typed-text content appears only if you explicitly pass
--include-text(this also gates the repeated-text pattern detector). Be clear about the boundary: the capture database itself does store what the recorder sees, locally, so protect it like any sensitive file (FileVault, permissions). - Audio off by default.
aframes record --audioto opt in. - No LLM in the compile path. Compilation is plain code, so no language model, local or remote, is involved in producing memory. The capture engine does run on-device OCR to read what is on screen; that stays on your machine.
- You choose what leaves, when you paste a context block into an agent. Note that window titles and page entities originate from your screen and can contain third-party text; agents should treat them as data, not instructions.
capture engine compiler (this package) your agent
------------------ --------------------------- -----------------
screen snapshots --> sessionize (dwell, gaps, --> MCP tools /
accessibility tree flicker merge) context blocks /
input events entity typing (25+ sites) JSON, YAML, md
(local SQLite) enrichment, patterns
The default capture engine is nocta-recorder: aframes record provisions a pinned, MIT-licensed build, verifies its published sha256 before first run, and manages it for you (see ACKNOWLEDGMENTS.md); a nocta-recorder binary already on your PATH is used as-is (bring your own build). Already running your own recorder? Point $AFRAMES_DB at any capture database with compatible frames / ui_events / elements tables and skip aframes record entirely.
aframes record # start capture (--stop / --status / --audio)
aframes today # today's frames (YAML*)
aframes day 2026-07-03 -f json # any day, JSON
aframes context --hours 3 # agent context block
aframes apps # per-app time ledger
aframes patterns --days 7 # repetitive workflow detection
aframes comms --hours 24 # email/messaging surfaces + titles seen
aframes steps --frame f-0002 # one frame's click-by-click script (or --find "task query")
aframes mcp # MCP stdio server*YAML output uses PyYAML (pip install "activity-frames[yaml]"); without it the CLI falls back to JSON.
Run aframes <cmd> --help for the full flag set (--db, --include-text, --layout, ...).
| AGENTS.md | Integration guide for AI agents (install, MCP tools, replay loop, cautions) |
| SPEC.md | The schema contract: measured vs inferred tiers |
| docs/mcp.md | MCP setup and the six tools in detail |
| docs/python.md | Python API reference |
| docs/cli.md | Every command and flag |
| docs/troubleshooting.md | Keyed by the actual error you see |
| research/ | The cost instrument, measurements, and replay executor behind the paper |
| llms.txt | Compressed repo map for LLM context windows |
v0.2. Developed and tested on macOS (Apple Silicon); an Intel macOS engine build is published but less exercised - reports welcome. No prebuilt Linux engine yet: on Linux, run your own recorder and point $AFRAMES_DB at its database (the compiler itself is tested on Linux in CI). Entity parsers cover LinkedIn, GitHub, GitLab, Google (Search/Docs/Gmail/Maps/Meet/Calendar), YouTube, X, Instagram, Reddit, Luma, Partiful, Product Hunt, Vercel, Supabase, Stripe (dashboard), Discord, Slack, Notion, Figma, Linear, Stack Overflow, Calendly, Crunchbase, Atlassian (Jira/Confluence), ChatGPT/Claude, localhost; unknown sites fall back to a generic page reference - always total, never lossy. Issues and parser PRs welcome.
Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay
Nossa Iyamu, arXiv 2026
@misc{iyamu2026activityframes,
title={Activity Frames: Deterministic Screen-Activity Compilation for Agent Memory and Replay},
author={Nossa Iyamu},
year={2026},
eprint={2608.05784},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2608.05784}
}Built by Nossa Iyamu, maker of Nocta. MIT.