PyMidscene

Python SDK for Midscene.js - AI-powered UI automation using natural language

Features | Installation | Quick Start | Documentation | 中文文档

---

What is PyMidscene?

PyMidscene is a Python port of Midscene.js - an AI-powered UI automation framework. It allows you to control web browsers using natural language instead of CSS selectors or XPath.

No more fragile selectors! Just describe what you want to click, type, or extract:

# Instead of: page.click("#submit-btn-primary")awaitagent.ai_click("the blue Submit button")
# Instead of: page.fill("input[name='email']", "test@example.com") awaitagent.ai_input("email input field", "test@example.com")
# Extract structured data with natural languageresult=awaitagent.ai_query({
"title": "the page title",
"price": "the product price as a number"
})

Features

• Natural Language Automation - Describe elements in plain English/Chinese, no selectors needed
• Multi-Model Support - Works with Doubao, Qwen, GPT-4V, Claude, and other vision LLMs
• Playwright Integration - Seamless integration with Playwright for web automation
• Android Integration - Control real devices over ADB with pymidscene[android] (see pymidscene/android/README.md)
• iOS Integration - Drive iPhones / simulators through WebDriverAgent (see pymidscene/ios/README.md)
• XPath Caching - Smart caching system compatible with Midscene.js format
• Visual Reports - Generate beautiful HTML reports for debugging and sharing
• Type-Safe - Full type hints for excellent IDE support

Installation

pip install pymidscene
# Install Playwright browsers
playwright install chromium

Or with Poetry:

poetry add pymidscene
playwright install chromium

Quick Start

1. Set up your API key

# For Doubao (recommended for Chinese users)export MIDSCENE_MODEL_NAME="doubao-seed-1-6-251015"export MIDSCENE_MODEL_API_KEY="your-api-key"export MIDSCENE_MODEL_BASE_URL="https://ark.cn-beijing.volces.com/api/v3"export MIDSCENE_MODEL_FAMILY="doubao-vision"# For Qwenexport MIDSCENE_MODEL_NAME="qwen-vl-max"export MIDSCENE_MODEL_API_KEY="your-api-key"export MIDSCENE_MODEL_BASE_URL="https://dashscope.aliyuncs.com/compatible-mode/v1"export MIDSCENE_MODEL_FAMILY="qwen2.5-vl"

2. Write your automation script

importasyncioimportosfromplaywright.async_apiimportasync_playwrightfrompymidsceneimportPlaywrightAgentasyncdefmain():
# Configure model (or use environment variables)os.environ["MIDSCENE_MODEL_NAME"] ="doubao-seed-1-6-251015"os.environ["MIDSCENE_MODEL_API_KEY"] ="your-api-key"os.environ["MIDSCENE_MODEL_BASE_URL"] ="https://ark.cn-beijing.volces.com/api/v3"os.environ["MIDSCENE_MODEL_FAMILY"] ="doubao-vision"asyncwithasync_playwright() asp:
browser=awaitp.chromium.launch(headless=False)
page=awaitbrowser.new_page()
# Create agent with optional cachingagent=PlaywrightAgent(page, cache_id="my_task")
# Navigate to pageawaitpage.goto("https://www.example.com")
# Use natural language to interactawaitagent.ai_click("the search box")
awaitagent.ai_input("search input", "Python automation")
awaitagent.ai_click("search button")
# Extract dataresult=awaitagent.ai_query({
"results_count": "number of search results",
"first_title": "title of the first result"
})
print(f"Found: {result}")
# Assert page stateawaitagent.ai_assert("search results are displayed")
# Generate visual reportreport_path=agent.finish()
print(f"Report saved to: {report_path}")
awaitbrowser.close()
if__name__=="__main__":
asyncio.run(main())

CLI

Run automation written as YAML scripts — no Python needed — with the pymidscene command (installed with the package):

pymidscene ./script.yaml # run a single script
pymidscene ./scripts/ # run every *.yaml in a directory
pymidscene --files a.yaml b.yaml --concurrent 2
pymidscene --config ./suite.yaml --continue-on-error

A script declares one platform target (web / android / ios) plus tasks:

web:
url: https://www.bing.comtasks:
- name: searchflow:
- aiInput: the search boxvalue: midscene
- aiKeyboardPress: Enter
- aiWaitFor: search results are visibletimeout: 15000
- aiQuery: "{ titles: string[] }"name: results
- aiAssert: a list of results is shown

See docs/CLI.md for the full flag and flow-item reference, and examples/cli/ for ready-to-run scripts.

Documentation

Core API

Method	Description
ai_click(description)	Click an element described in natural language
ai_input(description, text)	Type text into an input field
ai_locate(description)	Locate an element and return its coordinates
ai_query(schema)	Extract structured data from the page
ai_assert(assertion)	Assert that a condition is true
ai_action(task)	Execute a complex task with AI planning loop (plan-execute-replan)
ai_wait_for(assertion, timeout)	Wait until a page condition is met (polling)
ai_scroll(direction, distance)	Scroll the page with AI assistance
finish()	Generate HTML report and return the path

Supported Models

Model	Family	Provider
doubao-seed-1-6-251015	doubao-vision	Bytedance/Volcano
qwen-vl-max	qwen2.5-vl	Alibaba
gpt-4-vision-preview	openai	OpenAI
claude-3-opus	claude	Anthropic

Cache System

PyMidscene uses XPath-based caching compatible with Midscene.js:

# midscene_run/cache/my_task.cache.yamlmidsceneVersion: 1.0.0cacheId: my_taskcaches:
- type: locateprompt: the login buttoncache:
xpaths:
- /html/body/div[1]/button[1]

This means:

• Cache files are interchangeable between JS and Python versions
• XPath-based caching works across different window sizes
• Cache invalidation happens automatically when elements move

Examples

Check out the examples/ directory:

• basic_usage.py - Getting started
• login_demo.py / login_demo.html - Login automation with a visual report
• android_basic.py / ios_basic.py - Mobile automation
• cli/ - YAML scripts for the pymidscene CLI (web / android / suite)

Project Structure

pymidscene/
├── pymidscene/ # Main package
│ ├── core/ # Core automation logic
│ │ ├── agent/ # Agent implementation
│ │ ├── ai_model/ # AI model integration
│ │ └── dump.py # Report generation
│ ├── web_integration/ # Browser integrations
│ │ └── playwright/ # Playwright adapter
│ └── shared/ # Shared utilities
├── examples/ # Usage examples
├── tests/ # Test suite
└── docs/ # Documentation

Related Projects

This is the Python implementation of Midscene.js.

• Midscene.js - Original JavaScript version
• Official Documentation
• Awesome Midscene - Community projects

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

# Development setup
git clone https://github.com/AIPythoner/pymidscene.git
cd pymidscene
pip install -e ".[dev]"# Run tests
pytest
# Format code
black pymidscene tests

License

MIT License - see LICENSE file for details.

Acknowledgments

• Midscene.js - The original JavaScript framework by Bytedance
• Playwright - Browser automation library

---

Made with love by the PyMidscene community