AI-powered automation framework for Web and Android with natural language-driven UI operations - Java version
Midscene Java is a revolutionary AI-powered automation framework designed for UI automation operations on Web and Android platforms. It is the Java implementation of Midscene Python, inheriting its core philosophy: making automation as simple as speaking.
- Natural Language Operations - Describe operation intentions in everyday language, and AI will automatically understand and execute them
- Intelligent Element Locating - Multi-strategy fusion, automatically selects the optimal positioning method, adapts to page changes
- Structured Data Extraction - Use natural language to extract complex structured data
- Intelligent Assertion Verification - Describe verification conditions in natural language, AI automatically judges
- Multi-Platform Support - Unified interface supports Web and Android platforms
- Visual Debugging - Detailed execution screenshots and decision process recording
- Code Optimization and Refactoring - Systematically refactored for more modular and maintainable code
midscene-java/
├── packages/
│ ├── core/ # Core module, providing Agent and AI engine
│ ├── web/ # Web automation module
│ │ ├── playwright/ # Playwright implementation
│ │ └── selenium/ # Selenium implementation
│ ├── android/ # Android automation module
│ ├── cli/ # Command line tool
│ ├── examples/ # Example code
│ ├── playground/ # Development testing environment
│ └── tests/ # Test cases
├── apps/ # Application examples
├── docs/ # Project documentation and optimization plans
└── wiki/ # Project wiki documentation
- Java 17+
- Maven 3.6+ or Gradle 7.0+
- Browser (Chrome/Firefox/Edge, for Web automation)
- AI model API Key (Choose one from OpenAI, Claude, Qwen, or Gemini)
Add Midscene Java dependencies to your pom.xml file:
<dependencies>
<!-- Core module -->
<dependency>
<groupId>com.midscene</groupId>
<artifactId>midscene-core</artifactId>
<version>0.1.1</version>
</dependency>
<!-- Web automation modules (choose as needed) -->
<dependency>
<groupId>com.midscene</groupId>
<artifactId>midscene-web-playwright</artifactId>
<version>0.1.1</version>
</dependency>
<dependency>
<groupId>com.midscene</groupId>
<artifactId>midscene-web-selenium</artifactId>
<version>0.1.1</version>
</dependency>
<!-- Android automation module (choose as needed) -->
<dependency>
<groupId>com.midscene</groupId>
<artifactId>midscene-android</artifactId>
<version>0.1.1</version>
</dependency>
</dependencies>Create an application.properties or application.yml file to configure the AI model:
# application.propertiesmidscene.ai.provider=openai
midscene.ai.model=gpt-4-vision-preview
midscene.ai.api-key=your_openai_api_key_herepackagecom.example;
importcom.midscene.core.Agent;
importcom.midscene.web.playwright.PlaywrightPage;
importcom.midscene.web.playwright.PlaywrightUIContextProvider;
importcom.microsoft.playwright.Playwright;
importcom.microsoft.playwright.Browser;
importcom.microsoft.playwright.Page;
publicclassSearchExample {
publicstaticvoidmain(String[] args) {
try (Playwrightplaywright = Playwright.create()) {
// Create browser instanceBrowserbrowser = playwright.chromium().launch();
Pagepage = browser.newPage();
// Create PlaywrightPage wrapperPlaywrightPageplaywrightPage = newPlaywrightPage(page);
// Create AgentAgentagent = newAgent(newPlaywrightUIContextProvider(playwrightPage));
// Navigate to websitepage.navigate("https://www.baidu.com");
// Use natural language for searchagent.aiAction("Type 'Java tutorial' in the search box");
agent.aiAction("Click the search button");
// Verify search resultsagent.aiAssert("The page displays search results for Java tutorials");
System.out.println("✅ Search operation completed!");
// Close browserbrowser.close();
}
}
}packagecom.example;
importcom.midscene.core.Agent;
importcom.midscene.web.playwright.PlaywrightPage;
importcom.midscene.web.playwright.PlaywrightUIContextProvider;
importcom.microsoft.playwright.Playwright;
importcom.microsoft.playwright.Browser;
importcom.microsoft.playwright.Page;
importjava.util.HashMap;
importjava.util.List;
importjava.util.Map;
publicclassExtractExample {
publicstaticvoidmain(String[] args) {
try (Playwrightplaywright = Playwright.create()) {
Browserbrowser = playwright.chromium().launch();
Pagepage = browser.newPage();
PlaywrightPageplaywrightPage = newPlaywrightPage(page);
Agentagent = newAgent(newPlaywrightUIContextProvider(playwrightPage));
// Visit news websitepage.navigate("https://news.example.com");
// Extract structured dataMap<String, Object> schema = newHashMap<>();
schema.put("articles", List.of(
Map.of(
"title", "News title",
"time", "Publish time",
"summary", "News summary"
)
));
Map<String, Object> newsData = agent.aiExtract(schema);
// Output resultsList<Map<String, String>> articles = (List<Map<String, String>>) newsData.get("articles");
for (Map<String, String> article : articles) {
System.out.println("📰 " + article.get("title"));
System.out.println("⏰ " + article.get("time"));
System.out.println("📄 " + article.get("summary") + "\n");
}
browser.close();
}
}
}packagecom.example;
importcom.midscene.core.Agent;
importcom.midscene.android.AndroidDevice;
importcom.midscene.android.AndroidUIContextProvider;
importjava.util.concurrent.CompletableFuture;
publicclassAndroidExample {
publicstaticvoidmain(String[] args) {
// Connect to Android deviceAndroidDevicedevice = newAndroidDevice();
CompletableFuture<Void> connectFuture = device.connect();
connectFuture.join(); // Wait for connection to completetry {
// Create AgentAgentagent = newAgent(newAndroidUIContextProvider(device));
// Launch applicationagent.aiAction("Launch the settings app");
// Perform operationsagent.aiAction("Tap on the Wi-Fi option");
agent.aiAssert("The Wi-Fi settings page is open");
System.out.println("✅ Android automation operation completed!");
} finally {
device.disconnect();
}
}
}- Project Overview - Chinese only
- Installation and Configuration - Chinese only
- Quick Start - Chinese only
- API Reference
- Example Code
- Frequently Asked Questions - Chinese only
- Core Concepts - Chinese only
- Platform Integration
| Feature | Traditional Automation Tools | Midscene Java |
|---|---|---|
| Learning Curve | Steep, requires learning complex APIs | Gentle, natural language driven |
| Code Readability | Obscure and hard to understand | Intuitive and easy to understand |
| Maintenance Cost | High, requires extensive modifications for page changes | Low, AI automatically adapts to changes |
| Element Locating | Manual selector writing | AI intelligent locating |
| Error Handling | Manual handling of various exceptions | AI automatic retry and recovery |
| Cross-Platform | Requires learning different tools | Unified interface |
| Code Quality | Varies by project | Systematically refactored, modular design |
We welcome all forms of contributions! Whether it's submitting bug reports, feature requests, documentation improvements, or code contributions.
- Fork this repository
- Create your feature branch (
git checkout -b feature/AmazingFeature) - Commit your changes (
git commit -m 'Add some AmazingFeature') - Push to the branch (
git push origin feature/AmazingFeature) - Create a Pull Request
# Clone the repository
git clone https://github.com/Master-Frank/midscene-java.git
cd midscene-java
# Build the project
mvn clean install
# Run tests
mvn test- Follow commit message conventions from Conventional Commits
- Add corresponding test cases for new features
- Add JavaDoc documentation for public APIs
- Keep code modular, avoid overly long methods
This project is licensed under the MIT License - see the LICENSE file for details.
Thanks to Midscene Project: https://github.com/web-infra-dev/midscene for inspiration and technical references
- GitHub: Master-Frank/midscene-java
- Issue Reporting: GitHub Issues
- Discussions: GitHub Discussions
⭐ If this project helps you, please give us a star!