Midscene Java

Midscene Java is an AI-powered automation SDK that allows you to control web browsers using natural language instructions. It integrates with Selenium WebDriver and Playwright to serve as an intelligent agent layer on top of your existing test automation framework.

Features

• Natural Language Control: "Search for 'Headphones' and click the first result."
• Advanced Interaction: Click, type, scroll, drag-and-drop, and more using simple natural language commands.
• Multimodal Understanding: Uses screenshots to understand page context (Visual Grounding).
• DOM Grounding: Automatically extracts a highly compressed JSON snapshot of the visible DOM, drastically reducing token usage and providing 100% accurate element coordinates.
• Smart Planning: Automatically plans, executes, and retries actions.
• Service Layer: Low-level AI capabilities for locating, extracting, and describing elements.
• Structured Planning & Assertions: Relies on structured XML tag responses for robust and predictable planning and assertions.
• Any AI Model: Built natively on top of LangChain4j, allowing you to use any supported model (OpenAI, Gemini, Claude, local Ollama, etc.) with custom configurations.
• YAML Script Support: Execute declarative test scripts defined in YAML.
• Framework Agnostic: Works seamlessly with Selenium and Playwright.
• Visual Reports: Generates detailed HTML reports with execution traces, screenshots, and reasoning.

Modules

• midscene-core: The brain of the agent. Contains Agent, Service, ScriptPlayer, and core logic.
• midscene-web: Adapters for browser automation tools (Selenium, Playwright).
• midscene-visualizer: Generates visual HTML reports from execution contexts.

Installation

Add the necessary dependencies to your project's pom.xml. To run the Agent and generate reports, you should add both midscene-web and midscene-visualizer:

<dependency>
<groupId>io.github.alstafeev</groupId>
<artifactId>midscene-web</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>io.github.alstafeev</groupId>
<artifactId>midscene-visualizer</artifactId>
<version>1.0.0</version>
</dependency>

Quick Start (Agent Mode)

Midscene Agent is the primary way to interact with your application. Because Midscene Java relies on standard LangChain4j, you can inject any ChatModel instance directly.

Warning

Do not configure JSON response formats (such as responseFormat("json_object") or ResponseFormat.JSON) on the ChatModel builder. Midscene uses XML tags for planning and assertions. Setting the response format to JSON will cause parsing errors.

Selenium Example

packagecom.midscene.web;
importcom.microsoft.playwright.Browser;
importcom.microsoft.playwright.Page;
importcom.microsoft.playwright.Playwright;
importcom.midscene.core.agent.Agent;
importcom.midscene.web.driver.PlaywrightDriver;
importcom.midscene.web.driver.SeleniumDriver;
importdev.langchain4j.model.chat.ChatModel;
importdev.langchain4j.model.googleai.GoogleAiGeminiChatModel;
importorg.junit.jupiter.api.Test;
importorg.openqa.selenium.chrome.ChromeDriver;
publicclassLocalTestingTest {
@TestpublicvoidlocalMidsceneSeleniumTest() {
ChatModelchatModel = GoogleAiGeminiChatModel.builder()
.apiKey("token")
.modelName("gemini-3.1-flash-lite")
.temperature(0.0)
.build();
ChromeDriverchromeDriver = newChromeDriver();
try {
chromeDriver.get("https://alstafeev.github.io/");
SeleniumDriverseleniumDriver = newSeleniumDriver(chromeDriver);
Agentagent = newAgent(seleniumDriver, chatModel, null, 10);
agent.aiAction("Search for 'midscene-java' block on the page and click it");
agent.aiAssert("Github repository 'midscene-java' opened");
} finally {
chromeDriver.close();
}
}
}

Playwright Example

packagecom.midscene.web;
importcom.microsoft.playwright.Browser;
importcom.microsoft.playwright.Page;
importcom.microsoft.playwright.Playwright;
importcom.midscene.core.agent.Agent;
importcom.midscene.web.driver.PlaywrightDriver;
importcom.midscene.web.driver.SeleniumDriver;
importdev.langchain4j.model.chat.ChatModel;
importdev.langchain4j.model.googleai.GoogleAiGeminiChatModel;
importorg.junit.jupiter.api.Test;
importorg.openqa.selenium.chrome.ChromeDriver;
publicclassLocalTestingTest {
@TestpublicvoidlocalMidscenePlaywrightTest() {
ChatModelchatModel = GoogleAiGeminiChatModel.builder()
.apiKey("token")
.modelName("gemini-3.1-flash-lite")
.temperature(0.0)
.build();
try (Playwrightplaywright = Playwright.create();
BrowserchromeBrowser = playwright.chromium().launch()) {
PagegooglePage = chromeBrowser.newPage();
googlePage.navigate("https://alstafeev.github.io/");
PlaywrightDriverpageDriver = newPlaywrightDriver(googlePage);
Agentagent = newAgent(pageDriver, chatModel, null, 10);
agent.aiAction("Search for 'midscene-java' block on the page and click it");
agent.aiAssert("Github repository 'midscene-java' opened");
}
}
}

(Note: You can still use the MidsceneConfig utility class and Agent.create(config, driver) for backwards-compatible quick initialization).

Advanced Features

1. Expanded API Methods

The Agent class provides specific methods for precise control:

importcom.midscene.core.pojo.options.ScrollOptions;
// Interactionsagent.aiTap("Submit button");
agent.aiInput("Username field", "admin");
agent.aiScroll(ScrollOptions.down());
agent.aiHover("User profile icon");
// Assertions & Waitsagent.aiAssert("The login button should be visible");
agent.aiWaitFor("Welcome message to appear");
// Data QueryStringprice = agent.aiString("What is the price of the first item?");
booleanisLoggedIn = agent.aiBoolean("Is the user logged in?");

2. Service Layer (Low-Level AI)

Use the Service class for direct AI tasks without full agent planning:

importcom.midscene.core.service.Service;
importcom.midscene.core.pojo.response.LocateResult;
importcom.midscene.core.pojo.response.ExtractResult;
importcom.midscene.core.pojo.response.DescribeResult;
Serviceservice = newService(pageDriver, chatModel);
// Locate element coordinatesLocateResultresult = service.locate("The blue checkout button");
System.out.println("Button at: " + result.getRect());
// Extract dataExtractResult<String> price = service.extract("Price of the main item");
// Describe elementDescribeResultdesc = service.describe(100, 200); // describe item at x=100, y=200

3. YAML Script Support

Define test flows declaratively in YAML:

target:
url: "https://saucedemo.com"tasks:
- name: "Login Flow"flow:
- aiAction: "Type 'standard_user' into username field"
- aiAction: "Type 'secret_sauce' into password field"
- aiAction: "Click Login"
- aiAssert: "User should be on the inventory page"
- logScreenshot: "Inventory Page"

Run it with Java:

importcom.midscene.core.yaml.ScriptPlayer;
importcom.midscene.core.yaml.ScriptResult;
ScriptPlayerplayer = newScriptPlayer("login_script.yaml", agent);
ScriptResultresult = player.run();

4. Caching

Midscene caches planning results to speed up execution and save tokens. You can inject a custom TaskCache instance when building the Agent.

importcom.midscene.core.cache.TaskCache;
Agentagent = newAgent(pageDriver, chatModel, myTaskCache);

Supported Drivers

• Selenium: new SeleniumDriver(webDriver)
• Playwright: new PlaywrightDriver(page)

Contributing

Build from source:

git clone https://github.com/alstafeev/midscene-java.git
cd midscene-java
mvn clean install