Add smoke test JUnit framework

[PerfectSlayer]

What Does This Do

This PR adds a JUnit 5 / Java smoke test framework that allows to write smoke tests using Java/JUnit 5 instead of Groovy/Spock.

The framework has four parts:

• App runners: AbstractSmokeApp with SmokeServerApp (long-running/HTTP) and
  SmokeCliApp (batch) concrete types, registered as a @RegisterExtension.
  Owns the launched app's lifecycle and wires in the agent jar + test agent session token.

• Backends: a common TraceBackend facade with two interchangeable implementations:
  MockAgentBackend (in-process JavaTestHttpServer, no Docker) and TestAgentBackend
  (a Testcontainers-managed, or external/CI, dd-apm-test-agent, scoped per test via
  X-Datadog-Test-Session-Token).
  A test written against the facade runs unchanged on either backend but the goal is moving toward TestAgentBackend by default.

• Capture surfaces: Traces (msgpack/JSON decoded through the shared decoder), Logs (std out/error logs with clear, waitFor capabilities),Telemetry (with message-batch flattening), and RemoteConfig (push a config the
  tracer polls for, and read back its /v0.7/config polls, with products/capabilities
  decoders).
  Traces, telemetry, and remote config are all isolated per test for owned backends.

Motivation

The goal is to provide framework for testing tracing instrumentations and products.
Unlike the existing solution, it verifies the signals against an agent, not our our decoders, and it no longer relies on Groovy / Spock but Java and JUnit.

Additional Notes

Contributor Checklist

• Format the title according to the contribution guidelines
• Assign the type: and (comp: or inst:) labels in addition to any other useful labels
• Avoid using close, fix, or any linking keywords when referencing an issue
  Use solves instead, and assign the PR milestone to the issue
• Update the CODEOWNERS file on source file addition, migration, or deletion
• Update public documentation with any new configuration flags or behaviors
• Once approved, use merge queue to merge the PR

Jira ticket: APMLP-1247

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.00 s	13.95 s	[-0.5%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent	12.84 s	12.99 s	[-1.9%; -0.4%] (maybe better)
startup:petclinic:appsec:Agent	16.90 s	17.30 s	[-6.5%; +1.9%] (no difference)
startup:petclinic:iast:Agent	17.42 s	17.56 s	[-1.7%; +0.1%] (no difference)
startup:petclinic:profiling:Agent	17.42 s	17.29 s	[-0.6%; +2.1%] (no difference)
startup:petclinic:sca:Agent	17.46 s	17.43 s	[-0.8%; +1.2%] (no difference)
startup:petclinic:tracing:Agent	16.61 s	16.65 s	[-1.2%; +0.7%] (no difference)

Commit:9d9b24d8 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[datadog-datadog-us1-prod]

Bits has a CI fix ready

🟢 Investigated · 🟢 Fix prepared · ⚪ Validation skipped · 🟠 Ready

spotless failed because the new smoke-test backend sources did not match the repository Java formatter. Reformatted RemoteConfig.java, TelemetryDecoder.java, and TestAgentBackend.java without changing behavior.

Commit fix to this PR

---

View in Datadog | Reviewed commit 4abdba0 · Any feedback? Reach out in #deveng-pr-agent

[PerfectSlayer]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:4abdba0567

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Reject additional traces that arrive after an early match

For the default count-exact assertion, PollingConditions returns immediately when one snapshot matches. Because both backends receive traces asynchronously, the expected traces can match first and an additional trace from the same test action can arrive just afterward, causing the test to pass even though the final collection violates the exact-count contract; this also makes IGNORE_ADDITIONAL_TRACES ineffective as the distinction between strict and permissive behavior in that scenario. The strict path needs a stabilization/final-count check rather than accepting the first transient match.

Useful? React with 👍 / 👎.

[PerfectSlayer]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:9c1176468d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve startup traces for shared CLI backends

When a SmokeCliApp uses a backend that is also registered as a shared extension, the CLI process can finish and emit all of its traces during beforeAll, after which this beforeEach callback clears them before the test body runs. The CLI app's deliberate lack of an owned-backend reset cannot prevent this; TestAgentBackend only works if callers discover and enable retainAcrossTests(), while MockAgentBackend offers no retention mode, so shared CLI trace assertions are otherwise unusable.

Useful? React with 👍 / 👎.

[PerfectSlayer]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:b6b0809cde

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[PerfectSlayer]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:20e23f3f65

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Wait for forced termination after interruption

When teardown is interrupted while waiting for the child, this branch restores the interrupt flag and returns immediately after destroyForcibly(), which is asynchronous and does not guarantee the process has exited. During Gradle cancellation or another teardown interrupt, the child can therefore retain its port and files after the extension completes; the interrupted path should still confirm termination while preserving the interrupt status.

Useful? React with 👍 / 👎.

[datadog-datadog-us1-prod]

The new JUnit smoke framework cannot compile as configured: the PR removes the module's direct JUnit Jupiter compile dependency while its new main sources import JUnit extension APIs. Restore that compile-only dependency before shipping. No additional tests recommended: this is a build-configuration defect, and the existing tests cannot compile until the dependency is restored.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 76991be · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:76991be5ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[PerfectSlayer]

As using stacked PR, I will keep rebasing the PR multiple times a day.

[bric3]

Looks good to me. But the test agent seems to be missing env vars.

[bric3]

question: I was looking at gitlab file, and noticed a few other env vars

dd-trace-java/.gitlab-ci.yml

Lines 901 to 903 in e7bda6a

	DD_SUPPRESS_TRACE_PARSE_ERRORS: "true"
	DD_POOL_TRACE_CHECK_FAILURES: "true"
	DD_DISABLE_ERROR_RESPONSES: "true"

That got me looking at the test agent readme

https://github.com/DataDog/dd-apm-test-agent/blob/v1.64.1/README.md#testtrace_checkfailures-get

Shouldn;t those env var be added, in particular it seems that configuring ENABLED_CHECKS only is not enough, and that should be toggled on (DD_POOL_TRACE_CHECK_FAILURES is true).

https://github.com/DataDog/dd-apm-test-agent/blob/v1.64.1/README.md#testtrace_checkfailures-get

The others, like DD_DISABLE_ERROR_RESPONSES look sensible too.

Without it, I assume assertNoInvariantFailures() could return successfully even after a failed check.

[PerfectSlayer]

Yes we should. It’s missing from the container mode (working in CI but not failing locally).

I did not port the DD_SUPPRESS_TRACE_PARSE_ERRORS... I wonder what should we do about it 🤔
This could be reviewed in another PR (like disabling it on CI first to check the existing non compliant behavior).

[sarahchen6]

LGTM! I think the organization is done really well and helped with understanding what's going on, e.g. separate SmokeCliApp and SmokeServerApp, each test file is relatively short and focused, backend folder is clearly organized... nice!

It was also helpful to see the stacked PRs with this framework effectively in action.

[AlexeyKuznetsov-DD]

nit: maybe better to use TODO (we have 700+ TODOs and 100+ FIXME)?
optional: WDYT if this list to be in resources? But in this case we need to think how to keep comments for each know issue.

[PerfectSlayer]

I don’t want to touch / changed this code. It is only moved from its original place following feedback from this PR.

By the way, I think TODO and FIXME don’t have the same meaning (and editors usually handle them differently).
To me TODO is more changes that need to be completed, whereas FIXME are known bugs or limitations.
I can set a TODO in an empty method I need to implement later, while I will add a FIXME around a collection that has a race condition for example.

[AlexeyKuznetsov-DD]

nit: kind of not expected to see deprecated code in brand new class...

[PerfectSlayer]

This was added following feedback from the other reviewers.
What other alternative do you envision to convey the mock agent is for local rapid development only, to avoid spinning container?

[AlexeyKuznetsov-DD]

Just curious why we need ddapm-test-agent? In Groovy version we do not have it.

[PerfectSlayer]

I might be missing the context of the question but this is the whole point of the framework, to stop testing against ourselves but against an agent implementation.
We will stop testing our encoding and trace pipeline using our own decoding. Same for telemetry, remote config, etc... Does it answer your questions?

[AlexeyKuznetsov-DD]

Just curious if there any need to wrap values from builder one more time?
Or we expect builder to be reused?

[PerfectSlayer]

There is no way to ensure the builder won’t be re-used.
I have example where I could have re-used the builder (like the Spring Boot RabbitMQ smoke tests that creates two similar servers to interact with the queues).
Additionally, this is test code, so I’m fine allocating collection performance wise (if it can save some troubleshooting time later or avoid race conditions).

[AlexeyKuznetsov-DD]

Maybe we can think some simple class around command args to make code a bit more readable, like some sort of builder/utility class with handy methods to make code more readable, something like below (just idea), and maybe worth to have it as common component for building cmd line args:

command
.agent(agentJar) // -javaagent:xxx
.agentFlag("dd.agent.host", backend.url().getHost()) // -Ddd.agent.host=zzz
...
.jar(this.jar)

[PerfectSlayer]

Maybe we can think some simple class around command args to make code a bit more readable

Sorry, I might misunderstand the feedback but this is exactly what this code is doing.
The builder is an abstraction of the original List<String> command for the ProcessBuilder.
It has methods like javaAgent(), jar(), mainClass(), classPath(), args(), jvmArgs() to avoid users having to manually construct the tedious string array.

[bric3]

nitpick:

Suggested change

	* <pre>{@code
	* // shared: JUnit drives the lifecycle, every app reports to this one backend
	* @RegisterExtension
	* staticfinalAgentBackendagent = AgentBackend.testAgent();
	*
	* // owned: the app starts, resets and stops its own backend
	* @RegisterExtension
	* staticfinalSmokeCliAppapp = SmokeCliApp.named("my-app")
	* .backend(AgentBackend.testAgent())
	* .build();
	* }</pre>
	* <strong>shared:</strong> JUnitdrivesthelifecycle, everyappreportstothisonebackend
	* <pre>{@code
	* @RegisterExtension
	* staticfinalAgentBackendagent = AgentBackend.testAgent();
	* </pre>
	*
	* <strong>owned:</strong> theappstarts, resetsandstopsitsownbackend (notetheannotationappliesto {@codeSmokeCliApp})
	* <pre>{@code
	* @RegisterExtension
	* staticfinalSmokeCliAppapp = SmokeCliApp.named("my-app")
	* .backend(AgentBackend.testAgent())
	* .build();
	* }</pre>

[bric3]

question: Would it work to make this package visible, to enforce AgentBackend use ?

[PerfectSlayer]

It’s not excluded that mock agent and test agent API might differ. Like test agent getting more features.
I would rather avoid enforcing to declare the behavior into the common abstraction to be reachable, leaving stub methods in the mock agent class.

[bric3]

Annother batch of comments after looking at later PRs.
I'm weighing on renames to avoid confusion when reading code, while the build rename can be skipped, I thing the assertion methods need to be more explicit on the waiting.

Anyway pre-approving.

[bric3]

thought: I was looking at later PR,s and one thing stroke me, it is now tad obscure if the app has finished. I understand this how JUnit works and as such clearly intended, but I wonder if this is something that may trip readers and writers of smoke tests.

I wonder if the build method can be renamed to suggest this happens before the test ? E.g.

 @RegisterExtension
static final SmokeCliApp app =
SmokeCliApp.named("opentelemetry")
.jar(APPLICATION_JAR)
...
- .build();+ .startBeforeAll();
@Test
void receivesTraces() {
app.traces()
.assertTraces(...);
}

WDYT ?

[PerfectSlayer]

I wonder if the build method can be renamed to suggest this happens before the test ?

Any field initilazer will be executed before method call.

it is now tad obscure if the app has finished

How would that solve the lack of clarity around the finish state? To me, it’s the same as @Shared or @AutoCleanup from Spock, the annotation holds the lifecycle. It’s only because we never used it that we don’t know about it. But I don’t think we changed the semantic of the @Shared field initialized methods to convey its lifecycle.

[bric3]

thought: After reading an example with otel app, I'm weighing if this method should be renamed, because in essence it actually awaits for traces to be visible within a specific deadline (given by the timeout).

So what about waitForTraces ? (And same for the overloads)

[PerfectSlayer]

Yes, there is something off. It does not match the awaitLogLines for example…
It need to manually fix it to ensure coherence here. Thanks for catching it.

[bric3]

nitpick: Maybe adding an overload accepting a class could be useful, thus avoiding calling .getName() on it.

[bric3]

suggestion: Actually looking at usage, what about passing the classpath as a second argument ? Maybe by default System.getProperty('datadog.smoketest.shadowJar.path') if it exists and it's not blank ?

[bric3]

suggestion: What about renaming this to pushConfig, to follow what the javadoc says.

[datadog-datadog-us1-prod]

A zero log timeout calls Object.wait(0), which waits without a time limit. A smoke test that requests no wait can hang until the job timeout.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 9d9b24d · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[datadog-datadog-us1-prod]

Reject a zero timeout before waiting

A smoke test that requests no wait can hang until the job timeout.

Assertion details

• Input: Call processTestLogLines(predicate, 0) when no matching log line is available.
• Expected: When no matching log exists, a non-positive timeout must end at once with TimeoutException.
• Actual: The first wait uses Object.wait(0). Java treats zero as an unlimited wait. A negative value throws IllegalArgumentException.

Suggested change

	waitTime = timeoutMillis;
	if (timeoutMillis <= 0) {
	thrownewTimeoutException();
	}
	waitTime = timeoutMillis;

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}