Request for feedback: Restructuring the environment object

[gregweb]

As creator of a HIL (Hardware in the Loop) system for designers and manufacturers of hardware with their own software I very much value your effort in bringing CTRF forward. It in most places is a very good fit without getting unnecessarily complex.

But now to the part of the CTRF where I hope for some improvement:

In section "10. Environment Object" the spec states ([n.] added by me for easier referencing):

The environment object describes the execution environment [1.], system configuration [2.], and CI/CD context [3.] in which the test run was executed.

Comments:

1. "Execution environment" makes it completely clear, that the contents within the environment object is about the environment the software/app/service/device under test (xUT in short) was executed.
2. "System configuration" is a little broad and unspecific (which probably is intended). Most importantly it does not contradict "execution environment". So, fine but maybe superfluous?
3. "CI/CD context" is also clear. From the traceability viewpoint, CI/CD information is very valuable. The only question I'd like to raise is if this is part of the "execution environment" or more an artifact of how the xUT was built.

In the following I distinguish between the following:

• "test execution environment" (aka "execution environment" [1.])
• "build execution environment" (e.g. CI/CD context [3], the host that executes a make file, ...)
• "software/app/service/device under test (xUT)": the system, artefact or entity that is tested

Now to the single items (with "extra" omitted):

• section "10.1. reportName":
  Has nothing to do with the environment, maybe move to top level to its cousin reportId?
• section "10.2. appName" and "10.3. appVersion":
  • The xUT's name and version: understood
  • Is the xUT part of the environment?
• sections "10.4. buildId", "10.5. buildName", "10.6. buildNumber" and "10.7. buildUrl":
  • The ID/name/number/url generated by the build execution environment when it built the xUT for later identification.
  • This actually could be seen as property of the xUT.
• sections "10.8. repositoryName" and "10.9. repositoryUrl":
  The spec says "[...] source code repository associated with the test run.".
  • Please clarify: Is it about the xUT's repository or about the test environments repository?
  • I assume the test environment and tests are also under version control.
  • Repository information make sense for all of them.
• section "10.10. commit":
  The spec says "The commit hash or revision identifier of the code under test.".
  • explicitely xUT related
  • not about the test execution environment
  • Is the xUT part of the environment?
  • It seems more to be a cousin of "9.27. device" (reslts.tests[].device).
• section "10.11. branchName":
  The spec says "[...] branch from which the test run was executed."
  • Please clarify: Is it about from which branch the xUT was built by the build execution environment or about the branch the test execution environment is from?
  • I assume the test environment and tests are also under version control.
  • Branch information make sense for all of them.
• sections "10.12. osPlatform", "10.13. osRelease" and "10.14. osVersion":
  • the test environments OS related information; understood so far
• section "10.15. testEnvironment":
  • test execution environment related
• section "10.16. shardId":
  • test execution environment related
• section "10.17. healthy":
  • test execution environment related
  • The part "the overall system or" more confuses me than clarifying things.

For traceability reasons it would make sense to collect:

• repositoryName
• repositoryUrl
• commit
• branchName
• osPlatform, osRelease, osVersion

for each of:

• the "test execution environment"
• the "build execution environment"
• the xUT itself

separately.

So the question arises, what to include under "environment" (in the sense of test execution environment) or broaden its coverage officially.

[Ma11hewThomas]

Thanks @gregweb, this is a great analysis, and we agree that the issue highlights a genuine problem in the current model.

Over time, environment has become somewhat of a catch-all for different concepts. It's something we are already assessing.

We’d like to accept this issue as a design discussion and request for feedback before implementing a schema change. Before v1, we should decide how any improved structure could look.

Suggestions and real-world examples are very welcome, so thanks for this!

I will keep this ticket open and invite further feedback - I've amended the title to reflect this.

[Ma11hewThomas]

Note This issue is being kept open to gather use cases and discuss an improved pre-v1 structure.

[gregweb]

Initiator: Proposed Replacement for Concept of "build execution environment"

Introducing the concept of a "build execution environment" and trying to find a suitable model for it in CTRF somehow feels wrong.
It is not about "executing a build". It is more about information related to the instance that initiates or triggers a test run.

Why? -- Traceability!

Could you agree upon the following?

The intention is to be able to identify the circumstances that led to the initiation of this specific test run.

Examples:

• CI/CD system as test run initiator:
  • initiator name: "Project A CI/CD", ...
  • initiator login name: cicd, ...
  • initiator role: "regression preventor", ...
  • initiator type: system
  • trigger reason: "merge request", "incremental build", "full build", "nightly build", ...
  • referer url and/or id: e.g. pipleine url/id
• developer or QA staff as test run initiator:
  • initiator name: "Caren", "Steve", ..
  • initiator login name: caren, steve
  • initiator role: "developer", "qa staff", ...
  • initiator type: human
  • trigger reason: "development", "reproducing a bug", "release testing", ...
  • referer url and/or id: maybe a link to a (Jira) task as context information
• AI agent as test run initiator:
  • initiator name: whatever name you give your agents :-)
  • initiator login name: do agents have a login name?
  • initiator role: "quality supervisor", ...
  • initiator type: "AI agent"
  • trigger reason: ...
  • referer url and/or id: maybe a link to a (Jira) task as context information

Questions answered:

• Who/what initiated the particular test run?
• Why was this particular test run initiated?
  --> maybe replace "trigger reason" by "purpose"?
• In what context should this specific test run be seen?
  --> maybe replace "referer" by "context" or "scope"?

In my mind's eye, I see the following picture:
The initiator sends a package (i.e., a request, containing: test scope, software version, ...) to the test environment, which executes the request, documents the result, and reports back on the execution.
(This picture may help identify the minimum necessary entities and actors.)

Please comment and refine!

Update: added initiator login name

[gregweb]

Proposal: Additional Objects systemUnderTest and initiator

The goal is to explicitly distinguish between the three concepts.

Draft definition/description of the three concepts:

• Environment: The system, platform, or context in which tests were executed, including build, commit, branch, OS, and other metadata [of the environment]. --> from chapter "3. Terminology"
• System Under Test: The system, software, device or service the tests were executed for, including build, commit, branch, OS, and other metadata regarding the System Under Test. --> this is the "xUT" mentioned above
• Initiator: The initiator that triggered the test run for the System Under Test within the Environment, including necessary metadata to identify the circumstances that led to the execution of the test run.

It probably makes sense to split the three aspects into three objects:

• results.environment object (already existing, see chapter "10. Environment Object")
• results.entities object (new, alternative: "components", ...) --> notes: [1] [2] [3]
• results.initiatorobject (new)

Notes:
[1] The term "entity" is inspired by the SNMP ENTITY-MIB:

[...] Effectively, there is some "overall" physical entity which houses the sum of the things managed by that one agent, i.e., there are multiple "logical" entities within a single physical entity. Sometimes, the overall physical entity contains multiple (smaller) physical entities, and each logical entity is associated with a particular physical entity. Sometimes, the overall physical entity is a "compound" of multiple physical entities (e.g., a stack of stackable hubs). [...]
(https://datatracker.ietf.org/doc/html/rfc4133#page-3)

[2] Excuse my incremental thinking, but "system under test" is not really a good fit because in typical scenarios tests are executed for a system under test with help of another components/entities. Example: A web service (here the system under test) is tested against one or multiple web browsers. Therefore information about the service and information about the web browsers are relevant.

[3] Given the service/browser example from note [2] above, typically the web service is tested against different browser/OS combinations. That's why I tend to think that the information about the entities involved should be part of the results.tests[] objects. Optionally.

Updates:

• instead of proposing results.systemUnderTest now proposing entities
• added notes [1], [2] and [3] regarding the change
• improved reference to ENTITY-MIB and citation related to the term "entity"