Clarify spec intent: common xtra field names, labels vs xtra.traits, stdout/stderr line-splitting

[Evangelink]

Context

We are adding a CTRF reporter to Microsoft.Testing.Platform (the engine behind MSTest, xunit.v3, NUnit, and various other .NET test frameworks). Brad Wilson (xunit) recently published a side-by-side comparison (microsoft/testfx#8903) of his xunit.v3 CTRF emitter vs ours, and several differences boiled down to "the spec doesn't say". Before each ecosystem settles on its own conventions, we would love your guidance on the following so that CTRF documents produced by different .NET reporters are interoperable.

---

1. Canonical key names for common extra fields

The spec marks environment and per-test as additionalProperties: false for everything except extra, which is the freeform escape hatch. That works, but means two producers reporting the same concept (machine name, exception type, etc.) can pick different keys inside extra, hurting tooling that wants to consume both.

Two concrete examples we hit:

• Machine / hostname: we emit environment.extra.machine; xunit.v3 emits environment.extra.computer.
• Exception type name: we emit tests[].extra.exceptionType (e.g. System.InvalidOperationException); xunit.v3 emits tests[].extra.exception.

Are there "recommended" key names for these (and other widely useful fields like build agent, runtime version, framework name)? It would be great either to (a) bless one of the two existing names per concept, or (b) publish a non-normative "common extra keys" appendix so future producers can converge.

2. What is labels for?

The spec says labels is "Structured key-value metadata for the test case (e.g. priority, severity, external identifiers)" typed as an object whose values are string | number | boolean.

We read the e.g. as non-exhaustive and use labels for the test framework's user-defined key/value traits (e.g. Category=Smoke, Owner=alice). xunit.v3 instead puts traits under tests[].extra.traits because Brad reads labels as restricted to the three listed examples (priority/severity/external IDs).

Could you clarify the intent?

• Option A - labels is the right place for any structured key/value test metadata (our reading).
• Option B - labels is reserved for a narrow administrative set; arbitrary trait-style metadata should go under extra.
• Option C - something else (e.g. parameters for data-driven inputs, labels only for the three listed examples, extra.traits for everything else).

We are happy to follow whichever interpretation matches your intent - just want to avoid two .NET CTRF producers diverging on something this central.

3. Are stdout / stderr meant to be one entry per physical line?

The schema describes both as "Standard output lines from test execution", typed as an array of strings. We initially emitted a single multi-line string (one array element with embedded \n); we have since switched to one element per LF-terminated line (with CRLF normalization and the trailing newline not producing an extra empty entry).

Two follow-ups:

• Is that one-element-per-physical-line interpretation what the spec author intended?
• For tests that produce no captured output, should the array be present-and-empty ([]), or omitted entirely? We currently omit it; the spec doesn't say.

Thanks for the work on CTRF - it has made a real difference for us standardizing across .NET test frameworks.

[bradwilson]

I will add my comment on issue 2, which is that I don't see labels anywhere in the spec, so Amaury's description of our differences isn't quite correct.

The one thing I did interpret on my own is that tags is described as User-defined tags or labels for categorization (e.g., ["smoke", "regression"]), so the mapping I chose was traits named Category. A user who adds [Trait("Category", "Smoke")] not only gets that trait listed in tests[].extra.traits (along with all other traits, regardless of name) but also gettings tests[].tags set to ["Smoke"].

[Ma11hewThomas]

It's great to hear you are adding a CTRF reporter to MTP, and thanks for the feedback.

For point 1, it’s preferred to promote common properties to first class CTRF fields rather than establish recommendations or non-normative conventions for extra, which could become difficult to maintain over time.

Further specification guidance is being drafted for v1 about avoiding collisions in extra, and optional namespacing. While that should help producers avoid conflicts, it would not address the issue raised here unfortunately.

Regarding point 2, option A is correct.

Labels were added recently to help plug the gap between keyless tags and key/value metadata. As CTRF grew, it became clear we’d need to support frameworks that expose simple tag-style classifications, as well as those that emit key/value metadata like traits. @bradwilson was an early adopter of CTRF, and labels weren’t part of the spec at the time.

I'll add an item to clarify this in the schema and specification prior to v1.

Regarding stdout and stderr
Yes, the intended representation for stdout and stderr is one array entry per output line.

For tests with no captured output, both omission and an empty array are valid, this is currently left to producer to decide.

Thanks again for the feedback!

[bradwilson]

Labels were added recently to help plug the gap between keyless tags and key/value metadata.

Can you point to this in the spec?

Also, you should allow arrays for label values, since our users can specify multiple values for a single name.

[Ma11hewThomas]

https://github.com/ctrf-io/ctrf/blob/main/spec/ctrf.md#915-labels

I'm investigating an inconsistency with the website documentation at the moment, so apologies for the confusion.

Regarding allowing arrays for label values, this is a reasonable use case, and I can confirm I will add support for it. This means the following will be CTRF compliant:

"labels": {
 "priority": "high",
 "owners": ["qa", "platform"],
 "category": ["smoke", "staging"]
}

Arrays will accept the same primitive types as individual label values: string, number, and boolean.

Thanks for raising this and again for the feedback, much appreciated.

[bradwilson]

Makes sense, I had been relying on the website spec, not the GitHub spec. Thanks for the clarity!

[bradwilson]

I guess I should ask: will labels replace tags in the 1.0 spec, then? It feels like tags becomes a degenerate case for a specific "kind" of label (without prescribing what that should be, where I chose Category as the most likely mapping based on wording of the spec).

[Ma11hewThomas]

My preference is to keep tags and labels as complementary fields rather than make labels replace tags.

• tags are for simple, keyless classification, e.g. ["smoke", "regression", "api"]
• labels are for structured key/value metadata, e.g. { "priority": "high", "owner": "qa" }

While tags could technically be represented as a specific label such as tags, that would prescribe a mapping that CTRF should not impose. Keeping both allows simple categorisation to stay simple, while labels provide a more structured option where needed.

Prior to 1.0, I intend to provide clearer guidance and documentation around the intended use of tags and labels.