Extensibility docs should signpost & warn where developers could be caught out by cross-process marshalling

[craigfowler]

This relates to something that I have had this explained to me somewhere before, but it was years ago and I can't find where that discussion happened. I've looked through SO questions and issues/discussions on the NUnit repo.

There is a limitation, really a feature/consequence of NUnit's design (and not one I consider to be bad) In the default usage the "discovery and test suite/case creation" phase occurs at arms length to the "test execution" phase, I believe in a separate .NET process. From what I recall of the last time this was explained to me, objects/values which are created during discovery/creation are then marshalled in some manner over to the execution. Examples of this are where test case parameter values are injected by something which implements ITestBuilder.

When those objects turn up in the test case as a method parameters, they are not actually the precise same objects as were created in the test builder, because they have crossed a boundary (again, I think that's a process boundary). This can have some unexpected and baffling consequences to a developer if they had not had this boundary-crossing pointed out to them. For example, an object which emits events passed as a test case parameter. If an event listener is subscribed to that object over in the "discovery & creation" part of the test suite then events which are emitted during the test (triggered by logic that executes within the test) won't be observed by the subscriber. That's because the object emitting the events isn't actually the precise same object that was subscribed-to. The object emitting the events is a marshalled-copy of the original object.

I think the docs should point this out for extension points which can lead to this behaviour. It's not obvious if you are not already familiar with the deep internals of NUnit. Even having already been told once, over the course of a few years I managed to forget and was caught out by this a second time.

[OsirisTerje]

@CharliePoole Could you comment here?

[CharliePoole]

The division into two "phases" in the life-cycle of a test is intrinsic to NUnit and is the cause of a lot of problems for users. On the custom attribute page (which I might have written myself some time in the distant past), this is stated in a very non-emphatic way, saying we are grouping the interfaces "for convenience of understanding." I think this general explanation could be improved.

The general problem that is caused by this split, of course, is that users sometimes try to change things in their tests, which are already completely determined before the tests begin execution. My suggestion is that someone should add a section to the docs that describe this in a bit more detail at a user level, i.e. stating clearly what you may not change in your tests execution. This is slightly tricky to express because code invoked by attributes is sometimes considered by the users as "part of the test" whereas it's actually an annotation of the test.

However, it's incorrect to say that the load-time instances are marshalled to the execution phase. Marshalling is used when a process, e.g. a runner, is communicating with the framework running in a separate domain or process. Within the engine, the instances of Test and its derivatives remain in memory throughout execution, unless unloaded at the command of the runner or at the end of the process.

What the framework does marshal, is the result of a run and the individual events that take place in that run. This is most likely what is happening here. Note that such marshaling only occurs when separate processes or domains are involved. This is all of the time if we are talking about the standard (.NET Framework) console runner but not when the current .NET Core runner is used. So if something is written about this, it needs to clarify exactly when the problem occurs. If events are actually a problem here, we need to know which events, and what data is being injected into them.

Yet another problem that sometimes comes up may or may not apply here. In the specific situation of using the VS Adapter as the runner, the framework is actually invoked twice. Once to display the tests and a second time to execute those "same" tests. If that problem turns out to be relevant here, we might want to document it somewhere else.

Just a note for everyone's info... IIRC the explanation on the "Custom Attributes" paged originated in notes I wrote for the NUnit developers. It was "promoted" to user-level documentation for extenders without thinking too deeply about who would be reading it, always a mistake when writing anything!