build: make test-ci target produce tap output

[bnoordhuis]

Make make test-ci print output in TAP format. The default output
doesn't show up in the Jenkins web view until the test suite finishes.
The web view is line buffered and the test runner doesn't emit newlines
until it's done.

A nice side effect is that the test runner now prints each test's
running time.

R=@rvagg? I assume the Jenkins TAP plugin must be activated first.

[trevnorris]

Don't have the know how to sign off, but +1 from me.

[jbergstroem]

Yes! There's probably some plug to generate build artefacts so we can single out individual tests from jenkins reports.

[rvagg]

we haven't actually been using test-ci, I switched to test-simple + test-message because of the broken addon stuff. I'll give this a try now and see how it goes.

[rvagg]

https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/169/ with this branch using test-ci

The TAP plugin for Jenkins requires a TAP file be written (as far as I can see). I could adjust the Jenkins invocation to write stdout to a file but that's not going to work for Windows. Should we add an option to tools/test.py to print the progress output to a file as well as the console?

[rvagg]

I don't think the problems with addons & test-ci have been resolved by the look of it, /cc @chrisdickinson

[jbergstroem]

@rvagg as for storing results to file, would you prefer just redirecting sys.stdout or should I do something fancier?

[chrisdickinson]

@rvagg iirc, addons break (I think) because this expression is evaluated and expanded before any of the make targets run, so node-gyp doesn't run on them, and thus they don't get compiled.

Now as for how to fix that, I'm unsure. @isaacs?

[rvagg]

@jbergstroem my Python fu is not strong enough, I started tinkering with this but probably shouldn't waste my time if someone else can do it!

Here's my proposal: a --output (or similar) option that make the progress runner in use write to the file name provided as well as stdout. It's actually very helpful watching the test output come through as it runs since they take so long. I was thinking that perhaps the base progress class could have a Print that does both of these things in one go and replace all the print statements with that.

If it's too hard, then just to a file will do, I can make Jenkins print out the full output after it's finished I think which is better than nothing.

[jbergstroem]

@rvagg I'm thinking using import logging to control output, then use different loglevels depending on what we'd prefer (make test-ci) to get more or less info into stdout or a file.

I'll see what patch I can whip together over the weekend.

[bnoordhuis]

I don't think the problems with addons & test-ci have been resolved by the look of it

I was working on that in #335 but it stalled (I forgot about it.) I'll pick it up again.

[jbergstroem]

@bnoordhuis Ok. Lets get that in first then.

[rvagg]

TODO: put addons back in test-ci

closing because this was mostly done in #938