investigate flaky internet/test-dns-idna2008 test

[Trott]

test/internet/test-dns-idna2008.js (added in four days ago when #25679 landed) failed in CI a few hours ago:

https://ci.nodejs.org/job/node-test-commit-custom-suites/855/default/console

Host: test-rackspace-ubuntu1604-x64-2

00:01:48 not ok 9 internet/test-dns-idna2008
00:01:48 ---
00:01:48 duration_ms: 1.14
00:01:48 severity: fail
00:01:48 exitcode: 1
00:01:48 stack: |-
00:01:48 (node:4568) ExperimentalWarning: The dns.promises API is experimental
00:01:48 assert.js:753
00:01:48 throw newErr;
00:01:48 ^
00:01:48 
00:01:48 AssertionError [ERR_ASSERTION]: ifError got unwanted exception: queryA ESERVFAIL straße.de
00:01:48 at QueryReqWrap.dns.resolve4.mustCall (/home/iojs/build/workspace/node-test-commit-custom-suites/default/test/internet/test-dns-idna2008.js:27:10)
00:01:48 at QueryReqWrap.callback (/home/iojs/build/workspace/node-test-commit-custom-suites/default/test/common/index.js:368:15)
00:01:48 at QueryReqWrap.onresolve [as oncomplete] (dns.js:199:10)
00:01:48 at QueryReqWrap.onresolve [as oncomplete] (dns.js:199:19)
00:01:48 ...

It had previously succeeded in CI (see https://ci.nodejs.org/job/node-test-commit-custom-suites/852/default/console).

I guess ESERVFAIL is a DNS problem likely unrelated to the test itself? Although other DNS tests in internet ran just fine, so maybe it is in fact specific to this test?

@bnoordhuis

[bnoordhuis]

It's been a one-time failure so far? ESERVFAIL is reported by the DNS server; could be a temporary service outage.

[Trott]

It's been a one-time failure so far? ESERVFAIL is reported by the DNS server; could be a temporary service outage.

I had the temporary-hiccup idea too, but other DNS tests in the internet suite didn't fail. Still, the implication I'm getting from your comment--that there's probably not much point in chasing it down unless we see it more than once--is no doubt correct. Maybe leave this open for a bit and close it after a week or so if we don't see it again?

[Trott]

Failed again today:

https://ci.nodejs.org/job/node-test-commit-custom-suites/856/default/console

test-rackspace-ubuntu1604-x64-1

00:16:53 not ok 9 internet/test-dns-idna2008
00:16:53 ---
00:16:53 duration_ms: 1.14
00:16:53 severity: fail
00:16:53 exitcode: 1
00:16:53 stack: |-
00:16:53 (node:22610) ExperimentalWarning: The dns.promises API is experimental
00:16:53 assert.js:753
00:16:53 throw newErr;
00:16:53 ^
00:16:54 
00:16:54 AssertionError [ERR_ASSERTION]: ifError got unwanted exception: queryA ESERVFAIL straße.de
00:16:54 at QueryReqWrap.dns.resolve4.mustCall (/home/iojs/build/workspace/node-test-commit-custom-suites/default/test/internet/test-dns-idna2008.js:27:10)
00:16:54 at QueryReqWrap.callback (/home/iojs/build/workspace/node-test-commit-custom-suites/default/test/common/index.js:367:15)
00:16:54 at QueryReqWrap.onresolve [as oncomplete] (dns.js:199:10)
00:16:54 at QueryReqWrap.onresolve [as oncomplete] (dns.js:199:19)
00:16:54 ...

[bnoordhuis]

I'll take a look but I wonder if it's an environmental issue (i.e., network issue) because it never fails for me locally. What about you?

[Trott]

I'll take a look but I wonder if it's an environmental issue (i.e., network issue) because it never fails for me locally. What about you?

I've not had it fail locally either. It passed in last night's run on test-rackspace-ubuntu1604-x64-2.

To get a baseline (or to see if it passes 7 times, and then the firewall kicks in and it fails the remaining 93 times or whatever), here's 100 runs in CI:

https://ci.nodejs.org/job/node-stress-single-test/2147/

[Trott]

To get a baseline (or to see if it passes 7 times, and then the firewall kicks in and it fails the remaining 93 times or whatever), here's 100 runs in CI:

https://ci.nodejs.org/job/node-stress-single-test/2147/

No failures in stress test on test-rackspace-ubuntu1604-x64-2.

Re-running, but with all internet tests, but only 10 consecutive runs:

https://ci.nodejs.org/job/node-stress-single-test/2148/

[Trott]

Re-running, but with all internet tests, but only 10 consecutive runs:

https://ci.nodejs.org/job/node-stress-single-test/2148/

Also successful on test-rackspace-ubuntu1604-x64-2.
¯\(ツ)/¯

[Trott]

Happened in CI again today.

https://ci.nodejs.org/job/node-test-commit-custom-suites/897/default/console

test-rackspace-ubuntu1604-x64-1

00:05:33 not ok 9 internet/test-dns-idna2008
00:05:33 ---
00:05:33 duration_ms: 25.277
00:05:33 severity: fail
00:05:33 exitcode: 1
00:05:33 stack: |-
00:05:33 (node:18514) ExperimentalWarning: The dns.promises API is experimental
00:05:33 assert.js:742
00:05:33 throw newErr;
00:05:33 ^
00:05:33 
00:05:33 AssertionError [ERR_ASSERTION]: ifError got unwanted exception: queryA ESERVFAIL straße.de
00:05:33 at QueryReqWrap.dns.resolve4.mustCall (/home/iojs/build/workspace/node-test-commit-custom-suites/default/test/internet/test-dns-idna2008.js:27:10)
00:05:33 at QueryReqWrap.callback (/home/iojs/build/workspace/node-test-commit-custom-suites/default/test/common/index.js:367:15)
00:05:33 at QueryReqWrap.onresolve [as oncomplete] (dns.js:199:10)
00:05:33 at QueryReqWrap.onresolve [as oncomplete] (dns.js:199:19)
00:05:33 ...

[Trott]

Again today:

https://ci.nodejs.org/job/node-test-commit-custom-suites/898/default/console

test-rackspace-ubuntu1604-x64-1

00:04:15 not ok 9 internet/test-dns-idna2008
00:04:15 ---
00:04:15 duration_ms: 35.304
00:04:15 severity: fail
00:04:16 exitcode: 1
00:04:16 stack: |-
00:04:16 (node:25878) ExperimentalWarning: The dns.promises API is experimental
00:04:16 /home/iojs/build/workspace/node-test-commit-custom-suites/default/test/common/index.js:675
00:04:16 const crashOnUnhandledRejection = (err) => { throw err; };
00:04:16 ^
00:04:16 
00:04:16 Error: queryA ESERVFAIL straße.de
00:04:16 at QueryReqWrap.onresolve [as oncomplete] (internal/dns/promises.js:167:17)
00:04:16 ...

[Trott]

It failed again today. So it failed each of the last three runs. (The test is run once a day in CI.) Interesting that when it fails, it fails reliably. And when it doesn't fail, it doesn't fail at all.

https://ci.nodejs.org/job/node-test-commit-custom-suites/899/default/console

test-rackspace-ubuntu1604-x64-1

00:04:16 not ok 9 internet/test-dns-idna2008
00:04:16 ---
00:04:16 duration_ms: 30.292
00:04:16 severity: fail
00:04:16 exitcode: 1
00:04:16 stack: |-
00:04:16 (node:22481) ExperimentalWarning: The dns.promises API is experimental
00:04:16 /home/iojs/build/workspace/node-test-commit-custom-suites/default/test/common/index.js:675
00:04:16 const crashOnUnhandledRejection = (err) => { throw err; };
00:04:16 ^
00:04:16 
00:04:16 Error: queryA ESERVFAIL straße.de
00:04:16 at QueryReqWrap.onresolve [as oncomplete] (internal/dns/promises.js:167:17)
00:04:16 ...

[Trott]

@nodejs/build Thoughts on this one?

[Trott]

Ooh, this time around things are different from when initially reported. The test is taking 30 seconds or so before failing. Probably some throttling going on somewhere, yeah? I'm logged on and will try running the lookups/resolves in series rather than in parallel and see if that does anything...