Are our tests reliable?

[bjohansebas]

We use supertest to run our tests in this module. While that tool does a great job, I'm not sure if we're making reliable tests, since Supertest also handles the body according to the content-type, which makes me question whether we're doing good tests or if in the future the tests will stop being reliable as we update that dependency.

Ref: forwardemail/superagent#1812

cc: @Phillip9587, @jonchurch, @wesleytodd, @UlisesGascon

[jonchurch]

Supertest only parses the body asserts against the parsed body if you assert against an object, but when doing a string assertion it compares against the raw res.text

https://github.com/ladjs/supertest/blob/2ae1c36f48fdc5a60e8239c2d69c47e169042829/lib/test.js#L186C5-L201C22

[jonchurch]

Which is to say, it's something important to think about. And we could explore lower level tests to ensure we are asserting against an unparsed response body.

Currently, we should be fine asserting against res.text by passing the body to expect as a string or regexp

[kilisamemarisaaa]

Hi @bjohansebas — digging into this after @jonchurch's clarification. I ran a quick reliability scan of the current suite and CI; the headline finding is less about supertest and more about platform coverage.

CI only runs on ubuntu-latest (Node 18.0.0 / 18–26, fail-fast: false) — no windows/macos legs. For core middleware that's the biggest practical reliability gap right now. We hit this exact pattern in multer: adding Windows legs (#1334) surfaced real cross-platform failures that ubuntu-only CI could never see — path assertions (path.sep in #1452), and an abort-cleanup race specific to Windows (#1456, file-handle EPERM). Body-parser is the same class of package, and the "tests unreliable in the future" risk is much more likely to come from a platform/behavior gap like that than from supertest's body parsing.

On the supertest concern itself, the suite is mostly already aligned with jonchurch's guidance: assertions target the raw body rather than supertest's parsed object — e.g. urlencoded.js calls JSON.parse(res.text) itself (lines 253, 283, 344, 992), and string/regexp expectations are the norm. So a superagent behavior change would not silently shift what those tests assert.

Small notes from the scan:

• One timing-based wait: text.js:180 (setTimeout(…, 100) before expecting the 413 for gzip + limit). Low risk today; poll-until-expect would be more durable.
• No path.* / fs / tmp usage anywhere in test/, so the classic platform landmines aren't present.

If it would help, I can open a PR adding windows-latest (and optionally macos-latest) to the ci.yml test matrix — begin with continue-on-error: true like multer's experimental legs — or add a small lower-level test that asserts against the unparsed response to lock in raw-body behavior.