stream: the position of _read() is wrong

[helloyou2012]

Fixes: #33940

[VoltrexKeyva]

Hi @helloyou2012, could you remove the merge commit from your branch? Merge commits tend to break the tooling, you can use git rebase instead.

node/doc/guides/contributing/pull-requests.md

Lines 215 to 217 in 54322b8

	As a best practice, once you have committed your changes, it is a good idea
	to use `git rebase` (not `git merge`) to synchronize your work with the main
	repository.

[benjamingr]

@nodejs/streams

[mcollina]

Why is wrong? Can you please expand?

Can you please add a unit test?

[ronag]

Needs test and also a little unsure what exactly we are solving here. If I would guess:

• Currently this.pos could actually be larger than the file size, which is a little weird.
• Currently this.pos is what we have requested, but necessarily what we have read.

[helloyou2012]

@mcollina see this issue: #33940, this

node/lib/internal/streams/readable.js

Line 487 in d4f33f1

this._read(state.highWaterMark);

will call _read(n) multiple times with n= highWaterMark, but the bytesRead may be not equal to n, then the data will loss. Example:

// file data is: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
// pos = 0;
_read(10)
// if bytesRead = 3 then buf = [0, 1, 2]
// pos = 10
_read(10)
// if bytesRead = 10 then buf = [10, 11, 12], so [3, 4, 5, 6, 7, 8, 9] is loss.

[Ayase-252]

The problem arised because current implementation assumed next position to start reading is this.pos + n

node/lib/internal/fs/streams.js

Lines 275 to 277 in d4f33f1

	if(this.pos!==undefined){
	this.pos+=n;
	}

But in reality, it does not necessarily read n bytes from file. In the callback of file handle read, it is possible that bytesRead does not equal n (less than n in my exprimentation)

node/lib/internal/fs/streams.js

Line 246 in d4f33f1

.read(this.fd,buf,0,n,this.pos,(er,bytesRead,buf)=>{

As a result, some bytes in file may be skipped. The "assumed" reading is longer than the "actual". pos moves too far.

About the test, the issue occurred because read in the above line reads less bytes bytesRead than requested n. How could we reproduce (or mock) such condition in test?

[ronag]

I think this change could make sense. But we need a test.

[benjamingr]

About the test, the issue occurred because read in the above line reads less bytes bytesRead than requested n. How could we reproduce (or mock) such condition in test?

Good question. I'm not sure I don't think I ever wrote such a test in Node -

A good way to try is to take the test case from #33940 and to try to make a smaller isolated case from it (shorter interval and smaller watermark)

@nodejs/fs

[helloyou2012]

I added test case. @ronag@benjamingr

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/37573/

[benjamingr]

Any reason this is a mustCAllAtLeast and not a mustCall?

[helloyou2012]

mustCall is exact times, but this place will call at least 1 time not exact 1 time.

[benjamingr]

LGTM if the test passes after this and fails in master

[benjamingr]

Also thank you for the meaningful contribution and working with us 🙏

[mcollina]

lgtm, hopefully the setInterval test would not fail in CI.

[ronag]

LGTM. Was wondering why this hasn't landed yet. Turns out it's my fault.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/37713/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/37732/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/37743/

[ronag]

I think there is a problem on arm with the new test. @helloyou2012

[helloyou2012]

I think there is a problem on arm with the new test. @helloyou2012

Yes, the test timed out. I think it is better to set a time after which to exit safely. I will fix later.

[jasnell]

Removing the author ready label as there's still work to be done here.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/37862/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/37873/

[ronag]

@helloyou2012 this still seems to have problems passing. Could you try and look at the CI failures? I started another CI just in case.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/37875/

[helloyou2012]

CI: https://ci.nodejs.org/job/node-test-pull-request/37873/

It failed cause of inspector-cli/test-inspector-cli-pid timed out, not of this.

[helloyou2012]

CI: https://ci.nodejs.org/job/node-test-pull-request/37875/

This CI all passed.

[mcollina]

Landed in d826f6b