<streambuf>: basic_streambuf doesn't support 64-bit properly

[wohlstad]

std::basic_streambuf doesn't support 64-bit properly, because it holds the counters in 32-bit ints:

STL/stl/inc/streambuf

Lines 392 to 395 in b614834

	int _Gcount; // length of read buffer
	int _Pcount; // length of write buffer
	int* _IGcount; // pointer to length of read buffer
	int* _IPcount; // pointer to length of write buffer

Then e.g. in std::basic_streambuf::setg(...) you have:

STL/stl/inc/streambuf

Line 210 in b614834

*_IGcount = static_cast<int>(_Last - _Next);

and forcing a cast to int prevents from using with a memory range bigger than ~2GB (even on Win64 builds).

Also tracked by DevCom-102668, DevCom-290124, DevCom-320893, DevCom-859581, and Microsoft-internal VSO-485517 / AB#485517.

vNext note: Resolving this issue will require breaking binary compatibility. We won't be able to accept pull requests for this issue until the vNext branch is available. See #169 for more information.

[JeanPhilippeKernel]

Stephan T. Lavavej (@StephanTLavavej) can you assign me this one ?

[AdamBucior]

Jean Philippe (@JeanPhilippeKernel) vNext branch is not available yet and as Stephan T. Lavavej (@StephanTLavavej) said in #290 it probably won't be before the second half of calendar year 2020.

[JeanPhilippeKernel]

ooh 😥😥😪 sad news !

[StephanTLavavej]

Yeah - we're eager to resume work on vNext but first we have to finish migrating our build/test infrastructure, and then we need to manually port the changes we accumulated for vNext in a Team Foundation Version Control branch years ago (the codebase has evolved considerably since then, so the port will be time-consuming). Thanks for your patience, we hope vNext will be worth the wait :-)

[JeanPhilippeKernel]

Oh okay, Sure;) !

[cschreib-ibex]

I have just found out our library is affected by exactly that same issue. Do you recommand a workaround to bypass this limitation while waiting for the fix to be implemented?

I seem to have gone around it by changing our custom streambuffer class to always use a read range of at most std::numeric_limits<int>::max() bytes, and override underflow() to move this range further whenever more bytes need to be read. Feels a bit hackish though.

[StephanTLavavej]

I'm not familiar enough with iostreams to say whether such a workaround is reliable (despite working on this codebase for 13 years). Billy O'Neal (@BillyONeal) may be able to comment, as he's delved into iostreams' arcane mysteries.

[BillyONeal]

I seem to have gone around it by changing our custom streambuffer class to always use a read range of at most std::numeric_limits<int>::max() bytes, and override underflow() to move this range further whenever more bytes need to be read. Feels a bit hackish though.

To the best of my understanding that should work (it's similar to how the stream buffers exposed don't contain all a file's contents) but we can't do it to the built in ones without breaking ABI.

[cschreib-ibex]

Thank you, we'll go for that workaround then. It passed all our tests, but of course they can't cover every possible case.

[JoelLinn]

Found this out the hard way…
I ended up overwriting xsputn and using my own pointer and length variables when inheriting from streambuf. It's enough for my use-case (ostream->pre-allocated memory buffer)

Please make sure this gets fixed in the next ABI version as 2GB isn‘t much these days.

[jimver04]

Any update on this ? see also my comment at : #578 (comment)