PHOENIX-6436 OrderedResultIterator overestimates memory requirements.

[lhofhansl]

See Jira.

the OrderedResultIterator will store topN values in either a BufferedQueue or a SizeBoundQueue. Each limit the memory used to the passed thresholdBytes (default is 20MB), the BufferQueue will spool to disk when reaching that size, SizeBoundQueue will fail.

Hence we additionally limit the worst case memory consumption to thresholdBytes.

I noticed this when implementing limit and topN pushdown for the Trino Phoenix connector: trinodb/trino#7490

[lhofhansl]

It turns out that for large sets Trino is actually performing the topN faster, even though it has to pull in all the data. This is due to the final client merge work that Phoenix is doing.
So it's less important now to fix this here, since I'm limiting topN pushdown in Trino now. Still wrong, though.

[stoty]

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	5m 34s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	16m 3s	master passed
+0	hbaserecompile	22m 25s	HBase recompiled.
+1 💚	compile	1m 1s	master passed
+1 💚	checkstyle	0m 38s	master passed
+1 💚	javadoc	0m 52s	master passed
+0 🆗	spotbugs	3m 4s	phoenix-core in master has 956 extant spotbugs warnings.
		_ Patch Compile Tests _
+1 💚	mvninstall	8m 6s	the patch passed
+0	hbaserecompile	18m 42s	HBase recompiled.
+1 💚	compile	1m 3s	the patch passed
+1 💚	javac	1m 3s	the patch passed
-1 ❌	checkstyle	0m 37s	phoenix-core: The patch generated 3 new + 193 unchanged - 2 fixed = 196 total (was 195)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 51s	the patch passed
+1 💚	spotbugs	3m 16s	the patch passed
		_ Other Tests _
-1 ❌	unit	107m 44s	phoenix-core in the patch failed.
+1 💚	asflicense	0m 45s	The patch does not generate ASF License warnings.
		166m 45s

Reason	Tests
Failed junit tests	phoenix.end2end.PermissionNSEnabledWithCustomAccessControllerIT

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1189/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#1189
Optional Tests	dupname asflicense javac javadoc unit spotbugs hbaserebuild hbaseanti checkstyle compile
uname	Linux 328f92a531e7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev/phoenix-personality.sh
git revision	master / 6e1b35d
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstyle	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1189/1/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unit	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1189/1/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Results	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1189/1/testReport/
Max. process+thread count	14326 (vs. ulimit of 30000)
modules	C: phoenix-core U: phoenix-core
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-1189/1/console
versions	git=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

[gjacoby126]

+1, thanks @lhofhansl

[lhofhansl]

Thanks @gjacoby126