TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null by jshmchenxi · Pull Request #117 · apache/tez

jshmchenxi · 2021-04-08T07:08:53Z

Issue: TEZ-4302 NullPointerException in CodecUtils when bufferSizeProp is null

hadoop-yetus · 2021-04-08T07:49:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	12m 56s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	12m 54s	master passed
+1 💚	compile	0m 35s	master passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 33s	master passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 8s	master passed
+1 💚	javadoc	0m 40s	master passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 30s	master passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+0 🆗	spotbugs	1m 25s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 23s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 21s	the patch passed
+1 💚	compile	0m 22s	the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 22s	the patch passed
+1 💚	compile	0m 19s	the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	0m 19s	the patch passed
+1 💚	checkstyle	0m 15s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 19s	the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 16s	the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	findbugs	0m 53s	the patch passed
		_ Other Tests _
+1 💚	unit	5m 18s	tez-runtime-library in the patch passed.
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		39m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-117/1/artifact/out/Dockerfile
GITHUB PR	#117
JIRA Issue	TEZ-4302
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux ff7d77fb1ce3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `2b671d6`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-117/1/testReport/
Max. process+thread count	116 (vs. ulimit of 5500)
modules	C: tez-runtime-library U: tez-runtime-library
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-117/1/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

abstractdog · 2021-04-08T09:08:35Z

 String bufferSizeProp = TezRuntimeUtils.getBufferSizeProperty(codec);
 Configurable configurableCodec = (Configurable) codec;
- int originalSize = configurableCodec.getConf().getInt(bufferSizeProp, DEFAULT_BUFFER_SIZE);
+ int originalSize = bufferSizeProp == null ? DEFAULT_BUFFER_SIZE :


thanks @jshmchenxi, this looks like an important bugfix
before merging this, could you please clarify what kind of codec have you used?
if bufferSizeProp is null, it means that the codec is not supported by TezRuntimeUtils.getBufferSizeProperty
if it's a codec supported by hadoop, we should add it, otherwise the DEFAULT_BUFFER_SIZE fallback makes sense

@abstractdog thanks for the quick reply! I've updated the patch. In our environment the codec is GzipCodec

thanks for the fix, I'm about to merge this soon if no objections
@jteagles: could you please contributor rights to @jshmchenxi to the Tez project in Jira?

@abstractdog, I have added contributor rights to @jshmchenxi to the Tez project in jira. I have also added you (@abstractdog) to the administrator role to the Tez which grants permissions to add contributors and other project level changes.

hadoop-yetus · 2021-04-08T12:45:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 30s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	12m 58s	master passed
+1 💚	compile	0m 34s	master passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	0m 33s	master passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 5s	master passed
+1 💚	javadoc	0m 40s	master passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 31s	master passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+0 🆗	spotbugs	1m 25s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	1m 23s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 21s	the patch passed
+1 💚	compile	0m 21s	the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	0m 21s	the patch passed
+1 💚	compile	0m 19s	the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	0m 19s	the patch passed
+1 💚	checkstyle	0m 14s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 18s	the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	0m 16s	the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	findbugs	0m 52s	the patch passed
		_ Other Tests _
+1 💚	unit	5m 9s	tez-runtime-library in the patch passed.
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		27m 10s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-117/2/artifact/out/Dockerfile
GITHUB PR	#117
JIRA Issue	TEZ-4302
Optional Tests	dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname	Linux 80b0de175ea2 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/tez.sh
git revision	master / `2b671d6`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-117/2/testReport/
Max. process+thread count	121 (vs. ulimit of 5500)
modules	C: tez-runtime-library U: tez-runtime-library
Console output	https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-117/2/console
versions	git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…i Chen reviewed by Laszlo Bodor)

abstractdog reviewed Apr 8, 2021
View reviewed changes

TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null
759f1af

jshmchenxi force-pushed the TEZ-4302 branch from 92247e2 to 759f1afCompare April 8, 2021 12:17

abstractdog merged commit b934644 into apache:masterApr 10, 2021

asfgit pushed a commit that referenced this pull request Apr 10, 2021
TEZ-4302: NullPointerException in CodecUtils with GzipCodec (#117) (X…
9e97913
…i Chen reviewed by Laszlo Bodor)

Navigation Menu

Uh oh!

TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null - #117

TEZ-4302: NullPointerException in CodecUtils when bufferSizeProp is null#117
abstractdog merged 1 commit into
apache:masterfrom
jshmchenxi:TEZ-4302

jshmchenxi commented Apr 8, 2021

Uh oh!

hadoop-yetus commented Apr 8, 2021

Uh oh!

abstractdogApr 8, 2021•
edited
Loading

Uh oh!

jshmchenxiApr 8, 2021

Uh oh!

abstractdogApr 8, 2021

Uh oh!

jteaglesApr 9, 2021

Uh oh!

hadoop-yetus commented Apr 8, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

jshmchenxi commented Apr 8, 2021

Uh oh!

hadoop-yetus commented Apr 8, 2021

Uh oh!

abstractdogApr 8, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jshmchenxiApr 8, 2021

Choose a reason for hiding this comment

Uh oh!

abstractdogApr 8, 2021

Choose a reason for hiding this comment

Uh oh!

jteaglesApr 9, 2021

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Apr 8, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

abstractdogApr 8, 2021•
edited
Loading