[SPARK-25492][TEST] Refactor WideSchemaBenchmark to use main method by wangyum · Pull Request #22501 · apache/spark

wangyum · 2018-09-20T16:06:52Z

What changes were proposed in this pull request?

Refactor WideSchemaBenchmark to use main method.

use spark-submit:

bin/spark-submit --class org.apache.spark.sql.execution.benchmark.WideSchemaBenchmark --jars ./core/target/spark-core_2.11-3.0.0-SNAPSHOT-tests.jar ./sql/core/target/spark-sql_2.11-3.0.0-SNAPSHOT-tests.jar

Generate benchmark result:

SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.WideSchemaBenchmark"

How was this patch tested?

manual tests

SparkQA · 2018-09-20T16:25:30Z

Test build #96369 has finished for PR 22501 at commit f56b732.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-09-24T03:33:25Z

+ * 1. without sbt: bin/spark-submit --class <this class> <spark sql test jar>
+ * 2. build/sbt "sql/test:runMain <this class>"
+ * 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain <this class>"
+ * Results will be written to "benchmarks/WideSchemaBenchmark-results.txt".


Could you fix doc generation failure?

Thanks @dongjoon-hyun. Actually I'm waiting for #22484. I want to move withTempDir() to RunBenchmarkWithCodegen.scala.

# Conflicts: # sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/WideSchemaBenchmark.scala

SparkQA · 2018-10-06T21:07:13Z

Test build #97056 has finished for PR 22501 at commit e6f39f3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-10-06T22:07:43Z

 file.createNewFile()
 }
- output = Some(new FileOutputStream(file))
+ output = Option(new FileOutputStream(file))


This looks like irrelevant pig-back.

Change here because: #22443 (comment)

IIUC, @HyukjinKwon meant when you need to touch this file.

I am worried that I will forget it after a long time, so I am changing this time. I should revert it?

Why do you replace Some to Option? Are you worrying new FileOutputStream(file) becomes null?

My point was that there's no point of checking null below from my cursory look. If there's no chance that it becomes null, we can leave it Some and remove null check below.

SparkQA · 2018-10-18T16:07:17Z

Test build #97534 has finished for PR 22501 at commit 82e2367.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-10-18T19:42:17Z

+1 cols x 100000 rows (read in-mem) 22 / 25 4.6 219.4 1.0X
+1 cols x 100000 rows (exec in-mem) 22 / 28 4.5 223.8 1.0X
+1 cols x 100000 rows (read parquet) 45 / 49 2.2 449.6 0.5X
+1 cols x 100000 rows (write parquet) 204 / 223 0.5 2044.4 0.1X


Given the difference on ratio, this might be a little regression on Parquet writer from Spark 2.1.0 (SPARK-17335).
cc @cloud-fan and @gatorsmile , @rdblue

I have no idea how this happens. Can you create a JIRA ticket to investigate this regression?

May be a parquet issue. I found that the binary write performance is a little worse after upgrading to parquet 1.10.0: apache/parquet-java#505. I will verify it later.

The following EC2 result shows the consistent ratio like Spark 2.1.0. The result on Mac seemed to be unstable for some unknown reason like #22501 (comment).
1 cols x 100000 rows (read parquet) 61/701.6610.20.6X 1 cols x 100000 rows (write parquet) 209/2330.52086.10.2X

@dongjoon-hyun, so you are saying that it doesn't appear that there is a performance regression, right?

For this part, right, @rdblue . I guess so.
After merging EC2 result to @wangyum 's PR, I'll compare the numbers one by one once again.

dongjoon-hyun · 2018-10-18T19:46:11Z

+2500 cols x 40 rows (read in-mem) 261 / 434 0.4 2607.3 0.1X
+2500 cols x 40 rows (exec in-mem) 624 / 701 0.2 6240.5 0.0X
+2500 cols x 40 rows (read parquet) 196 / 301 0.5 1963.4 0.1X
+2500 cols x 40 rows (write parquet) 687 / 1049 0.1 6870.6 0.0X


The gap between best and average is too high in line 32 and line 33.
I'll try to run this on EC2, too.

FYI, this large gap was removed at EC2 result.

dongjoon-hyun · 2018-10-18T19:51:31Z

+1 deep x 100000 rows (write parquet) 195 / 219 0.5 1945.1 0.1X
+100 deep x 1000 rows (read in-mem) 39 / 57 2.5 393.1 0.5X
+100 deep x 1000 rows (exec in-mem) 480 / 556 0.2 4795.7 0.0X
+100 deep x 1000 rows (read parquet) 7943 / 7950 0.0 79427.5 0.0X


Ur, @wangyum . Is this 4 times slower than before?
cc @dbtsai .

dongjoon-hyun · 2018-10-19T18:24:18Z

Hi, @wangyum . I ran the test on EC2 r3.xlarge, too. It looks more stable than this.
Could you review and merge wangyum#19 ?

cloud-fan · 2018-10-20T01:37:28Z

thank you guys for refreshing the benchmarks and results! It's very helpful.

If possible, can we post the perf regressions we found in the umbrella JIRA? Then people can see if the perf regression is reasonable(if we have addressed it) or investigate how the regression was introduced.

Thanks!

SparkQA · 2018-10-20T04:30:16Z

Test build #97627 has finished for PR 22501 at commit 64e5ede.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2018-10-20T04:50:22Z

seems jenkins is broken, cc @shaneknapp

Command "/tmp/tmp.JfFHaoRFPU/3.5/bin/python -c "import setuptools, tokenize;__file__='/home/jenkins/workspace/SparkPullRequestBuilder/python/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps" failed with error code 1 in /home/jenkins/workspace/SparkPullRequestBuilder/python/
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Cleaning up temporary directory - /tmp/tmp.JfFHaoRFPU
[error] running /home/jenkins/workspace/SparkPullRequestBuilder/dev/run-pip-tests ; received return code 1

dongjoon-hyun · 2018-10-20T06:39:54Z

@cloud-fan After updating on EC2, almost ratio and values looks more stable and reasonable for now. The following two are noticeable changes, but it looks like Parquet Writer improvement (instead of regression).

1. Read/Write ratio is reverted (0.8 -> 1.7)

-128 x 8 deep x 1000 rows (read parquet) 69/741.4693.90.2X
-128 x 8 deep x 1000 rows (write parquet) 78/831.3777.70.2X
+128 x 8 deep x 1000 rows (read parquet) 351/3790.33510.30.1X
+128 x 8 deep x 1000 rows (write parquet) 199/2030.51988.30.2X

2. Read/Write ratio is changed noticeably (4.6 -> 8.3)

-1024 x 11 deep x 100 rows (read parquet) 426/4330.24263.70.0X-1024 x 11 deep x 100 rows (write parquet) 91/981.1913.50.1X
+1024 x 11 deep x 100 rows (read parquet) 2063/20780.020629.20.0X+1024 x 11 deep x 100 rows (write parquet) 248/2660.42475.10.1X

Since this is the first attempt to track this and the previous result is too old, there exists some obvious limitation during comparison. From Spark 2.4.0, we can get a consistent compasison instead of different personal mac.

dongjoon-hyun · 2018-10-20T06:40:57Z

Retest this please.

SparkQA · 2018-10-20T07:05:02Z

Test build #97642 has finished for PR 22501 at commit 64e5ede.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2018-10-20T07:31:28Z

retest this please

SparkQA · 2018-10-20T10:52:10Z

Test build #97644 has finished for PR 22501 at commit 64e5ede.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

shaneknapp · 2018-10-20T15:44:30Z

@cloud-fan -- pip isn't broken... the actual error is found right above what you cut and pasted:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 2719: ordinal not in range(128)

i won't be able to look any deeper in to this until at least tomorrow at the earliest.

HyukjinKwon · 2018-10-20T16:03:33Z

I guess it's related with pip packaging tho.

 Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/setup.py", line 224, in <module>
'Programming Language :: Python :: Implementation :: PyPy']
File "/tmp/tmp.R2Y98bevgD/3.5/lib/python3.5/site-packages/setuptools/__init__.py", line 140, in setup
return distutils.core.setup(**attrs)
File "/tmp/tmp.R2Y98bevgD/3.5/lib/python3.5/distutils/core.py", line 148, in setup
dist.run_commands()
File "/tmp/tmp.R2Y98bevgD/3.5/lib/python3.5/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/tmp/tmp.R2Y98bevgD/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/tmp/tmp.R2Y98bevgD/3.5/lib/python3.5/site-packages/setuptools/command/develop.py", line 38, in run
self.install_for_development()
File "/tmp/tmp.R2Y98bevgD/3.5/lib/python3.5/site-packages/setuptools/command/develop.py", line 154, in install_for_development
self.process_distribution(None, self.dist, not self.no_deps)
File "/tmp/tmp.R2Y98bevgD/3.5/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 729, in process_distribution
self.install_egg_scripts(dist)
File "/tmp/tmp.R2Y98bevgD/3.5/lib/python3.5/site-packages/setuptools/command/develop.py", line 189, in install_egg_scripts
script_text = strm.read()
File "/tmp/tmp.R2Y98bevgD/3.5/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 2719: ordinal not in range(128)

It's from setup.py

kiszk · 2018-10-20T16:06:24Z

I am looking at each commit from the latest to old at https://github.com/apache/spark/commits/master

HyukjinKwon · 2018-10-20T16:09:12Z

Thanks. It might rather more be related to external factors.

kiszk · 2018-10-20T16:24:03Z

Thanks, when it was successful, this is a part of log from this

copying pyspark/streaming/util.py -> pyspark-3.0.0.dev0/pyspark/streaming
Writing pyspark-3.0.0.dev0/setup.cfg
Creating tar archive
removing 'pyspark-3.0.0.dev0' (and everything under it)
Installing dist into virtual env
Obtaining file:///home/jenkins/workspace/SparkPullRequestBuilder/python
Collecting py4j==0.10.7 (from pyspark==3.0.0.dev0)
Downloading https://files.pythonhosted.org/packages/e3/53/c737818eb9a7dc32a7cd4f1396e787bd94200c3997c72c1dbe028587bd76/py4j-0.10.7-py2.py3-none-any.whl (197kB)
mkl-random 1.0.1 requires cython, which is not installed.
Installing collected packages: py4j, pyspark
Running setup.py develop for pyspark
Successfully installed py4j-0.10.7 pyspark
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Run basic sanity check on pip installed version with spark-submit

Now, we are seeing the following. Is this a problem related to py4j or pyspark on external sites?

copying pyspark/streaming/util.py -> pyspark-3.0.0.dev0/pyspark/streaming
Writing pyspark-3.0.0.dev0/setup.cfg
Creating tar archive
removing 'pyspark-3.0.0.dev0' (and everything under it)
Installing dist into virtual env
Obtaining file:///home/jenkins/workspace/SparkPullRequestBuilder/python
Collecting py4j==0.10.7 (from pyspark==3.0.0.dev0)
Downloading https://files.pythonhosted.org/packages/e3/53/c737818eb9a7dc32a7cd4f1396e787bd94200c3997c72c1dbe028587bd76/py4j-0.10.7-py2.py3-none-any.whl (197kB)
mkl-random 1.0.1 requires cython, which is not installed.
Installing collected packages: py4j, pyspark
Running setup.py develop for pyspark
Complete output from command /tmp/tmp.EWtmCOYUBn/3.5/bin/python -c "import setuptools, tokenize;__file__='/home/jenkins/workspace/SparkPullRequestBuilder/python/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps:
running develop
running egg_info
writing dependency_links to pyspark.egg-info/dependency_links.txt
writing pyspark.egg-info/PKG-INFO
writing requirements to pyspark.egg-info/requires.txt
writing top-level names to pyspark.egg-info/top_level.txt
Could not import pypandoc - required to package PySpark
package init file 'deps/bin/__init__.py' not found (or not a regular file)
package init file 'deps/jars/__init__.py' not found (or not a regular file)
package init file 'pyspark/python/pyspark/__init__.py' not found (or not a regular file)
package init file 'lib/__init__.py' not found (or not a regular file)
package init file 'deps/data/__init__.py' not found (or not a regular file)
package init file 'deps/licenses/__init__.py' not found (or not a regular file)
package init file 'deps/examples/__init__.py' not found (or not a regular file)
reading manifest file 'pyspark.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*.py[cod]' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
writing manifest file 'pyspark.egg-info/SOURCES.txt'
running build_ext
Creating /tmp/tmp.EWtmCOYUBn/3.5/lib/python3.5/site-packages/pyspark.egg-link (link to .)
Adding pyspark 3.0.0.dev0 to easy-install.pth file
Installing load-spark-env.cmd script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing spark-submit script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing spark-class.cmd script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing beeline.cmd script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing find-spark-home.cmd script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing run-example script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing spark-shell2.cmd script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing pyspark script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing sparkR script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing spark-sql script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing spark-submit.cmd script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing spark-shell script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing beeline script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing spark-submit2.cmd script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing find-spark-home script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing sparkR.cmd script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing run-example.cmd script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing sparkR2.cmd script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing spark-shell.cmd script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing spark-sql.cmd script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Installing spark-class2.cmd script to /tmp/tmp.EWtmCOYUBn/3.5/bin
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/setup.py", line 224, in <module>
'Programming Language :: Python :: Implementation :: PyPy']
File "/tmp/tmp.EWtmCOYUBn/3.5/lib/python3.5/site-packages/setuptools/__init__.py", line 140, in setup
return distutils.core.setup(**attrs)
File "/tmp/tmp.EWtmCOYUBn/3.5/lib/python3.5/distutils/core.py", line 148, in setup
dist.run_commands()
File "/tmp/tmp.EWtmCOYUBn/3.5/lib/python3.5/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/tmp/tmp.EWtmCOYUBn/3.5/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/tmp/tmp.EWtmCOYUBn/3.5/lib/python3.5/site-packages/setuptools/command/develop.py", line 38, in run
self.install_for_development()
File "/tmp/tmp.EWtmCOYUBn/3.5/lib/python3.5/site-packages/setuptools/command/develop.py", line 154, in install_for_development
self.process_distribution(None, self.dist, not self.no_deps)
File "/tmp/tmp.EWtmCOYUBn/3.5/lib/python3.5/site-packages/setuptools/command/easy_install.py", line 729, in process_distribution
self.install_egg_scripts(dist)
File "/tmp/tmp.EWtmCOYUBn/3.5/lib/python3.5/site-packages/setuptools/command/develop.py", line 189, in install_egg_scripts
script_text = strm.read()
File "/tmp/tmp.EWtmCOYUBn/3.5/lib/python3.5/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 2719: ordinal not in range(128)
----------------------------------------

kiszk · 2018-10-20T16:41:22Z

Is this the oldest test failure related to this type of failure?

HyukjinKwon · 2018-10-20T16:51:47Z

Yup, I made a fix #22782

kiszk · 2018-10-20T16:53:04Z

Thanks, I found 0xc2 in docker-image-tool.sh. I will put my finding into #22782

dongjoon-hyun · 2018-10-20T19:12:00Z

Retest this please.

SparkQA · 2018-10-20T23:20:58Z

Test build #97665 has finished for PR 22501 at commit 64e5ede.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-10-21T00:30:31Z

Merged to master.

dongjoon-hyun · 2018-10-21T00:34:59Z

Thank you, @wangyum and all!

## What changes were proposed in this pull request? Refactor `WideSchemaBenchmark` to use main method. 1. use `spark-submit`: ```console bin/spark-submit --class org.apache.spark.sql.execution.benchmark.WideSchemaBenchmark --jars ./core/target/spark-core_2.11-3.0.0-SNAPSHOT-tests.jar ./sql/core/target/spark-sql_2.11-3.0.0-SNAPSHOT-tests.jar ``` 2. Generate benchmark result: ```console SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.WideSchemaBenchmark" ``` ## How was this patch tested? manual tests Closesapache#22501 from wangyum/SPARK-25492. Lead-authored-by: Yuming Wang <yumwang@ebay.com> Co-authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

MaxGekk · 2020-01-05T08:14:31Z

Any ideas how much memory does this benchmark require. I set 32Gb but it crashes with OOM on the master:

export SBT_OPTS="-Xmx32g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.WideSchemaBenchmark"

[error] Caused by: java.lang.reflect.InvocationTargetException
[error] at sun.reflect.GeneratedConstructorAccessor8.newInstance(Unknown Source)
[error] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error] at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[error] at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$makeCopy$7(TreeNode.scala:468)
[error] at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
[error] at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$makeCopy$1(TreeNode.scala:467)
[error] at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
[error] ... 132 more
[error] Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
[error] at java.util.Arrays.copyOf(Arrays.java:3332)
[error] at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
[error] at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
[error] at java.lang.StringBuilder.append(StringBuilder.java:136)
[error] at org.apache.spark.sql.types.StructType.catalogString(StructType.scala:411)
[error] at org.apache.spark.sql.types.StructType.$anonfun$catalogString$1(StructType.scala:410)
[error] at org.apache.spark.sql.types.StructType$$Lambda$2439/60886146.apply(Unknown Source)
[error] at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)

dongjoon-hyun · 2020-01-05T22:14:56Z

It sounds like a regression, @MaxGekk . This test suite was executed with the example in the code without any additional information like export SBT_OPTS="-Xmx32g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m".

dongjoon-hyun · 2020-01-05T22:18:02Z

FYI, this was tested in this PR, Oct 19, 2018, initially. The latest run was 854a0f7 , Oct 3, 2019 in the [SPARK-29320][TESTS] Compare sql/core module in JDK8/11 (Part 1) PR.

MaxGekk · 2020-01-06T07:44:15Z

I have rechecked the benchmark on the recent master - it is still failing with OOM. I opened the JIRA ticket: https://issues.apache.org/jira/browse/SPARK-30429

Refactor WideSchemaBenchmark
f56b732

dongjoon-hyun reviewed Sep 24, 2018
View reviewed changes

wangyum added 2 commits October 6, 2018 16:00

Merge remote-tracking branch 'upstream/master' intoSPARK-25492
7b2cb55
# Conflicts: # sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/WideSchemaBenchmark.scala

merge master
e6f39f3

dongjoon-hyun reviewed Oct 6, 2018
View reviewed changes

wangyum added 2 commits October 18, 2018 20:04

Merge remote-tracking branch 'upstream/master' intoSPARK-25492
085226c

revert BenchmarkBase
82e2367

dongjoon-hyun reviewed Oct 18, 2018
View reviewed changes

Update result (#19)
64e5ede

dongjoon-hyun approved these changes Oct 20, 2018
View reviewed changes

asfgit closed this in 62551ccOct 21, 2018

wangyum deleted the SPARK-25492 branch January 6, 2020 01:17

Uh oh!

Conversation

wangyum commented Sep 20, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Sep 20, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 6, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 18, 2018

Uh oh!

dongjoon-hyunOct 18, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyunOct 18, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Oct 19, 2018

Uh oh!

cloud-fan commented Oct 20, 2018

Uh oh!

SparkQA commented Oct 20, 2018

Uh oh!

cloud-fan commented Oct 20, 2018

Uh oh!

dongjoon-hyun commented Oct 20, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Oct 20, 2018

Uh oh!

SparkQA commented Oct 20, 2018

Uh oh!

cloud-fan commented Oct 20, 2018

Uh oh!

SparkQA commented Oct 20, 2018

Uh oh!

shaneknapp commented Oct 20, 2018

Uh oh!

HyukjinKwon commented Oct 20, 2018

Uh oh!

kiszk commented Oct 20, 2018

Uh oh!

HyukjinKwon commented Oct 20, 2018

Uh oh!

kiszk commented Oct 20, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

wangyum commented Sep 20, 2018•
edited
Loading

dongjoon-hyunOct 18, 2018•
edited
Loading

dongjoon-hyunOct 18, 2018•
edited
Loading

dongjoon-hyun commented Oct 20, 2018•
edited
Loading

kiszk commented Oct 20, 2018•
edited
Loading